Point cloud analysis method and device based on data multiplexing and calculation skipping, and readable medium

Abstract

The invention discloses a point cloud analysis method and device based on data multiplexing and calculation skipping and a readable medium, and the method comprises the steps: inputting to-be-analyzed point cloud data into a point cloud analysis model, entering a first hierarchy, outputting a sampling result through a sampling layer of the first hierarchy, multiplexing the sampling result of the sampling layer of the current level in the sampling process of the sampling layer of the next level, and adopting an inter-level data multiplexing method based on block-level pipeline delay aggregation for coordinates and features of reserved points in the sampling result of the current level to obtain aggregation features of the current level; and performing dynamic point pruning operation on the aggregation features of the current hierarchy through a pruning layer to obtain point cloud data output by the current hierarchy, taking the point cloud data output by the current hierarchy as point cloud data input by the next hierarchy, and taking the point cloud data output by the last hierarchy as an analysis result. According to the method, the calculation amount can be effectively reduced, and off-chip memory access is reduced to the maximum extent.

Description

Technical Field

[0001] The present invention relates to the field of point cloud processing, and in particular to a point cloud analysis method, device and readable medium based on data reuse and calculation skipping. Background Art

[0002] In recent years, point clouds have been increasingly used in various applications, such as cars, robots, drones, and depth cameras for smartphones. Point cloud data consists of a series of point coordinates and point feature (such as RGB color) data. Since only discrete points are needed to represent the features of the entire space, point cloud is an efficient spatial representation data format. Since devices that acquire point clouds, such as radars, can directly collect the exact coordinates of each point, the data representation of point clouds is more accurate and the resolution of point clouds is higher. Therefore, point clouds have significant advantages such as easy acquisition, high resolution, and efficient data format. Therefore, for deep learning applications such as autonomous driving, photography, virtual reality, and augmented reality, point clouds have become an important form as important as images and videos. Deep neural networks have been used for point cloud analysis. Point cloud neural networks are a type of neural network developed from PointNet, which performs well in point cloud analysis. Point cloud neural networks directly process points to extract features and solve the problems of data disorder and discreteness. In addition, the mapping operations (including sampling and combination operations) in point cloud neural networks enable the network to have multi-level and powerful feature extraction capabilities, thereby achieving high reasoning accuracy and robustness. The use of shared weight MLP and pooling in the point cloud neural network makes the model compact and computationally small.

[0003] When deploying point cloud neural networks to edge devices such as wearables, mobile devices, and Internet of Things (IoT) devices, these devices need to interact with humans in real time and perceive and understand the environment instantly, so high inference accuracy and low latency and energy consumption are crucial. However, the large amount of computation, large number of parameters, and low data reuse rate of point cloud neural networks lead to high energy consumption and long latency for computing and storage access, which cannot meet the needs of devices. Therefore, there is an urgent need to design a point cloud neural network that can minimize off-chip memory access while reducing the amount of computation.

[0004] like Figure 1As shown in the figure, the input of the point cloud neural network is point cloud data, and the output is the analysis result, such as identifying the category of the point cloud, the location and occupied area of ​​the object in the point cloud, and which object each point in the point cloud belongs to. The backbone of the architecture consists of multiple levels. Each level generally contains a mapping layer, several consecutive multilayer perceptron (MLP) layers, and an aggregation layer. The disorder and discreteness of point cloud data pose considerable challenges to feature extraction. In order to meet these challenges, the mapping layer of the point cloud neural network uses sampling and grouping operations to construct the relationship between input points and output points, realize powerful multi-level feature extraction, and thus achieve high accuracy and robustness of reasoning.

[0005] Sampling: Point cloud neural networks generally use the farthest point sampling (FPS) method to reduce the amount of data while enhancing the data representation ability. By selecting the point farthest from the previous sampling point in each iteration, a subset of the point cloud is gradually generated. The sampling process of the original FPS is to find all output points one by one from the input point cloud data to be analyzed. The process of sampling all output points is divided into two steps. In the first step, a point is randomly selected from the point cloud data as the initial sampling point, and then the Euclidean distance between each unsampled point and the sampled point cloud is calculated. In the second step, the point with the largest Euclidean distance is added to the sampled point cloud. The whole process of FPS is as follows: Figure 1 As shown in (a), assume that the initial sampling point a is randomly selected from the input point cloud data to be analyzed, and then calculate and compare the Euclidean distance between a and other points. Since b has the largest Euclidean distance, b is selected as the second sampling point in the sampled point cloud. Then a new round of iterations begins, and after calculating the distance between each unsampled point (c, d, e, f) and the sampled point cloud (a, b), {a, b, c} is used as the sampled point cloud. In short, FPS discards some points of the input point cloud (IPC), and the discarded points are called DP (Discarded Points), and the other points remain unchanged, and the retained points are called RP (Retained Points).

[0006] Combination: After FPS samples the point cloud, the combination operation uses each point in RP as the center point (Center Points, CP) and searches for the neighboring points (Neighbor Points, NP) of each CP in the point cloud IPC (including RP and DP) before FPS sampling. The combination operation does not change the coordinates of the point, but is only used to build the relationship between each point in RP and IPC, and obtain the mapping relationship table between all RP points and IPC, which is called NIT (Neighbor Index Table). Point cloud neural networks usually use KNN (K-Nearest-Neighbors) algorithm or BQ (Ball Query) algorithm to complete the combination operation. KNN will select the first K nearest points based on the distance from all points in IPC to CP. BQ further requires that these neighboring points must be located within a ball with a radius of R based on KNN. For example Figure 1 As shown in (b), assuming that the number of neighboring input points K is 3, for the output point a', the BQ algorithm searches for 4 input points a, c, d and e in a sphere with a radius of R centered on the output point a', and then selects the three points a, d and e closest to the output point a'. BQ regards points a, d and e as the neighboring point combination of the output point a'. The relationship between the output point a' and the input points a, d and e is called the mapping of the output point a'. The combination operation also combines the features of points a, d and e into a feature group, which is used by the subsequent MLP to calculate the features of the output point a'.

[0007] MLP: MLP is a convolutional layer with a kernel size of 1×1. In the MLP of the point cloud neural network, the filter is convolved with the features of all input points. Figure 1 (d) shows the changes in the point cloud within a layer, where p represents the coordinates of the input point, F represents the feature group (the combination operation combines the features of neighboring points), and R represents the part and group after executing multiple MLPs on the input feature group. The point cloud neural network stacks multiple MLPs in each layer to enhance the ability of feature extraction.

[0008] Aggregation: Point cloud neural networks often use max pooling to aggregate parts and groups computed by MLP from features of neighboring points. Figure 1 As shown in (d), the channel maximum values ​​of the partial and group R0, R2, R3, R4, R5 and R7 are calculated to obtain the feature F0' of the output point.

[0009] However, compared with 2D convolutional neural networks, point cloud neural networks can only read / write the features of one point in the memory at a time during processing of point cloud data, instead of reading / writing a whole block of feature maps and filters from the off-chip memory at a time as when running 2D convolutional neural networks, resulting in low efficiency of interface bandwidth. In addition, after delayed aggregation, the output feature maps need to be combined into output feature map groups, and at this time the number of channels of the output feature maps is usually twice that of the input feature maps, so the number of feature channels that need to be combined in the combination operation is also doubled, significantly increasing the search domain, computational complexity, storage space occupancy, and off-chip storage access of the combination operation. Therefore, the combination operation and aggregation operation limit the acceleration effect and become a new bottleneck besides convolution. Summary of the invention

[0010] The purpose of this application is to propose a point cloud analysis method, device and readable medium based on data reuse and calculation skipping to address the above-mentioned technical problems.

[0011] In a first aspect, the present invention provides a point cloud analysis method based on data reuse and calculation skipping, comprising the following steps:

[0012] Obtain point cloud data to be analyzed;

[0013] A point cloud analysis model based on data reuse and calculation skipping is constructed and trained to obtain a trained point cloud analysis model, wherein the point cloud neural network includes several layers, each layer includes a sampling layer, a combination layer, a multi-layer perceptron unit, an aggregation layer, and a pruning layer, and the multi-layer perceptron unit includes a plurality of multi-layer perceptrons connected in sequence;

[0014] The point cloud data to be analyzed is input into the trained point cloud analysis model. The point cloud data to be analyzed is input into the first level of the trained point cloud analysis model. The sampling result is output through the sampling layer of the first level. The discarded points and retained points of the next level of sampling layer are determined based on the sampling result of the current level of sampling layer. The index and coordinates of the retained points of each level of sampling layer are established in sequence and combined with the features of the retained points to output as the sampling result. The sampling result of the current level of sampling layer is reused in the sampling process of the next level of sampling layer. The coordinates and features of the retained points in the sampling results adopt an inter-level data reuse method based on block-level pipeline delayed aggregation in the combination layer, multi-layer perceptron unit and aggregation layer of the current level, and skip the combination operation, convolution operation and aggregation operation of the discarded points of the current level in the previous level to obtain the aggregation features of the current level. The aggregation features of the current level are subjected to dynamic point pruning operation through the pruning layer to obtain the point cloud data output by the current level. The point cloud data output by the current level is used as the point cloud data input by the next level, and the point cloud data output by the last level is used as the analysis result.

[0015] In a specific embodiment, the coordinates and features of the reserved points in the sampling results of the current level are reused in the combination layer, multi-layer perceptron unit and aggregation layer of the current level using an inter-level data multiplexing method based on block-level pipeline delay aggregation, specifically including:

[0016] The coordinates and features of the retained points in the sampling results of the current level are divided into multiple point cloud blocks, and the following pipeline process is repeated for different point cloud blocks at different time steps:

[0017] The coordinates of the reserved points in the point cloud block are input into the combination layer for combination operation to obtain the neighbor index table;

[0018] The features of the retained points in the point cloud block are input into the multi-layer perceptron unit for convolution operation to obtain the output feature map;

[0019] The neighbor index table and the output feature map are input into the aggregation layer for aggregation operation to obtain the aggregated features of the point cloud block;

[0020] The aggregate features of all point cloud blocks constitute the aggregate features of the current level.

[0021] In a specific embodiment, in the combination operation, each reserved point of the current level is used as the center point, and the neighboring points of each center point are searched in the point cloud data input by the sampling layer of the current level by using the neighborhood search process, and a mapping relationship between all the reserved points and their neighboring points in the point cloud data input by the sampling layer of the current level is established to obtain a neighbor index table;

[0022] In the aggregation operation, the index of the neighboring point in the neighbor index table is used as the address, and the feature maps corresponding to the K neighboring points of the center point are obtained and aggregated in the output feature map to obtain the aggregated features of the point cloud block.

[0023] In a specific embodiment, a mask is constructed according to the sampling results of the sampling layer at the current level, and the retained points in the mask are recorded as 1 and the discarded points are recorded as 0;

[0024] Before the combination operation of the first level combination layer, the distance set between all points in the point cloud data to be analyzed is calculated and sorted from small to large to form an index sequence of points with distances from the center point from small to large;

[0025] The mask obtained according to the sampling result of the sampling layer of the current level selectively filters out the index sequence of the current level in the index sequence of the previous level, and the distance in the index sequence of the current level is used in the neighborhood search process in the combination operation of the current level.

[0026] In a specific embodiment, it also includes:

[0027] Determine, in the currently input point cloud data to be analyzed, a newly added point cloud relative to the last input point cloud data to be analyzed, and determine, in the currently input point cloud data to be analyzed, an associated portion corresponding to the last input point cloud data to be analyzed;

[0028] In the current level, the calculation results of the newly added point cloud in the sampling layer, combination layer, multi-layer perceptron unit, and aggregation layer are calculated respectively, and combined with the calculation results of the associated parts in the last input point cloud data to be analyzed in the sampling layer, combination layer, multi-layer perceptron unit, and aggregation layer respectively, to obtain the aggregation features of the current input point cloud data to be analyzed at the current level, and then through the pruning layer, the point cloud data output at the current level is obtained.

[0029] In a specific embodiment, the following process is performed in the pruning layer:

[0030] S41, sorting the aggregated features of the current level according to the numerical values, and determining the threshold of the current level;

[0031] S42, setting the values ​​of the aggregated features of the current level that are lower than the threshold of the current level to zero;

[0032] S43, determine whether the threshold of the current level meets the requirements, if so, obtain the point cloud data output by the current level, otherwise adjust the threshold of the current level, and repeat steps S42-S43 until the threshold of the current level meets the requirements.

[0033] In a second aspect, the present invention provides a point cloud analysis device based on data reuse and calculation skipping, comprising:

[0034] A data acquisition module is configured to acquire point cloud data to be analyzed;

[0035] A model building module is configured to build and train a point cloud analysis model based on data reuse and calculation skipping to obtain a trained point cloud analysis model, wherein the point cloud neural network includes a plurality of layers, each of which includes a sampling layer, a combination layer, a multi-layer perceptron unit, an aggregation layer, and a pruning layer, and the multi-layer perceptron unit includes a plurality of multi-layer perceptrons connected in sequence;

[0036] The analysis module is configured to input the point cloud data to be analyzed into the trained point cloud analysis model, input the point cloud data to be analyzed into the first level of the trained point cloud analysis model, output the sampling result through the sampling layer of the first level, determine the discarded points and retained points of the next level of sampling layer based on the sampling result of the current level of sampling layer, establish the index and coordinates of the retained points of each level of sampling layer in sequence and output them as the sampling result in combination with the features of the retained points, reuse the sampling result of the current level of sampling layer in the sampling process of the next level of sampling layer, The coordinates and features of the retained points in the sampling results of the current level adopt an inter-level data reuse method based on block-level pipeline delayed aggregation in the combination layer, multi-layer perceptron unit and aggregation layer of the current level, and skip the combination operation, convolution operation and aggregation operation of the discarded points of the current level in the previous level to obtain the aggregation features of the current level. The aggregation features of the current level are subjected to dynamic point pruning operation through the pruning layer to obtain the point cloud data output by the current level. The point cloud data output by the current level is used as the point cloud data input to the next level, and the point cloud data output by the last level is used as the analysis result.

[0037] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0039] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The point cloud analysis method based on data reuse and computation skipping proposed in the present invention adopts inter-level data reuse based on block-level pipeline delay aggregation, which greatly reduces the search domain, hardware complexity and off-chip storage access of the combination operation, eliminates the combination operation bottleneck in hardware acceleration, and improves hardware utilization.

[0042] (2) The point cloud analysis method based on data reuse and calculation skipping proposed in the present invention adopts a low hardware complexity a priori calculation skipping based on sampling, which can skip the retained points that have been calculated in the sampling layer, and skip the neighborhood search with the discarded point as the center point or the neighboring point, the convolution operation and aggregation operation related to the discarded point, which greatly reduces the amount of calculation without affecting the calculation results.

[0043] (3) The point cloud analysis method based on data reuse and calculation skipping proposed in the present invention reuses the calculation results of the associated part between two adjacent input point cloud data to be analyzed, which can effectively reduce the amount of calculation, thereby greatly improving the energy efficiency of the point cloud analysis model and reducing the reasoning delay. The mask obtained by the sampling layer filters the index sequence of the previous level, reducing the calculation operations and storage access of the neighborhood search of the combination operation, and further dynamically prunes the points with smaller values ​​in the aggregated features through the pruning layer, so that the point cloud analysis model can achieve the desired balance between accuracy and hardware energy efficiency and delay according to the actual application scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 The calculation process of the point cloud neural network in the prior art;

[0046] Figure 2 A schematic flow chart of a point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a block-level pipeline delayed aggregation (BPDA) calculation process of a point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a pipeline for processing multiple point cloud blocks by convolution operations, combination operations, and aggregation operations of a point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a process of a priori calculation skipping based on farthest point sampling in a point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0050] Figure 6A schematic diagram of a specific implementation example of a priori calculation skipping based on farthest point sampling of a point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0051] Figure 7 A schematic diagram of reusing calculation results of associated parts in a single calculation unit of a point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0052] Figure 8 A schematic diagram of using calculation results of associated parts in a complete calculation process of a point cloud analysis method based on data reuse and calculation skipping in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of a combined inter-layer multiplexing neighboring point index of a point cloud analysis method based on data multiplexing and calculation skipping according to an embodiment of the present application;

[0054] Fig.10 A schematic diagram of a point cloud analysis method based on data reuse and calculation skipping in an embodiment of the present application, in which a combination operation at a next level reuses neighboring point indexes of a combination operation at a previous level in the entire calculation process;

[0055] Fig.11 A schematic diagram showing the comparison before and after dynamic point pruning of a point cloud analysis method based on data reuse and calculation skipping in an embodiment of the present application;

[0056] Fig.12 A flowchart of a dynamic point pruning method of a point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0057] Fig.13 A schematic diagram of the position of the pruning layer in the entire point cloud analysis model of the point cloud analysis method based on data reuse and calculation skipping according to an embodiment of the present application;

[0058] Fig.14 A schematic diagram of a point cloud analysis device based on data reuse and calculation skipping according to an embodiment of the present application;

[0059] Fig.15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Figure 2 A point cloud analysis method based on data reuse and calculation skipping provided by an embodiment of the present application is shown, characterized in that it includes the following steps:

[0062] S1, obtain the point cloud data to be analyzed.

[0063] Specifically, first obtain the point cloud data to be analyzed. The point cloud data to be analyzed input in the reasoning task is continuous point cloud data. In applications such as autonomous driving, drones, AR and VR, there is a strong correlation between the continuously input three-dimensional point clouds. There is a correlation between the currently input point cloud data to be analyzed and the next input point cloud data to be analyzed, that is, part of the content of the currently input point cloud data to be analyzed also exists in the next input point cloud data to be analyzed. There are also newly added point clouds in the next input point cloud data to be analyzed. The point cloud data to be analyzed contains the coordinates and features of each point, and the features include information such as color.

[0064] S2, constructs and trains a point cloud analysis model based on data reuse and calculation skipping to obtain a trained point cloud analysis model. The point cloud neural network includes several layers, each layer includes a sampling layer, a combination layer, a multi-layer perceptron unit, an aggregation layer and a pruning layer, and the multi-layer perceptron unit includes multiple multi-layer perceptrons connected in sequence.

[0065] Specifically, the number of multi-layer perceptrons in the multi-layer perceptron unit is M, and the multi-layer perceptron has a convolution layer with a convolution kernel size of 1×1.

[0066] S3, input the point cloud data to be analyzed into the trained point cloud analysis model, input the point cloud data to be analyzed into the first level of the trained point cloud analysis model, output the sampling result through the sampling layer of the first level, determine the discarded points and retained points of the next level of sampling layer based on the sampling result of the current level of sampling layer, establish the index and coordinates of the retained points of each level of sampling layer in sequence and output them as the sampling result in combination with the features of the retained points, reuse the sampling result of the current level of sampling layer in the sampling process of the next level of sampling layer, and The coordinates and features of the retained points in the sampling results of the current level are reused in the combination layer, multi-layer perceptron unit and aggregation layer of the current level based on the block-level pipeline delayed aggregation inter-level data reuse method, and the combination operation, convolution operation and aggregation operation of the discarded points of the current level in the previous level are skipped to obtain the aggregation features of the current level. The aggregation features of the current level are subjected to dynamic point pruning operation by the pruning layer to obtain the point cloud data output by the current level. The point cloud data output by the current level is used as the point cloud data input by the next level, and the point cloud data output by the last level is used as the analysis result.

[0067] In a specific embodiment, the coordinates and features of the reserved points in the sampling results of the current level are reused in the combination layer, multi-layer perceptron unit and aggregation layer of the current level using an inter-level data multiplexing method based on block-level pipeline delay aggregation, specifically including:

[0068] The coordinates and features of the retained points in the sampling results of the current level are divided into multiple point cloud blocks, and the following pipeline process is repeated for different point cloud blocks at different time steps:

[0069] The coordinates of the reserved points in the point cloud block are input into the combination layer for combination operation to obtain the neighbor index table;

[0070] The features of the retained points in the point cloud block are input into the multi-layer perceptron unit for convolution operation to obtain the output feature map;

[0071] The neighbor index table and the output feature map are input into the aggregation layer for aggregation operation to obtain the aggregated features of the point cloud block;

[0072] The aggregate features of all point cloud blocks constitute the aggregate features of the current level.

[0073] In a specific embodiment, in the combination operation, each reserved point of the current level is used as the center point, and the neighboring points of each center point are searched in the point cloud data input by the sampling layer of the current level by using the neighborhood search process, and a mapping relationship between all the reserved points and their neighboring points in the point cloud data input by the sampling layer of the current level is established to obtain a neighbor index table;

[0074] In the aggregation operation, the index of the neighboring point in the neighbor index table is used as the address, and the feature maps corresponding to the K neighboring points of the center point are obtained and aggregated in the output feature map to obtain the aggregated features of the point cloud block.

[0075] In a specific embodiment, a mask is constructed according to the sampling results of the sampling layer at the current level, and the retained points in the mask are recorded as 1 and the discarded points are recorded as 0;

[0076] Before the combination operation of the first level combination layer, the distance set between all points in the point cloud data to be analyzed is calculated and sorted from small to large to form an index sequence of points with distances from the center point from small to large;

[0077] The mask obtained according to the sampling result of the sampling layer of the current level selectively filters out the index sequence of the current level in the index sequence of the previous level, and the distance in the index sequence of the current level is used in the neighborhood search process in the combination operation of the current level.

[0078] In a specific embodiment, it also includes:

[0079] Determine, in the currently input point cloud data to be analyzed, a newly added point cloud relative to the last input point cloud data to be analyzed, and determine, in the currently input point cloud data to be analyzed, an associated portion corresponding to the last input point cloud data to be analyzed;

[0080] In the current level, the calculation results of the newly added point cloud in the sampling layer, combination layer, multi-layer perceptron unit, and aggregation layer are calculated respectively, and combined with the calculation results of the associated parts of the last input point cloud data to be analyzed in the sampling layer, combination layer, multi-layer perceptron unit, and aggregation layer respectively, to obtain the aggregation features of the currently input point cloud data to be analyzed at the current level, and then through the pruning layer, the point cloud data output at the current level is obtained.

[0081] In a specific embodiment, the following process is performed in the pruning layer:

[0082] S41, sorting the aggregated features of the current level according to the numerical values, and determining the threshold of the current level;

[0083] S42, setting the values ​​of the aggregated features of the current level that are lower than the threshold of the current level to zero;

[0084] S43, determine whether the threshold of the current level meets the requirements, if so, obtain the point cloud data output by the current level, otherwise adjust the threshold of the current level, and repeat steps S42-S43 until the threshold of the current level meets the requirements.

[0085] Specifically, the point cloud analysis model constructed in the embodiments of the present application is improved on the basis of the point cloud neural network mentioned in the background technology, and the improved parts are divided into the following points.

[0086] 1. Data reuse between layers based on block-level pipeline delay aggregation

[0087] Compared with two-dimensional convolution, the activation reuse of the multi-layer perceptron (MLP) of the point cloud neural network has only one dimension, and the activation reuse rate is low, resulting in greater bandwidth requirements. When the bandwidth is fixed, the utilization rate of the computing unit will be significantly reduced. Therefore, the point cloud analysis model proposed in the embodiment of the present application adopts delayed aggregation technology. The core idea of ​​delayed aggregation is to change the order of convolution operations and combination operations, and to set the combination layer in each level of the point cloud neural network before the multi-layer perceptron unit instead of running the combination layer and the multi-layer perceptron unit in each level in parallel, so as to first calculate the convolution results of a single point and the combination results of each point separately, and then use the combination results to aggregate the convolution results, instead of the calculation process of the native algorithm of first combining and then convolving.

[0088] The embodiment of the present application proposes a method for multiplexing data between layers based on block-wise pipeline delayed aggregation (BPDA). Figure 3 As shown, the following steps are included:

[0089] In step (1), the input data of each level includes the coordinates and features of the point cloud output by the previous level. The coordinates and features of the input point cloud of the current level are first divided into multiple point cloud blocks, and then the combination operation is performed within the point cloud block, that is, the neighboring point search is performed within the point cloud block to obtain the neighbor index table (NIT). Due to the delayed aggregation, the features of the points are first subjected to the convolution operation of M layers, which increases the number of feature channels by 2. M times.

[0090] In step (2), after performing convolution calculation on an input point cloud block in the multi-layer perceptron unit of the current level, the output feature map is obtained and stored.

[0091] In step (3), the aggregation layer uses the NIT pre-calculated by the combination layer, takes the index of the neighboring points in the NIT as the address of the output feature map, reads the feature maps of the K neighboring points of the center point from the output feature map, and then aggregates the features of the neighborhood to obtain the aggregated features of each point cloud block, and combines the aggregated features of all point cloud blocks into the aggregated features of the current level.

[0092] In the original point cloud neural network, each partial sum of the multi-layer perceptron (MLP) output is only used once by an aggregation operation (e.g., maximum pooling) to generate features of the corresponding output points based on the mapping relationship. In the BPDA proposed in the embodiment of the present application, the output results of the multi-layer perceptron are used by multiple aggregation operations. Therefore, BPDA increases the number of reuses of the convolution results to K', reduces the amount of calculation to approximately 1 / K' of the original, and shortens the calculation time of the multi-layer perceptron unit. At the same time, BPDA performs block convolution on the input point cloud, greatly reducing the search domain, hardware complexity, and off-chip storage access of the combination operation. Theoretically, when the number of point cloud blocks is B, the BPDA's block method reduces the search domain of the combination operation and the amount of off-chip storage access data to 1 / B of the original, eliminating the bottleneck of the combination operation in hardware acceleration. Figure 4 As shown, the convolution operations, combination operations and aggregation operations of multiple point cloud blocks in the layer are executed on the hardware in a pipeline manner. The convolution operations, combination operations and aggregation operations of each point cloud block are executed in the multi-layer perceptron unit, the combination layer and the aggregation layer in sequence, and the point cloud blocks are sent to these three layers in sequence at equal time intervals, thereby reducing the idle cycles of the convolution layer, the combination layer and the aggregation layer and improving the hardware utilization.

[0093] 2. Low hardware complexity a priori calculation skipping based on sampling

[0094] The embodiment of the present application also proposes a sampling-based skipping method (Sampling-Based Skipping, SBS), such as Figure 5 As shown, there are 4 types of calculation skipping:

[0095] ① Skip the calculated reserved points (RP) in the sampling operation;

[0096] ② is sparse skipping of the discarded point (DP), skipping the group operation with the discarded point (DP) as the center point and skipping the search process with the discarded point (DP) as the neighboring point;

[0097] ③ is to skip the convolution operation related to the discarded point (DP);

[0098] ④ is to skip the aggregation operation related to the discard point (DP).

[0099] For ①, if Figure 5 As shown in the figure, by fixing the initial point of the farthest point sampling (FPS) of each level to be the same, for example, p1, because in all sampling layers of the point cloud neural network, the coordinates of the points remain unchanged, only the number of points is reduced. Therefore, after fixing the initial point to be the same, in each sampling layer, the output order of the sampling operation from the first point p1 to the subsequent sampling points p2, p3, p4, etc. remains unchanged. The sampling layer of the first level has a complete point cloud, and will first output the sampling results of the sampling layer of the last level (such as p1-p4 of the Lth layer in the figure), and then output the sampling results of the sampling layer of the second to last level (such as p1-p6 of the (L-1)th layer in the figure), until the sampling results of the sampling layer of the first level are output, that is, the reserved points (RP) of the output point cloud. Note that the sampling of the (L-1)th level reuses the p1-p4 already calculated in the Lth layer, that is, the subsequent sampling layers reuse the sampling results of the sampling layer of the previous level. In general, it is achieved that only the sampling results of the sampling layer of the first level are calculated, and the sampling results of the subsequent sampling layers are output at the same time.

[0100] Since some points will be discarded in the sampling layer of the next level in the point cloud neural network, the combination operation, aggregation operation and convolution operation of these points in the previous level can be skipped in advance. The BPDA proposed in Improvement Point 1 decouples mapping construction and feature extraction, so that the mapping relationship can be established in advance before feature extraction. Therefore, the input points and output points of the mapping operation, including the input points and output points of the sampling operation and the combination operation, can be calculated in advance. By first performing the sampling operation of the next layer, the discarded point (DP) of the next sampling layer is obtained, and then the combination operation, aggregation operation and convolution operation of the discarded point (DP) in the previous level are skipped. Figure 5As shown in the figure, in the point set P, the result of the farthest point sampling (FPS) of the next level can be used to skip the neighborhood search operation of the discarded point (DP) of the current level, including two skipping situations: taking the discarded point (DP) as the center point and taking the discarded point (DP) as the neighboring point (② skip combination in the figure) and the aggregation operation guided by the combination operation (④ skip aggregation in the figure). In addition, since the convolution kernel size in the multi-layer perceptron is 1×1, there is no correlation between points during convolution, and SBS directly skips the convolution operation of some points without affecting the convolution calculation of other points. Therefore, Figure 5 As shown in the figure, SBS can skip the convolution operation of discarded points (DP) (③ skip convolution in the figure). Usually, when the sampling rate of FPS is 1 / 2, the SBS method can reduce 3 / 4 of the neighborhood search and aggregation operations and 1 / 2 of the convolution calculations by skipping 1 / 2 of the center points and 1 / 2 of the neighboring points at the same time, without affecting the calculation results. When the data set is large, such as the S3DIS data set, the sampling rate of FPS is often only 1 / 4. The SBS method can reduce 15 / 16 of the neighborhood search and aggregation operations, and the improvement effect is more significant.

[0101] Figure 6 A specific implementation example of the SBS method is given, but SBS is not limited to this implementation. Figure 6 As shown, only the sampling layer of the first level is calculated and the results of the sampling layers of all levels are output in order. After the sampling layer calculates the FPS of the next level, the index (Valid Index) of the reserved point (RP) is stored in the storage module. When the combination layer calculates the mapping relationship between the input point and the output point of the current level, the index of the reserved point (RP) is read from the storage module into the index buffer. Figure 6 As shown, when the combination module performs combination calculation, the neighborhood search process with the discarded point (DP) as the center point and the discarded point (DP) as the neighboring point is skipped.

[0102] The combination layer passes the calculated mapping relationship to the aggregation layer to aggregate the neighborhood features of the center point, and the aggregation layer also skips the aggregation operation with the sampled discard point (DP) of the next level as the center point. Figure 6 Therefore, the input features of the multi-layer perceptron unit of the current layer to the next layer only retain the features of valid points such as P1 and P2, and the features of other points such as P5 and P6 will neither participate in the calculation of the multi-layer perceptron unit of the next layer nor be stored on the chip.

[0103] 3. Inference-level data reuse based on spatial locality

[0104] like Figure 7 As shown, the input point cloud data to be analyzed of the point cloud analysis model is usually composed of the associated part of the point cloud data to be analyzed input last time ( Figure 7The new point cloud is composed of two parts: the area covered by the red slash in the image and the updated new point cloud on the edge. Therefore, during reasoning, it is only necessary to calculate and update the new point cloud on the edge, and then combine it with the calculation result of the associated part corresponding to the last input point cloud data to be analyzed to obtain the calculation result of the current point cloud. In a specific embodiment, the calculation result of the new point cloud is spliced ​​with the calculation result of the associated part to obtain the calculation result of the currently input point cloud data to be analyzed. At the same time, the calculation result of the current point cloud is stored in the storage unit to prepare for the reuse of the calculation result of the associated part in the calculation process of the next input point cloud data to be analyzed. Figure 8 As shown in the figure, during the entire calculation process of the point cloud analysis model, the calculation operations of the current point cloud, such as sampling, combination, convolution, aggregation, etc., only need to calculate the updated part of the point cloud, and then combine it with the associated part of the last input point cloud to form the complete calculation result of the current point cloud for reasoning, and store it for reuse during the next output point cloud reasoning. Therefore, this method can directly reuse the calculation results of the associated part of the last point cloud and the current input point cloud (such as mapping and convolution results), so as to achieve the purpose of reusing part of the results of the previous reasoning for the next reasoning. The stronger the correlation between the continuous input point clouds, the more the amount of calculation and data access storage can be reduced, thereby greatly improving the energy efficiency of the point cloud analysis model and reducing the reasoning delay.

[0105] 4. Inter-layer data reuse of search distance

[0106] The neighborhood search of the combined operation in the point cloud neural network needs to traverse the coordinates of all points multiple times and calculate the Euclidean distance between points. Therefore, the neighborhood search of the combined operation introduces a large number of computing operations and storage accesses. Fig. 9 As shown, an embodiment of the present application pre-constructs a mask (Mask) corresponding to the input point cloud of each level of the combination layer in the sampling layer of the point cloud analysis model, that is, which points are retained points that need to be retained, and which points are discarded points that need to be discarded. In the mask, 1 represents a retained point, and 0 represents a discarded point, which will not participate in the neighboring point search of the next combination layer of the sampling layer. Therefore, the input point cloud of the combination layer of the next level is a subset of the input point cloud of the combination layer of the current level, and the calculation operation of the combination layer of the next level can directly reuse the distance set between the points calculated by the neighboring search of the combination layer of the current level. Moreover, the combination layer of the first level previously calculated the complete distance set between all points and sorted them from small to large, forming an index sequence of points with distances to the center point from small to large, such as Fig. 9As shown. The combination layer of the subsequent layers only needs to selectively filter the index sequence according to the sampling result, that is, the mask output by the sampling layer, and then take the point index corresponding to the K minimum distances and output the mapping relationship of the combination operation. There is no need to recalculate the distance between points in the combination layer of the second layer. The combination layers of the subsequent layers are similar, and they only need to filter out the index sequence from the previous layer without calculating the distance between points. Fig.10 As shown in Figure 2, during the entire calculation process of the point cloud analysis model, the combined operation results of level 2 ( Fig.10 The green arrow in the figure is the mask of the sampling operation at level 2 ( Fig.10 The purple arrow in the figure filters the index sequence of level 1 ( Fig.10 Finally, the distance relationship is calculated once, and all combination layers can reuse the distance relationship to avoid repeated distance calculation at each combination layer.

[0107] 5. Dynamic point pruning based on eigenvalue importance sorting

[0108] In terms of computational skipping, each point in the point cloud space has different importance for network reasoning. For example, points on the edge contour have stronger representation capabilities than points in the middle of an object, so points on the edge contour are more important. Fig.11 As shown in the figure, blue points have larger values ​​and are located at the contour of the point. They have strong representation ability and play a key role in feature extraction. However, light orange points have small values ​​and are located inside the point cloud. They have weak representation ability and have limited effect on feature extraction. Fig.12 As shown in the figure, the numerical values ​​of all points in the aggregated features of each level in the statistical data set are counted, and the unimportant points with small values ​​are pruned by setting the threshold of each level. Therefore, the convolution calculation of these pruned points is skipped in the calculation of the next level. At the same time, the size of the threshold is adjusted so that the gas can achieve the desired balance between accuracy, hardware energy efficiency and latency according to the actual application scenario. Fig.13 As shown in Figure 2, during the entire calculation process of the point cloud neural network, the pruning operation ( Fig.13 The red box in the figure prunes the aggregated features of the previous level to obtain features with reduced size and inputs them to the next level.

[0109] Further references Fig.14 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a point cloud analysis device based on data reuse and calculation skipping, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0110] The embodiment of the present application provides a point cloud analysis device based on data reuse and calculation skipping, including:

[0111] A data acquisition module 1 is configured to acquire point cloud data to be analyzed;

[0112] The model building module 2 is configured to build and train a point cloud analysis model based on data reuse and calculation skipping to obtain a trained point cloud analysis model, wherein the point cloud neural network includes a plurality of layers, each of which includes a sampling layer, a combination layer, a multi-layer perceptron unit, an aggregation layer, and a pruning layer, and the multi-layer perceptron unit includes a plurality of multi-layer perceptrons connected in sequence;

[0113] The analysis module 3 is configured to input the point cloud data to be analyzed into the trained point cloud analysis model, input the point cloud data to be analyzed into the first level of the trained point cloud analysis model, output the sampling result through the sampling layer of the first level, determine the discarded points and retained points of the next level of sampling layer based on the sampling result of the current level of sampling layer, establish the index and coordinates of the retained points of each level of sampling layer in sequence and output them as the sampling result in combination with the features of the retained points, reuse the sampling results of the current level of sampling layer in the sampling process of the next level of sampling layer, and The coordinates and features of the retained points in the sampling results of the current level adopt an inter-level data reuse method based on block-level pipeline delayed aggregation in the combination layer, multi-layer perceptron unit and aggregation layer of the current level, and skip the combination operation, convolution operation and aggregation operation of the discarded points of the current level in the previous level to obtain the aggregation features of the current level. The aggregation features of the current level are subjected to dynamic point pruning operation through the pruning layer to obtain the point cloud data output by the current level. The point cloud data output by the current level is used as the point cloud data input to the next level, and the point cloud data output by the last level is used as the analysis result.

[0114] Fig.15 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Fig.15 As shown, the electronic device of this embodiment includes: a processor 1501 and a memory 1502; wherein the memory 1502 is used to store computer-executable instructions; the processor 1501 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0115] Optionally, the memory 1502 may be independent or integrated with the processor 1501 .

[0116] When the memory 1502 is independently provided, the electronic device further includes a bus 1503 for connecting the memory 1502 and the processor 1501 .

[0117] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 1501 executes the computer execution instructions, the above method is implemented.

[0118] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 1501, the above method is implemented.

[0119] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0120] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0121] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0122] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 1501 to perform some steps of the methods of various embodiments of the present application.

[0123] It should be understood that the processor 1501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 1501 may be any conventional processor 1501, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 1501 for execution, or may be executed by a combination of hardware and software modules in the processor 1501.

[0124] The memory 1502 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0125] The bus 1503 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1503 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 1503 in the drawings of the present application is not limited to only one bus 1503 or one type of bus 1503.

[0126] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0127] An exemplary storage medium is coupled to the processor 1501, so that the processor 1501 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 1501. The processor 1501 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 1501 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0128] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud analysis method based on data reuse and calculation skipping, characterized in that: The following steps are involved: Obtain point cloud data to be analyzed; A point cloud analysis model based on data reuse and calculation skipping is constructed and trained to obtain a trained point cloud analysis model, wherein the point cloud neural network includes several layers, each layer includes a sampling layer, a combination layer, a multi-layer perceptron unit, an aggregation layer and a pruning layer, and the multi-layer perceptron unit includes a plurality of multi-layer perceptrons connected in sequence; The point cloud data to be analyzed is input into the trained point cloud analysis model, the point cloud data to be analyzed is input into the first level of the trained point cloud analysis model, the sampling result is output through the sampling layer of the first level, the discarded points and the retained points of the sampling layer of the next level are determined based on the sampling result of the sampling layer of the current level, the index and coordinates of the retained points of the sampling layer of each level are established in sequence and output as the sampling result in combination with the features of the retained points, the sampling result of the sampling layer of the current level is reused in the sampling process of the sampling layer of the next level, and the current level is used as the sampling result. The coordinates and features of the retained points in the sampling results of the layer adopt an inter-layer data reuse method based on block-level pipeline delayed aggregation in the combination layer, multi-layer perceptron unit and the aggregation layer of the current layer, and skip the combination operation, convolution operation and aggregation operation of the discarded points of the current layer in the previous layer to obtain the aggregation features of the current layer. The aggregation features of the current layer are subjected to dynamic point pruning operation by the pruning layer to obtain the point cloud data output by the current layer. The point cloud data output by the current layer is used as the point cloud data input by the next layer, and the point cloud data output by the last layer is used as the analysis result.

2. The point cloud analysis method based on data reuse and calculation skipping according to claim 1, characterized in that: The coordinates and features of the reserved points in the sampling results of the current level are reused in the combination layer, the multi-layer perceptron unit and the aggregation layer of the current level using an inter-level data multiplexing method based on block-level pipeline delay aggregation, specifically including: The coordinates and features of the retained points in the sampling results of the current level are divided into multiple point cloud blocks, and the following pipeline process is repeated for different point cloud blocks at different time steps: Input the coordinates of the reserved points in the point cloud block into the combination layer for combination operation to obtain a neighbor index table; Inputting the features of the retained points in the point cloud block into the multi-layer perceptron unit for convolution operation to obtain an output feature map; Inputting the neighbor index table and the output feature map into the aggregation layer for aggregation operation to obtain the aggregation features of the point cloud block; The aggregate features of all point cloud blocks constitute the aggregate features of the current level.

3. The point cloud analysis method based on data reuse and calculation skipping according to claim 2, characterized in that: In the combined operation, each reserved point of the current level is taken as a center point, and the neighboring points of each center point are searched in the point cloud data input by the sampling layer of the current level by using the neighborhood search process, and a mapping relationship between all the reserved points and their neighboring points in the point cloud data input by the sampling layer of the current level is established to obtain the neighbor index table; In the aggregation operation, the index of the neighboring point in the neighbor index table is used as the address, and the feature maps corresponding to the K neighboring points of the center point are obtained and aggregated in the output feature map to obtain the aggregated features of the point cloud block.

4. The point cloud analysis method based on data reuse and calculation skipping according to claim 3 is characterized in that: Constructing a mask according to the sampling results of the sampling layer of the current level, wherein the retained points in the mask are marked as 1 and the discarded points are marked as 0; Before the combination operation of the combination layer of the first level, a distance set between all points in the point cloud data to be analyzed is calculated and sorted from small to large to form an index sequence of points with distances from the center point from small to large; The mask obtained according to the sampling result of the sampling layer of the current level selectively filters out the index sequence of the current level in the index sequence of the previous level, and the distance in the index sequence of the current level is used in the neighborhood search process in the combination operation of the current level.

5. The point cloud analysis method based on data reuse and calculation skipping according to claim 1, characterized in that: Also includes: Determine, in the currently input point cloud data to be analyzed, a newly added point cloud relative to the last input point cloud data to be analyzed, and determine, in the currently input point cloud data to be analyzed, an associated portion corresponding to the last input point cloud data to be analyzed; In the current level, the calculation results of the newly added point cloud in the sampling layer, combination layer, multi-layer perceptron unit, and aggregation layer are calculated respectively, and combined with the calculation results of the associated parts of the last input point cloud data to be analyzed in the sampling layer, combination layer, multi-layer perceptron unit, and aggregation layer respectively, to obtain the aggregation features of the currently input point cloud data to be analyzed at the current level, and then through the pruning layer, the point cloud data output at the current level is obtained.

6. The point cloud analysis method based on data reuse and calculation skipping according to claim 1, characterized in that: The following process is performed in the pruning layer: S41, sorting the aggregated features of the current level according to the numerical values, and determining the threshold of the current level; S42, setting the values ​​of the aggregated features of the current level that are lower than the threshold of the current level to zero; S43, determine whether the threshold of the current level meets the requirement, if so, obtain the point cloud data output by the current level, otherwise adjust the threshold of the current level, and repeat steps S42-S43 until the threshold of the current level meets the requirement.

7. A point cloud analysis device based on data reuse and calculation skipping, characterized in that: include: A data acquisition module is configured to acquire point cloud data to be analyzed; A model building module is configured to build and train a point cloud analysis model based on data reuse and calculation skipping to obtain a trained point cloud analysis model, wherein the point cloud neural network includes a plurality of layers, each of which includes a sampling layer, a combination layer, a multi-layer perceptron unit, an aggregation layer, and a pruning layer, and the multi-layer perceptron unit includes a plurality of multi-layer perceptrons connected in sequence; The analysis module is configured to input the point cloud data to be analyzed into the trained point cloud analysis model, input the point cloud data to be analyzed into the first level of the trained point cloud analysis model, output the sampling result through the sampling layer of the first level, determine the discarded points and the retained points of the sampling layer of the next level based on the sampling result of the sampling layer of the current level, establish the index and coordinates of the retained points of the sampling layer of each level in sequence and output them as the sampling result in combination with the features of the retained points, reuse the sampling result of the sampling layer of the current level in the sampling process of the sampling layer of the next level The coordinates and features of the retained points in the sampling results of the current level are reused in the combination layer, multi-layer perceptron unit and the aggregation layer of the current level using an inter-level data reuse method based on block-level pipeline delayed aggregation, and the combination operation, convolution operation and aggregation operation of the discarded points of the current level in the previous level are skipped to obtain the aggregation features of the current level. The aggregation features of the current level are subjected to dynamic point pruning operation by the pruning layer to obtain the point cloud data output by the current level. The point cloud data output by the current level is used as the point cloud data input by the next level, and the point cloud data output by the last level is used as the analysis result.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images