A high energy-efficient point cloud neural network inference method and device

Abstract

The application provides a high-energy-efficiency point cloud neural network inference method and device. The method and device implement quantization of a point cloud neural network by relying on a quantization bit width allocation algorithm based on semantic importance prediction. Since the semantic importance prediction is introduced, a higher quantization bit width is reserved for semantic key points, and the quantization bit width of semantic non-key points is as low as possible. The average quantization bit width of the network can be greatly reduced under the premise that the final accuracy of the quantized network does not decrease or decreases very slightly. In addition, the accuracy and computational energy efficiency of the quantized point cloud neural network are evaluated by using a point cloud dataset. In the case that the accuracy and computational energy efficiency of the quantized point cloud neural network do not reach the expected target, the point cloud neural network is re-quantized, and iteration is repeated until the accuracy and computational energy efficiency reach the expected target. The quantized point cloud neural network has high inference energy efficiency.

Description

Technical Field

[0001] This invention relates to the field of model quantization technology, and in particular to a high-efficiency point cloud neural network inference method and apparatus. Background Technology

[0002] With the rapid application of deep learning technology in various fields, including computer vision (CV), natural language processing (NLP), and speech recognition, a large number of deep learning-based network models have emerged. These models share a common characteristic: they are large and complex, suitable for inference on NVIDIA GPUs, but not for use in embedded devices such as mobile phones. However, users typically need to deploy these complex models in low-cost embedded devices, creating a contradiction. To effectively resolve this contradiction, model quantization has emerged. It can compress models with a slight loss of accuracy, making it possible to apply these complex models to embedded terminals such as mobile phones and robots.

[0003] The paper "Qin H, Cai Z, Zhang M, et al. Bipointnet: Binary neural network for point clouds[J]. arXiv preprint arXiv:2010.05501, 2020." discloses a quantization technique for 3D point cloud networks. This technique first proposes a maximum entropy aggregation method to alleviate the feature generalization attenuation caused by network quantization. The authors found that the widespread use of maximum pooling layers in 3D point cloud networks leads to feature homogenization after quantization. To address this, they proposed replacing traditional maximum pooling with a maximum entropy aggregation operator, effectively maintaining the feature generalization ability of the quantized 3D point cloud network. Furthermore, the authors discovered that quantization of 3D point cloud networks causes the numerical range of features to gradually distort with increasing layer depth. Therefore, they proposed a layer-by-layer scale recovery technique to effectively alleviate the numerical distortion caused by network quantization. Verification showed that this technique effectively reduces the quantization bit width of 3D point cloud networks on multiple datasets such as ModelNet40 and S3DIS, thereby effectively improving the inference speed of the network.

[0004] However, this technology has limitations. Firstly, it primarily targets point cloud data generated by computer simulations, which are characterized by small data volume and high point density. In contrast, real-world point cloud data often comes from LiDAR or millimeter-wave radar, characterized by a large number of points and sparse distribution. Secondly, this technology uses the same quantization bit width for all input points, ignoring differences in semantic richness and sensitivity between points, thus limiting further reductions in the network's quantization bit width. Finally, this technology is mainly designed for small networks like PointNet and lacks the ability to extend to large-scale 3D point cloud networks. Summary of the Invention

[0005] To address the issue that existing point cloud quantization techniques are only applicable to small, dense 3D point clouds and small point cloud networks, this invention provides a high-efficiency point cloud neural network inference method and apparatus that utilizes online semantic importance prediction to extract key points in 3D point clouds and assigns different quantization bit widths to key and non-key points, thereby reducing the overall quantization bit width of the point cloud.

[0006] In a first aspect, the present invention provides a high-efficiency point cloud neural network inference method, comprising:

[0007] Step 1: Initialize the average bit width and semantic importance threshold of the point cloud;

[0008] Step 2: Input each point cloud in the point cloud dataset into the quantized point cloud neural network, and infer the semantic segmentation result of each point cloud in the point cloud dataset; the quantization of the point cloud neural network is achieved by a quantization bit width allocation algorithm based on semantic importance prediction; the average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction.

[0009] Step 3: Determine the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation results of each point cloud in the point cloud dataset.

[0010] Step 4: If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjust the average bit width of the point cloud and / or the semantic importance threshold, and return to step 2; otherwise, output the quantized point cloud neural network.

[0011] According to the high-efficiency point cloud neural network inference method provided by the present invention, the point cloud neural network includes multiple sequentially connected three-dimensional submanifold convolutional layers, each three-dimensional submanifold convolutional layer being a layer to be quantized; the quantization operation of the quantized point cloud neural network includes:

[0012] For the first three-dimensional submanifold convolutional layer of the point cloud neural network, the value of each point in the input point cloud of the first three-dimensional submanifold convolutional layer is normalized to a floating-point number between 0 and 1 to obtain the first point cloud;

[0013] According to the coordinates of each point in the first point cloud, the first point cloud is divided into multiple point cloud blocks of fixed size;

[0014] The semantic importance of each point in each point cloud block is predicted by an additional semantic prediction branch network introduced into the point cloud neural network.

[0015] The maximum value of the semantic importance of each point in each point cloud block is taken as the semantic importance of each point cloud block;

[0016] Based on the principle that points located within the same point cloud block share the same quantization bit width, the quantization bit width of each point in the first point cloud is determined according to the semantic importance of each point cloud block and using a quantization bit width allocation algorithm based on semantic importance prediction.

[0017] Using the quantization bit width of each point in the first point cloud, each point in the first point cloud is quantized to obtain the quantization result of the input point cloud of the first three-dimensional submanifold convolutional layer;

[0018] For the Kth three-dimensional submanifold convolutional layer of the point cloud neural network, based on the semantic importance of each point in the input point cloud of the (K-1)th three-dimensional submanifold convolutional layer of the point cloud neural network, the semantic importance transfer algorithm is used to calculate the semantic importance of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer.

[0019] Based on the semantic importance of each point in the input point cloud of the Kth 3D submanifold convolutional layer, a quantization bit width allocation algorithm based on semantic importance prediction is used to determine the quantization bit width of each point in the input point cloud of the Kth 3D submanifold convolutional layer.

[0020] Using the quantization bit width of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer, quantize each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer to obtain the quantization result of the input point cloud of the Kth three-dimensional submanifold convolutional layer;

[0021] Wherein, K≠1, the input point cloud of the Kth 3D submanifold convolutional layer is the inverse quantization result of the convolution output of the (K-1)th 3D submanifold convolutional layer; the convolution output of the (K-1)th 3D submanifold convolutional layer is obtained by substituting the quantization result of the input point cloud of the (K-1)th 3D submanifold convolutional layer into the (K-1)th 3D submanifold convolutional layer for convolution calculation.

[0022] According to the high-efficiency point cloud neural network inference method provided by the present invention, the semantic prediction branch network is a neural network pre-trained using a semantic prediction dataset and a preset semantic prediction loss function;

[0023] Each sample in the semantic prediction dataset consists of a point cloud block and the semantic classification label of each point within it; the semantic prediction branch network is a neural network pre-trained using the semantic prediction dataset and a preset semantic prediction loss function.

[0024] Each sample in the semantic prediction dataset consists of a point cloud block and the semantic classification label of each point within it; the expression of the preset semantic prediction loss function L is as follows:

[0025] ;

[0026] in, For the first point cloud block Semantic classification labels for each point, For the first point cloud block The semantic importance of each point This represents the number of points contained in the point cloud block.

[0027] According to the high-efficiency point cloud neural network inference method provided by the present invention, the quantization bit width allocation algorithm based on semantic importance prediction includes:

[0028] For any point cloud, allocate an 8-bit quantization width to points whose semantic importance exceeds the semantic importance threshold, and allocate... The quantization width of a bit;

[0029] in, This represents the floor function. This represents the average bit width of the point cloud. This represents the proportion of points in the point cloud whose semantic importance exceeds the semantic importance threshold.

[0030] According to the high-efficiency point cloud neural network inference method provided by the present invention, the semantic importance transfer algorithm includes:

[0031] For two consecutive 3D submanifold convolutional layers, any point in the input point cloud of the latter 3D submanifold convolutional layer... semantic importance Calculate using the following formula:

[0032] ;

[0033] in, Let j represent the points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j. Point semantic importance, This represents the total number of points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j.

[0034] According to the high-efficiency point cloud neural network inference method provided by the present invention, if the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjusting the average bit width of the point cloud and / or the semantic importance threshold includes:

[0035] If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold, then the average bit width of the point cloud is increased and / or the semantic importance threshold is decreased.

[0036] If the computational efficiency of the quantized point cloud neural network is lower than the preset computational efficiency, then the average bit width of the point cloud is reduced and / or the semantic importance threshold is increased.

[0037] According to the high-efficiency point cloud neural network inference method provided by the present invention, the step of increasing the average bit width of the point cloud and / or decreasing the semantic importance threshold includes:

[0038] The closed interval formed by the current value of the average bit width of the point cloud and the maximum feasible value is used as the adjustment domain of the average bit width of the point cloud, and the closed interval formed by the current value of the semantic importance threshold and the minimum feasible value is used as the adjustment domain of the semantic importance threshold.

[0039] The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid.

[0040] The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results;

[0041] The reduction of the average bit width of the point cloud and / or the increase of the semantic importance threshold include:

[0042] The closed interval formed by the current value of the average bit width of the point cloud and the minimum feasible value is used as the adjustment domain of the average bit width of the point cloud, and the closed interval formed by the current value of the semantic importance threshold and the maximum feasible value is used as the adjustment domain of the semantic importance threshold.

[0043] The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid.

[0044] The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results;

[0045] The optimal search for the average bit width of the point cloud and the semantic importance threshold on the two-dimensional search grid includes:

[0046] The accuracy and computational efficiency of each grid point of the two-dimensional search grid are simulated; the accuracy and computational efficiency of each grid point of the two-dimensional search grid are the accuracy and computational efficiency of the quantized point cloud neural network when the grid point value of each grid point of the two-dimensional search grid is the value of the average bit width of the point cloud and the semantic importance threshold.

[0047] Based on the accuracy and computational efficiency of each grid point of the two-dimensional search grid, a Pareto front composed of the grid points of the two-dimensional search grid is plotted in a two-dimensional plane.

[0048] Among the grid points that simultaneously satisfy the first and second constraints, the grid point with the highest accuracy is selected as the optimal grid point, and the grid point value of the optimal grid point is used as the optimal search for the average bit width of the point cloud and the semantic importance threshold.

[0049] The first constraint is a grid point located on the Pareto front;

[0050] The second constraint is to satisfy the grid points that have an accuracy higher than a preset accuracy threshold and a computational energy efficiency higher than a preset computational energy efficiency.

[0051] In a second aspect, the present invention provides a high-efficiency point cloud neural network inference device, comprising:

[0052] An initialization unit is used to initialize the average bit width and semantic importance threshold of the point cloud;

[0053] The inference unit is used to input each point cloud in the point cloud dataset into a quantized point cloud neural network and infer the semantic segmentation result of each point cloud in the point cloud dataset; the quantization of the point cloud neural network is achieved by a quantization bit width allocation algorithm based on semantic importance prediction; the average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction.

[0054] The accuracy and computational efficiency determination unit is used to determine the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation results of each point cloud in the point cloud dataset.

[0055] The hyperparameter adjustment unit is used to adjust the average bit width of the point cloud and / or the semantic importance threshold if the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency threshold, and then return to the inference unit; otherwise, it outputs the quantized point cloud neural network.

[0056] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the high-efficiency point cloud neural network inference method as described in the first aspect.

[0057] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the high-efficiency point cloud neural network inference method as described in the first aspect.

[0058] This invention provides a high-efficiency point cloud neural network inference method and apparatus. First, the average bit width and semantic importance threshold of the point cloud are initialized. Then, the point cloud neural network is quantized using a quantization bit width allocation algorithm based on semantic importance prediction. The average bit width and semantic importance threshold are two hyperparameters used in this algorithm. Semantic importance prediction is introduced here to retain a higher quantization bit width for semantically critical points while minimizing the quantization bit width for semantically non-critical points. This significantly reduces the average quantization bit width of the network while ensuring that the final accuracy of the quantized network is not reduced or only slightly reduced. Next, the accuracy and computational efficiency of the quantized point cloud neural network are determined using a point cloud dataset. Finally, if either the accuracy or computational efficiency of the quantized point cloud neural network fails to meet the expected target, the average bit width and semantic importance threshold are adjusted to requantize the point cloud neural network. This process is iterated until both the accuracy and computational efficiency of the quantized point cloud neural network reach the expected target. The final quantized point cloud neural network exhibits high inference efficiency. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0060] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating the high-efficiency point cloud neural network inference method provided by the present invention;

[0062] Figure 2 This is a schematic diagram of the point cloud neural network provided by the present invention;

[0063] Figure 3 This is a schematic diagram of the structure of the high-efficiency point cloud neural network inference device provided by the present invention;

[0064] Figure 4 This is a schematic diagram of the structure of the electronic device for implementing a high-efficiency point cloud neural network inference method provided by the present invention;

[0065] Figure label:

[0066] 410: Processor; 420: Communication interface; 430: Memory; 440: Communication bus. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0068] The following is combined with Figures 1-4 The present invention describes a high-efficiency point cloud neural network inference method and apparatus.

[0069] In a first aspect, the present invention provides a high-efficiency point cloud neural network inference method, such as... Figure 1 As shown, the method includes:

[0070] Step 1: Initialize the average bit width and semantic importance threshold of the point cloud;

[0071] Step 2: Input each point cloud in the point cloud dataset into the quantized point cloud neural network, and infer the semantic segmentation result of each point cloud in the point cloud dataset; the quantization of the point cloud neural network is achieved by a quantization bit width allocation algorithm based on semantic importance prediction; the average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction.

[0072] Step 3: Determine the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation results of each point cloud in the point cloud dataset.

[0073] Step 4: If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjust the average bit width of the point cloud and / or the semantic importance threshold, and return to step 2; otherwise, output the quantized point cloud neural network.

[0074] Specifically, step 1, which initializes the average bit width of the point cloud and the semantic importance threshold, includes:

[0075] An initial value for the average bit width of the point cloud and an initial value for the semantic importance threshold are set manually based on experience and scenario requirements.

[0076] The initial average bit width of the point cloud is equal to the initial value of the average bit width of the point cloud, and the initial semantic importance threshold is equal to the initial value of the semantic importance threshold.

[0077] Specifically, the point cloud neural network quantized in step 2 is obtained by quantizing the point cloud neural network using a quantization bit width allocation algorithm based on semantic importance prediction. Neural network quantization aims to use a smaller quantization bit width when the network input data is transferred between different layers. To this end, this invention designs a quantization bit width allocation algorithm based on semantic importance prediction, introducing an additional semantic prediction branch network to predict the semantic importance of each point in the input point cloud of the point cloud neural network, thereby extracting semantic key points in the input point cloud. Higher quantization bit widths are then allocated to semantic key points, and lower quantization bit widths are allocated to semantic non-key points. Since semantic key points account for a very small proportion in the point cloud, the overall quantization bit width of the point cloud can be effectively reduced. Compared to the point cloud neural network, the introduced semantic prediction branch network only has 3 convolutional layers, so it does not significantly increase the overall computational cost of the network. Because semantic key points still maintain a high quantization bit width, the accuracy of the network does not decrease or decreases only slightly after quantization. When used in conjunction with hardware that supports variable bit width data computation, the inference energy efficiency of the point cloud neural network quantized by this invention can be effectively improved.

[0078] Furthermore, Figure 2 A schematic diagram of the structure of a point cloud neural network is shown in the example. Figure 2 As can be seen, the point cloud neural network contains multiple sequentially connected three-dimensional submanifold convolutional layers and detection boxes; the detection boxes are connected to the last three-dimensional submanifold convolutional layer and are used to determine the semantic segmentation result of the point cloud of the output point cloud neural network based on the output of the last three-dimensional submanifold convolutional layer.

[0079] In quantizing a point cloud neural network, each 3D submanifold convolutional layer is a layer to be quantized; the quantization operation of the point cloud neural network specifically includes:

[0080] For the first three-dimensional submanifold convolutional layer of the point cloud neural network, SA: normalize the value of each point in the input point cloud of the first three-dimensional submanifold convolutional layer to a floating-point number between 0 and 1 to obtain the first point cloud;

[0081] SB: Based on the coordinates of each point in the first point cloud, the first point cloud is divided into multiple point cloud blocks of a fixed size. Generally speaking, the finer the granularity of the blocks, the better the accuracy of the quantized model. However, blocks that are too small will lead to low utilization of computing units during hardware deployment, thereby reducing the computational efficiency of the model. Therefore, it is necessary to select the block size according to the actual application to balance the accuracy of the model and the computational efficiency.

[0082] SC: Predicts the semantic importance of each point in each point cloud block by means of a semantic prediction branch network that is additionally introduced into the point cloud neural network;

[0083] Optionally, this invention designs a dedicated semantic prediction loss function to enable the training of a semantic prediction branch network based on a semantic prediction dataset and the semantic prediction loss function.

[0084] The semantic prediction dataset uses several classic public datasets commonly used in the industry, such as KITTI. The training samples consist of point cloud blocks (point cloud blocks segmented from a frame of point cloud, preferably similar in size to the point cloud blocks segmented in SB) and the semantic classification labels of each point within them. Since a frame of point cloud contains tens of thousands of points, which belong to different objects and are located in different spatial positions, in order to facilitate the prediction of semantic importance, the semantic classification label of each point can be as follows: when it is a semantic foreground point (i.e., containing semantics, a point on an object), the semantic classification label is marked as 1; when it is a semantic background point (i.e., not containing semantics, a point not on an object), the semantic classification label is marked as 0.

[0085] Accordingly, a binary classification loss function is designed so that the trained semantic prediction branch network generates a value between 0 and 1, i.e., I, for each point.

[0086] The binary classification loss function, i.e., the semantic prediction loss function The expression is as follows: ;

[0087] in, For the first point cloud block Semantic classification labels for each point, For the first point cloud block The semantic importance of each point This represents the number of points contained in the point cloud block.

[0088] SD: The maximum value of the semantic importance of each point in each point cloud block is taken as the semantic importance of each point cloud block;

[0089] SC: Based on the principle that points located within the same point cloud block share the same quantization bit width, the quantization bit width of each point in the first point cloud is determined according to the semantic importance of each point cloud block and using a quantization bit width allocation algorithm based on semantic importance prediction.

[0090] That is, before performing the convolution calculation of this layer, a quantization bit width is assigned to each point cloud block based on the semantic importance of each point cloud block predicted in SD (which is also equivalent to assigning a quantization bit width to each point in the point cloud).

[0091] for A point cloud composed of points, with each point having a quantization bit width. satisfy Therefore, the quantization bit width allocation algorithm based on semantic importance prediction is designed as follows:

[0092] For any point cloud, allocate an 8-bit quantization width to points whose semantic importance exceeds the semantic importance threshold, and allocate... The quantization width of a bit;

[0093] in, This represents the floor function. This represents the average bit width of the point cloud. This represents the proportion of points in the point cloud whose semantic importance exceeds the semantic importance threshold.

[0094] It should be noted that in this step, the semantic importance of each point in each point cloud block is calculated as if it were the same as the semantic importance of the entire point cloud block; 8 bits is an empirical bit width that is generally accepted in the industry and will not cause a loss of precision, but it can also be set to other values ​​as needed.

[0095] This step ensures that a high quantization bit width is maintained for semantically critical blocks while the quantization bit width is reduced as much as possible for semantically non-critical blocks.

[0096] SD: Quantize each point in the first point cloud using the quantization bit width of each point in the first point cloud to obtain the quantization result of the input point cloud of the first three-dimensional submanifold convolutional layer;

[0097] Quantization of a point involves converting its floating-point value into a fixed-point value. That is, assuming the floating-point value of a point is... The quantization bit width of the point is Then the number of fixed points is .

[0098] For the Kth three-dimensional submanifold convolutional layer of the point cloud neural network, SI: Based on the semantic importance of each point in the input point cloud of the (K-1)th three-dimensional submanifold convolutional layer of the point cloud neural network, the semantic importance of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer is calculated using the semantic importance transfer algorithm.

[0099] Specifically, in point cloud neural networks, many layers such as pooling layers and convolutions with a stride of 2 alter the distribution of the point cloud, causing a mismatch between the point cloud distribution at the input and the point cloud distribution at the first 3D submanifold convolutional layer. This means the semantic importance of each point in the input point cloud of the first 3D submanifold convolutional layer cannot be used in other 3D submanifold convolutional layers. Re-predicting semantic importance at every 3D submanifold convolutional layer would incur unacceptable computational costs. Therefore, this invention designs a lightweight importance transfer algorithm that modifies the predicted semantic importance at the network input according to the computational characteristics of each layer to adapt to the point cloud distribution at the input of subsequent 3D submanifold convolutional layers.

[0100] Point cloud distribution refers to the number and location of points in a point cloud. Due to network computation, the specific locations and total number of points in the point cloud will experience some perturbation. Since this perturbation is entirely determined by the characteristics of convolutional computation and is predictable, an importance propagation algorithm can be used to propagate importance layer by layer. The importance propagation algorithm is as follows:

[0101] For two consecutive 3D submanifold convolutional layers, any point in the input point cloud of the latter 3D submanifold convolutional layer... semantic importance Calculate using the following formula:

[0102] ;

[0103] in, Let j represent the points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j. Point semantic importance, This represents the total number of points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j.

[0104] The neighborhood space of point j is a fixed value determined by the network, such as a 3x3x3 neighborhood space.

[0105] SII: Based on the semantic importance of each point in the input point cloud of the Kth 3D submanifold convolutional layer, a quantization bit width allocation algorithm based on semantic importance prediction is used to determine the quantization bit width of each point in the input point cloud of the Kth 3D submanifold convolutional layer;

[0106] SIII: Using the quantization bit width of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer, quantize each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer to obtain the quantization result of the input point cloud of the Kth three-dimensional submanifold convolutional layer;

[0107] in, 1. The input point cloud of the Kth three-dimensional submanifold convolutional layer is the inverse quantization result of the convolution output of the (K-1)th three-dimensional submanifold convolutional layer; the convolution output of the (K-1)th three-dimensional submanifold convolutional layer is obtained by substituting the quantization result of the input point cloud of the (K-1)th three-dimensional submanifold convolutional layer into the (K-1)th three-dimensional submanifold convolutional layer for convolution calculation.

[0108] Here, in the convolution calculation, for a point located at coordinates (x, y, z), its output is... As shown below:

[0109] ;

[0110] in, It is the network weight corresponding to the neighborhood space of that point. Located at coordinate point ( ) is the fixed-point number of points.

[0111] Specifically, in step 3, the accuracy of the quantified point cloud neural network, i.e. the accuracy detection result of the semantic segmentation results of all points in the point cloud dataset, is given by official accuracy testing tools, such as the accuracy evaluation code provided by public data websites like KITTI and NuScence.

[0112] The computational efficiency (inference efficiency) of a quantized point cloud neural network is the quotient of the network's computational load and its power consumption. When the network remains constant, the computational load is customizable, determined by the number of layers and the size of the convolutional kernels. Power consumption is obtained using hardware simulation tools. These tools use Deisgn Compiler to simulate the hardware area and Prime Time to simulate the hardware power consumption.

[0113] Neural network inference energy efficiency is a general metric for evaluating network hardware performance, which refers to the network's computation per second divided by the hardware's power consumption. Generally, the computational load of a given network remains constant; reducing the bit width reduces the hardware's power consumption, thus improving the network's inference energy efficiency.

[0114] Specifically, in step 4, if the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then the average bit width of the point cloud and / or the semantic importance threshold are adjusted, specifically including:

[0115] If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold, then the average bit width of the point cloud is increased and / or the semantic importance threshold is decreased.

[0116] If the computational efficiency of the quantized point cloud neural network is lower than the preset computational efficiency, then the average bit width of the point cloud is reduced and / or the semantic importance threshold is increased.

[0117] Furthermore, increasing the average bit width of the point cloud and / or decreasing the semantic importance threshold specifically includes:

[0118] The closed interval formed by the current value of the average bit width of the point cloud and the maximum feasible value is used as the adjustment domain of the average bit width of the point cloud, and the closed interval formed by the current value of the semantic importance threshold and the minimum feasible value is used as the adjustment domain of the semantic importance threshold.

[0119] The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid.

[0120] The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results;

[0121] The step of reducing the average bit width of the point cloud and / or increasing the semantic importance threshold specifically includes: using the closed interval formed by the current value of the average bit width of the point cloud and the minimum feasible value as the adjustment domain of the average bit width of the point cloud, and using the closed interval formed by the current value of the semantic importance threshold and the maximum feasible value as the adjustment domain of the semantic importance threshold.

[0122] The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid.

[0123] The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results;

[0124] The optimal search for the average bit width of the point cloud and the semantic importance threshold on the two-dimensional search grid includes:

[0125] The accuracy and computational efficiency of each grid point of the two-dimensional search grid are simulated; the accuracy and computational efficiency of each grid point of the two-dimensional search grid are the accuracy and computational efficiency of the quantized point cloud neural network when the grid point value of each grid point of the two-dimensional search grid is the value of the average bit width of the point cloud and the semantic importance threshold.

[0126] Based on the accuracy and computational efficiency of each grid point of the two-dimensional search grid, a Pareto front composed of the grid points of the two-dimensional search grid is plotted in a two-dimensional plane.

[0127] Among the grid points that simultaneously satisfy the first and second constraints, the grid point with the highest accuracy is selected as the optimal grid point, and the grid point value of the optimal grid point is used as the optimal search for the average bit width of the point cloud and the semantic importance threshold.

[0128] The first constraint is a grid point located on the Pareto front;

[0129] The second constraint is to satisfy the grid points that have an accuracy higher than a preset accuracy threshold and a computational energy efficiency higher than a preset computational energy efficiency.

[0130] The initial point cloud average bit width and semantic importance threshold have a certain degree of accuracy. Based on the initial point cloud average bit width and semantic importance threshold, a two-dimensional search grid is constructed, which can quickly determine the point cloud average bit width and semantic importance threshold through optimal search, reducing the amount of inference computation.

[0131] This invention provides a high-efficiency point cloud neural network inference method. First, the average bit width and semantic importance threshold of the point cloud are initialized. Then, the point cloud neural network is quantized using a quantization bit width allocation algorithm based on semantic importance prediction. The average bit width and semantic importance threshold are two hyperparameters used in this algorithm. Semantic importance prediction is introduced here to retain a higher quantization bit width for semantically critical points while minimizing the quantization bit width for semantically non-critical points. This significantly reduces the average quantization bit width of the network while ensuring that the final accuracy of the quantized network is not reduced or only slightly reduced. Next, the accuracy and computational efficiency of the quantized point cloud neural network are determined using a point cloud dataset. Finally, if either the accuracy or computational efficiency of the quantized point cloud neural network fails to meet the expected target, the average bit width and semantic importance threshold are adjusted to requantize the point cloud neural network. This process is iterated until both the accuracy and computational efficiency of the quantized point cloud neural network reach the expected target. The final quantized point cloud neural network exhibits high inference efficiency.

[0132] The present invention demonstrates, through quantization experiments on the SECOND network model (Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutional detection (J). Sensors, 2018, 18(10): 3337.) on the KITTI dataset (Geiger A, Lenz P, Stiller C, et al. Vision meetsrobotics: The kitti dataset[J]. The International Journal of Robotics Research, 2013, 32(11): 1231-1237.), that it can quantize 3D point clouds to an average width of 2 bits without significantly altering vehicle detection accuracy. Furthermore, quantization experiments on the NuScenes dataset (Caesar H, Bankiti V, Lang AH, et al. nuscenes:A multimodal dataset for autonomous driving[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2020: 11621-11631.) show that compared to the traditional global uniform quantization scheme, detection accuracy can be improved by 8.84% / 11.4% / 12.52% at 4 / 3 / 2 bits. Taking hardware that supports variable bit-width computation (Yuan Z, Wang J, Yang Y, et al. Asparse-adaptive CNN processor with area / performance balanced N-way set-associate PE arrays assisted by a collision-aware scheduler[C] / / 2019 IEEEAsian Solid-State Circuits Conference (A-SSCC). IEEE, 2019: 61-64.) as an example, the present invention can improve energy efficiency by up to 3.11x (NuScenes).

[0133] Secondly, the high-efficiency point cloud neural network inference device provided by the present invention will be described below. The high-efficiency point cloud neural network inference device described below can be referred to in correspondence with the high-efficiency point cloud neural network inference method described above. Figure 3 A schematic diagram of a high-efficiency point cloud neural network inference device is shown, such as... Figure 3 As shown, the device includes:

[0134] Initialization unit 21 is used to initialize the average bit width of the point cloud and the semantic importance threshold;

[0135] The inference unit 22 is used to input each point cloud in the point cloud dataset into a quantized point cloud neural network and infer the semantic segmentation result of each point cloud in the point cloud dataset; the quantization of the point cloud neural network is achieved by a quantization bit width allocation algorithm based on semantic importance prediction; the average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction.

[0136] The accuracy and computational efficiency determination unit 23 is used to determine the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation result of each point cloud in the point cloud dataset.

[0137] The hyperparameter adjustment unit 24 is used to adjust the average bit width of the point cloud and / or the semantic importance threshold if the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency threshold, and then return to the inference unit; otherwise, it outputs the quantized point cloud neural network.

[0138] Based on the above embodiments, as an optional embodiment, the point cloud neural network includes multiple sequentially connected three-dimensional submanifold convolutional layers, each of which is a layer to be quantized; the quantization operation of the quantized point cloud neural network includes:

[0139] For the first three-dimensional submanifold convolutional layer of the point cloud neural network, the value of each point in the input point cloud of the first three-dimensional submanifold convolutional layer is normalized to a floating-point number between 0 and 1 to obtain the first point cloud;

[0140] According to the coordinates of each point in the first point cloud, the first point cloud is divided into multiple point cloud blocks of fixed size;

[0141] The semantic importance of each point in each point cloud block is predicted by an additional semantic prediction branch network introduced into the point cloud neural network.

[0142] The maximum value of the semantic importance of each point in each point cloud block is taken as the semantic importance of each point cloud block;

[0143] Based on the principle that points located within the same point cloud block share the same quantization bit width, the quantization bit width of each point in the first point cloud is determined according to the semantic importance of each point cloud block and using a quantization bit width allocation algorithm based on semantic importance prediction.

[0144] Using the quantization bit width of each point in the first point cloud, each point in the first point cloud is quantized to obtain the quantization result of the input point cloud of the first three-dimensional submanifold convolutional layer;

[0145] For the Kth three-dimensional submanifold convolutional layer of the point cloud neural network, based on the semantic importance of each point in the input point cloud of the (K-1)th three-dimensional submanifold convolutional layer of the point cloud neural network, the semantic importance transfer algorithm is used to calculate the semantic importance of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer.

[0146] Based on the semantic importance of each point in the input point cloud of the Kth 3D submanifold convolutional layer, a quantization bit width allocation algorithm based on semantic importance prediction is used to determine the quantization bit width of each point in the input point cloud of the Kth 3D submanifold convolutional layer.

[0147] Using the quantization bit width of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer, quantize each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer to obtain the quantization result of the input point cloud of the Kth three-dimensional submanifold convolutional layer;

[0148] in, 1. The input point cloud of the Kth three-dimensional submanifold convolutional layer is the inverse quantization result of the convolution output of the (K-1)th three-dimensional submanifold convolutional layer; the convolution output of the (K-1)th three-dimensional submanifold convolutional layer is obtained by substituting the quantization result of the input point cloud of the (K-1)th three-dimensional submanifold convolutional layer into the (K-1)th three-dimensional submanifold convolutional layer for convolution calculation.

[0149] Based on the above embodiments, as an optional embodiment, the semantic prediction branch network is a neural network pre-trained using a semantic prediction dataset and a preset semantic prediction loss function;

[0150] Each sample in the semantic prediction dataset consists of a point cloud block and the semantic classification label of each point within it; the preset semantic prediction loss function The expression is as follows:

[0151] ;

[0152] in, For the first point cloud block Semantic classification labels for each point, For the first point cloud block The semantic importance of each point This represents the number of points contained in the point cloud block.

[0153] Based on the above embodiments, as an optional embodiment, the quantization bit width allocation algorithm based on semantic importance prediction includes:

[0154] For any point cloud, allocate an 8-bit quantization width to points whose semantic importance exceeds the semantic importance threshold, and allocate... The quantization width of a bit;

[0155] in, This represents the floor function. This represents the average bit width of the point cloud. This represents the proportion of points in the point cloud whose semantic importance exceeds the semantic importance threshold.

[0156] Based on the above embodiments, as an optional embodiment, the semantic importance transfer algorithm includes:

[0157] For two consecutive 3D submanifold convolutional layers, any point in the input point cloud of the latter 3D submanifold convolutional layer... semantic importance Calculate using the following formula:

[0158] ;

[0159] in, Let j represent the points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j. Point semantic importance, This represents the total number of points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j.

[0160] Based on the above embodiments, as an optional embodiment, if the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjusting the average bit width of the point cloud and / or the semantic importance threshold includes:

[0161] If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold, then the average bit width of the point cloud is increased and / or the semantic importance threshold is decreased.

[0162] If the computational efficiency of the quantized point cloud neural network is lower than the preset computational efficiency, then the average bit width of the point cloud is reduced and / or the semantic importance threshold is increased.

[0163] Based on the above embodiments, as an optional embodiment, increasing the average bit width of the point cloud and / or decreasing the semantic importance threshold includes:

[0164] The closed interval formed by the current value of the average bit width of the point cloud and the maximum feasible value is used as the adjustment domain of the average bit width of the point cloud, and the closed interval formed by the current value of the semantic importance threshold and the minimum feasible value is used as the adjustment domain of the semantic importance threshold.

[0165] The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid.

[0166] The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results;

[0167] The reduction of the average bit width of the point cloud and / or the increase of the semantic importance threshold include:

[0168] The closed interval formed by the current value of the average bit width of the point cloud and the minimum feasible value is used as the adjustment domain of the average bit width of the point cloud, and the closed interval formed by the current value of the semantic importance threshold and the maximum feasible value is used as the adjustment domain of the semantic importance threshold.

[0169] The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid.

[0170] The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results;

[0171] The optimal search for the average bit width of the point cloud and the semantic importance threshold on the two-dimensional search grid includes:

[0172] The accuracy and computational efficiency of each grid point of the two-dimensional search grid are simulated; the accuracy and computational efficiency of each grid point of the two-dimensional search grid are the accuracy and computational efficiency of the quantized point cloud neural network when the grid point value of each grid point of the two-dimensional search grid is the value of the average bit width of the point cloud and the semantic importance threshold.

[0173] Based on the accuracy and computational efficiency of each grid point of the two-dimensional search grid, a Pareto front composed of the grid points of the two-dimensional search grid is plotted in a two-dimensional plane.

[0174] Among the grid points that simultaneously satisfy the first and second constraints, the grid point with the highest accuracy is selected as the optimal grid point, and the grid point value of the optimal grid point is used as the optimal search for the average bit width of the point cloud and the semantic importance threshold.

[0175] The first constraint is a grid point located on the Pareto front;

[0176] The second constraint is to satisfy the grid points that have an accuracy higher than a preset accuracy threshold and a computational energy efficiency higher than a preset computational energy efficiency.

[0177] Thirdly, Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. Processor 410 can call logic instructions in memory 430 to execute a high-efficiency point cloud neural network inference method, which includes: Step 1: Initializing the average bit width and semantic importance threshold of the point cloud; Step 2: Inputting each point cloud in the point cloud dataset into a quantized point cloud neural network, and inferring the semantic segmentation result of each point cloud in the point cloud dataset; the quantization of the point cloud neural network is implemented by a quantization bit width allocation algorithm based on semantic importance prediction; the average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction; Step 3: Determining the accuracy and computational efficiency of the quantized point cloud neural network according to the semantic segmentation result of each point cloud in the point cloud dataset; Step 4: If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjusting the average bit width of the point cloud and / or the semantic importance threshold, and returning to Step 2; otherwise, outputting the quantized point cloud neural network.

[0178] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] Fourthly, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the high-efficiency point cloud neural network inference method provided by the above methods. The method includes: Step 1: initializing the average bit width and semantic importance threshold of the point cloud; Step 2: inputting each point cloud in the point cloud dataset into a quantized point cloud neural network, and inferring the semantic segmentation result of each point cloud in the point cloud dataset; the quantization of the point cloud neural network depends on semantic importance. The algorithm for quantizing bit width allocation based on semantic importance prediction is implemented; the average bit width of the point cloud and the semantic importance threshold are hyperparameters of the algorithm for quantizing bit width allocation based on semantic importance prediction; Step 3: Determine the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation result of each point cloud in the point cloud dataset; Step 4: If the accuracy of the quantized point cloud neural network is lower than the preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than the preset computational efficiency, then adjust the average bit width of the point cloud and / or the semantic importance threshold, and return to Step 2; otherwise, output the quantized point cloud neural network.

[0180] Fifthly, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the high-efficiency point cloud neural network inference method provided by the above methods. The method includes: Step 1: Initializing the average bit width and semantic importance threshold of the point cloud; Step 2: Inputting each point cloud in the point cloud dataset into a quantized point cloud neural network, and inferring the semantic segmentation result of each point cloud in the point cloud dataset; the quantization of the point cloud neural network is achieved by a quantization bit width allocation algorithm based on semantic importance prediction; the average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction; Step 3: Determining the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation result of each point cloud in the point cloud dataset; Step 4: If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjusting the average bit width of the point cloud and / or the semantic importance threshold, and returning to Step 2; otherwise, outputting the quantized point cloud neural network.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-efficiency point cloud neural network inference method, characterized in that, include: Step 1: Initialize the average bit width and semantic importance threshold of the point cloud; Step 2; Each point cloud in the point cloud dataset is input into a quantized point cloud neural network, and semantic segmentation results are obtained for each point cloud in the dataset. The quantization of the point cloud neural network is achieved by predicting the semantic importance of point cloud blocks and allocating quantization bit widths through a semantic prediction branch network. The average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction. A point cloud composed of points, with each point having a quantization bit width. satisfy ; For any point cloud, allocate an 8-bit quantization width to points whose semantic importance exceeds the semantic importance threshold, and allocate... The quantization width of a bit; where, This represents the floor function. This represents the average bit width of the point cloud. This represents the proportion of points in any given point cloud whose semantic importance exceeds the semantic importance threshold. Step 3: Determine the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation results of each point cloud in the point cloud dataset. Step 4: If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjust the average bit width of the point cloud and / or the semantic importance threshold, and return to step 2; otherwise, output the quantized point cloud neural network.

2. The high-efficiency point cloud neural network inference method according to claim 1, characterized in that, The point cloud neural network comprises multiple sequentially connected three-dimensional submanifold convolutional layers, each of which is a layer to be quantized; the quantization operation of the quantized point cloud neural network includes: For the first three-dimensional submanifold convolutional layer of the point cloud neural network, the value of each point in the input point cloud of the first three-dimensional submanifold convolutional layer is normalized to a floating-point number between 0 and 1 to obtain the first point cloud; According to the coordinates of each point in the first point cloud, the first point cloud is divided into multiple point cloud blocks of fixed size; The semantic importance of each point in each point cloud block is predicted by an additional semantic prediction branch network introduced into the point cloud neural network. The maximum value of the semantic importance of each point in each point cloud block is taken as the semantic importance of each point cloud block; Based on the principle that points located within the same point cloud block share the same quantization bit width, the quantization bit width of each point in the first point cloud is determined according to the semantic importance of each point cloud block and using a quantization bit width allocation algorithm based on semantic importance prediction. Using the quantization bit width of each point in the first point cloud, each point in the first point cloud is quantized to obtain the quantization result of the input point cloud of the first three-dimensional submanifold convolutional layer; For the Kth three-dimensional submanifold convolutional layer of the point cloud neural network, based on the semantic importance of each point in the input point cloud of the (K-1)th three-dimensional submanifold convolutional layer of the point cloud neural network, the semantic importance transfer algorithm is used to calculate the semantic importance of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer. Based on the semantic importance of each point in the input point cloud of the Kth 3D submanifold convolutional layer, a quantization bit width allocation algorithm based on semantic importance prediction is used to determine the quantization bit width of each point in the input point cloud of the Kth 3D submanifold convolutional layer. Using the quantization bit width of each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer, quantize each point in the input point cloud of the Kth three-dimensional submanifold convolutional layer to obtain the quantization result of the input point cloud of the Kth three-dimensional submanifold convolutional layer; in, 1. The input point cloud of the Kth three-dimensional submanifold convolutional layer is the inverse quantization result of the convolution output of the (K-1)th three-dimensional submanifold convolutional layer; the convolution output of the (K-1)th three-dimensional submanifold convolutional layer is obtained by substituting the quantization result of the input point cloud of the (K-1)th three-dimensional submanifold convolutional layer into the (K-1)th three-dimensional submanifold convolutional layer for convolution calculation.

3. The high-efficiency point cloud neural network inference method according to claim 2, characterized in that, The semantic prediction branch network is a neural network that has been pre-trained using a semantic prediction dataset and a preset semantic prediction loss function. Each sample in the semantic prediction dataset consists of a point cloud block and the semantic classification label of each point within it; the preset semantic prediction loss function The expression is as follows: ； in, For the first point cloud block Semantic classification labels for each point, For the first point cloud block The semantic importance of each point This represents the number of points contained in the point cloud block.

4. The high-efficiency point cloud neural network inference method according to claim 2, characterized in that, The semantic importance transfer algorithm includes: For two consecutive 3D submanifold convolutional layers, any point in the input point cloud of the latter 3D submanifold convolutional layer... semantic importance Calculate using the following formula: ； in, Let j represent the points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j. Point semantic importance, This represents the total number of points in the input point cloud of the preceding 3D submanifold convolutional layer that are within the neighborhood space of point j.

5. The high-efficiency point cloud neural network inference method according to any one of claims 1 to 3, characterized in that, If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency, then adjusting the average bit width of the point cloud and / or the semantic importance threshold includes: If the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold, then the average bit width of the point cloud is increased and / or the semantic importance threshold is decreased. If the computational efficiency of the quantized point cloud neural network is lower than the preset computational efficiency, then the average bit width of the point cloud is reduced and / or the semantic importance threshold is increased.

6. The high-efficiency point cloud neural network inference method according to claim 5, characterized in that, Increasing the average bit width of the point cloud and / or decreasing the semantic importance threshold includes: The closed interval formed by the current value of the average bit width of the point cloud and the maximum feasible value is used as the adjustment domain of the average bit width of the point cloud, and the closed interval formed by the current value of the semantic importance threshold and the minimum feasible value is used as the adjustment domain of the semantic importance threshold. The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid. The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results; The reduction of the average bit width of the point cloud and / or the increase of the semantic importance threshold include: The closed interval formed by the current value of the average bit width of the point cloud and the minimum feasible value is used as the adjustment domain of the average bit width of the point cloud, and the closed interval formed by the current value of the semantic importance threshold and the maximum feasible value is used as the adjustment domain of the semantic importance threshold. The adjustment domain of the average bit width of the point cloud is divided with a step size of 1, and the adjustment domain of the semantic importance threshold is divided with a step size of 0.1 to generate a two-dimensional search grid. The optimal search for the average bit width of the point cloud and the semantic importance threshold is performed on the two-dimensional search grid, and the average bit width of the point cloud and / or the semantic importance threshold are adjusted based on the optimal search results; The optimal search for the average bit width of the point cloud and the semantic importance threshold on the two-dimensional search grid includes: The accuracy and computational efficiency of each grid point of the two-dimensional search grid are simulated; the accuracy and computational efficiency of each grid point of the two-dimensional search grid are the accuracy and computational efficiency of the quantized point cloud neural network when the grid point value of each grid point of the two-dimensional search grid is the value of the average bit width of the point cloud and the semantic importance threshold. Based on the accuracy and computational efficiency of each grid point of the two-dimensional search grid, a Pareto front composed of the grid points of the two-dimensional search grid is plotted in a two-dimensional plane. Among the grid points that simultaneously satisfy the first and second constraints, the grid point with the highest accuracy is selected as the optimal grid point, and the grid point value of the optimal grid point is used as the optimal search for the average bit width of the point cloud and the semantic importance threshold. The first constraint is a grid point located on the Pareto front; The second constraint is to satisfy the grid points that have an accuracy higher than a preset accuracy threshold and a computational energy efficiency higher than a preset computational energy efficiency.

7. A high-efficiency point cloud neural network inference device, characterized in that, include: An initialization unit is used to initialize the average bit width and semantic importance threshold of the point cloud; The inference unit is used to input each point cloud in the point cloud dataset into a quantized point cloud neural network and infer the semantic segmentation result for each point cloud in the point cloud dataset. The quantization of the point cloud neural network is achieved by predicting the semantic importance of point cloud blocks and allocating quantization bit widths through a semantic prediction branch network. The average bit width of the point cloud and the semantic importance threshold are hyperparameters of the quantization bit width allocation algorithm based on semantic importance prediction. A point cloud composed of points, with each point having a quantization bit width. satisfy ; For any point cloud, allocate an 8-bit quantization width to points whose semantic importance exceeds the semantic importance threshold, and allocate... The quantization width of a bit; where, This represents the floor function. This represents the average bit width of the point cloud. This represents the proportion of points in any given point cloud whose semantic importance exceeds the semantic importance threshold. The accuracy and computational efficiency determination unit is used to determine the accuracy and computational efficiency of the quantized point cloud neural network based on the semantic segmentation results of each point cloud in the point cloud dataset. The hyperparameter adjustment unit is used to adjust the average bit width of the point cloud and / or the semantic importance threshold if the accuracy of the quantized point cloud neural network is lower than a preset accuracy threshold or the computational efficiency of the quantized point cloud neural network is lower than a preset computational efficiency threshold, and then return to the inference unit; otherwise, it outputs the quantized point cloud neural network.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the high-efficiency point cloud neural network inference method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the high-efficiency point cloud neural network inference method as described in any one of claims 1 to 6.

Images