A neural network-based point cloud data processing method and system

Abstract

The application provides a neural network-based point cloud data processing method and system, which can improve the processing speed of point cloud data. The method comprises the following steps: obtaining a weight matrix of a neural network; determining the size of an index matrix according to the size of the weight matrix; determining the content of the index matrix according to the position of at least one zero-activation bit in the point cloud data to be processed and the position of at least one non-zero-activation bit in the point cloud data, wherein each index value included in the index matrix is used to represent whether the value of the bit at the corresponding position in the point cloud data is zero; rearranging the at least one non-zero-activation bit according to the position of the at least one non-zero-activation bit and the size of the weight matrix to determine an activation value matrix; determining at least one matched group according to the index matrix, the activation value matrix and the weight matrix, wherein each matched group comprises a non-zero-activation bit and a weight value; and performing convolution operation on the at least one matched group through the neural network to obtain the processing result of the point cloud data.

Description

[0001] This application claims priority to Chinese Patent Application No. 202210869441.X, filed on July 21, 2022, entitled "A Hardware Accelerator for a Neural Network Based on Submanifold Sparse Convolution", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of machine vision, specifically to a method and system for processing point cloud data based on neural networks. Background Technology

[0003] Point cloud data refers to a set of vectors in a three-dimensional coordinate system, typically including three-dimensional information. Point cloud data is widely used in the field of deep learning, and deep learning-based point cloud data processing methods play an important role in various vision tasks such as autonomous driving, virtual reality, and augmented reality.

[0004] Due to the sparsity of point cloud data, when processing point cloud data using neural networks (such as submanifold sparse convolutional neural networks), convolution calculations are limited to regions in the point cloud data where the center point is a non-zero activation value (referred to as a non-zero activation point). Within the neighborhood of each non-zero activation value at the center point, corresponding pairing groups are determined for subsequent convolution calculations. To clarify, pairing groups are introduced using two-dimensional data as an example; the pairing groups for three-dimensional point cloud data can be deduced analogously from the description below. Figure 1 As shown, non-zero activation sites in the input data are represented by gray-filled squares, and zero-value activation sites are represented by white-filled squares. The weight matrix is ​​a 3*3 matrix, including W0 to W8. Figure 1 In the corresponding example, the central site is a neighborhood containing non-zero activation sites A3 and A4. The nine sites within this neighborhood have indices X0 to X8, with A3 having a weight of W0 and A4 having a weight of W4. Therefore, the pairings corresponding to the central site A4 include (X0, W0) and (X4, W4). Figure 1 The subscript "4" in (X0, W0) and (X4, W4) indicates the pairing group number to which these two pairs belong.

[0005] Currently, in the process of processing point cloud data, it is usually necessary to use general computing platforms such as central processing units (CPUs) and graphics processing units (GPUs) to sequentially determine the pairing groups of each point cloud data. Then, the computing platform processes the pairing groups by running neural networks to determine the processing result of the point cloud data.

[0006] However, since point cloud data includes three-dimensional information, it has a higher dimension than the two-dimensional image data that computing platforms usually process. Moreover, point cloud data is sparse, resulting in a large amount of data. General computing platforms often suffer from high power consumption and slow processing speed when processing point cloud data. Summary of the Invention

[0007] This application provides a method and system for processing point cloud data based on neural networks, which can improve the processing speed of point cloud data.

[0008] Firstly, this application provides a method for processing point cloud data based on neural networks, including:

[0009] Obtain the weight matrix of the neural network, which includes at least one weight value;

[0010] The size of the index matrix is ​​determined based on the size of the weight matrix.

[0011] Based on the position of at least one zero-value activation site in the point cloud data to be processed, and the position of at least one non-zero activation site in the point cloud data, the contents of the index matrix are determined. The index matrix includes at least one index value, and each index value is used to indicate whether the value of the site at the corresponding position in the point cloud data is zero.

[0012] Based on the location of at least one non-zero activation site and the size of the weight matrix, rearrange at least one non-zero activation site to determine the activation value matrix;

[0013] At least one pairing group is determined based on the index matrix, activation value matrix and weight matrix. Each pairing group includes at least one pair. Each pair includes a non-zero activation site in the activation value matrix and a weight value in the weight matrix corresponding to the non-zero activation site.

[0014] The point cloud data is processed by performing convolution operations on at least one paired group using a neural network.

[0015] In one example, at least one index value includes a first index value and a second index value. The first index value indicates that the value of the corresponding point in the point cloud data is zero, and the second index value indicates that the value of the corresponding point in the point cloud data is not zero. At least one pairing group is determined based on the index matrix, the activation value matrix, and the weight matrix, including:

[0016] Based on the size of the weight matrix, determine the size of the extraction region in the index matrix used to extract each pair of at least one pair;

[0017] The positions of at least one extraction region are determined sequentially in the index matrix from the first to the last column, wherein each determined extraction region is the target extraction region;

[0018] After determining the location of the target extraction region each time, the following processing is performed:

[0019] If the index value of the center position of the target extraction region is the second index value, then the pairing group corresponding to the position in the point cloud data corresponding to the center position is determined according to the index value of the target extraction region.

[0020] If the index value of the center position of the target extraction region is the first index value, the process ends.

[0021] In one example, based on the index value of the target extraction region, the pairing groups corresponding to the positions in the point cloud data corresponding to the center position are determined, including:

[0022] By traversing the values ​​of all columns in the target extraction area, at least one pairing is determined for each column in the point cloud data.

[0023] Based on at least one pairing corresponding to each column in the point cloud data, determine the pairing group corresponding to the position corresponding to the center position in the point cloud data.

[0024] During the process of iterating through the values ​​in each column:

[0025] The first state value is determined based on the number of the second index values ​​in the current column of the target extraction region;

[0026] Based on the number of new second index values ​​in the same column of the current column compared to the previous extraction region, and the second status value of the previous extraction region in the same column, determine the second status value corresponding to the current column in the target extraction region;

[0027] Based on the first and second state values ​​of the current column in the target extraction region, determine the non-zero activation sites in the point cloud data corresponding to the current column from the activation value matrix;

[0028] Based on the index value of the current column in the target extraction region, determine the weight value of the position corresponding to the current column from the weight matrix;

[0029] Based on the non-zero activation sites and corresponding weight values ​​of the current column, determine at least one pair in the point cloud data that corresponds to the current column.

[0030] In one example, the method also includes:

[0031] Store each index value according to the number of rows and columns of each index value in the index matrix;

[0032] Store each activation value based on the number of rows and columns of each non-zero activation site in the activation value matrix;

[0033] Store each weight value according to the number of rows and columns of each weight value in the weight matrix.

[0034] In one example, based on the first and second state values ​​of the current column in the target extraction region, non-zero activation points corresponding to the current column in the point cloud data are determined from the activation value matrix. Based on the index value of the current column in the target extraction region, weight values ​​corresponding to the current column are determined from the weight matrix, including:

[0035] Based on the first and second state values ​​of the current column in the target extraction region, determine the highest and lowest bit addresses of the non-zero activation sites in the point cloud data corresponding to the current column.

[0036] Based on the highest and lowest bit addresses, extract the non-zero activation points in the point cloud data corresponding to the current column from the activation value matrix;

[0037] Based on the row and column number of the index value of the current column in the target extraction region, determine the address where the weight value of the position corresponding to the current column is stored;

[0038] Based on the address where the weight value is stored, extract the weight value of the position corresponding to the current column from the weight matrix.

[0039] In one example, the method further includes, before determining the non-zero activation sites in the point cloud data corresponding to the current column from the activation value matrix:

[0040] Determine the location of the next target extraction region, and determine whether the index value of the center location of the next target extraction region is the second index value.

[0041] In one example, before determining at least one pair in the point cloud data corresponding to the current column, the method further includes:

[0042] If the index value of the center position of the next target extraction region is the second index value, before determining the non-zero activation sites in the point cloud data corresponding to the current column from the activation value matrix...

[0043] Determine the location of the next target extraction region and determine whether the index value of the center position of the next target extraction region is the second index value.

[0044] In one example, a neural network is used to perform convolution operations on at least one paired group to obtain the processing results of point cloud data, including:

[0045] By traversing at least one pair of pairs, the convolution operation result of each pair of pairs is obtained;

[0046] Based on the convolution operation results of each pair of groups, obtain the processing results of the point cloud data;

[0047] During the traversal of each pair:

[0048] Obtain each pair in the pairing group. The non-zero activation sites of each pair include data from n+1 input channels, and the weight values ​​of each pair include data from n+1 input channels and m+1 output channels, where n and m are both integers greater than or equal to 0.

[0049] Each pair of data is input into the hardware that performs neural network computation, so that the hardware can perform convolution operations on n+1 input channels and m+1 output channels simultaneously to obtain the convolution operation result corresponding to each pair.

[0050] Based on the convolution operation result corresponding to each pair, obtain the convolution operation result of the paired group.

[0051] In one example, the method also includes the following steps before determining the contents of the index matrix:

[0052] The raw point cloud data is divided into multiple blocks;

[0053] Determine whether each block contains non-zero activation sites;

[0054] If it exists, determine that the block is point cloud data.

[0055] Secondly, a point cloud data processing system based on neural networks is provided, including a data preprocessing module and an accelerator:

[0056] The data preprocessing module is used to obtain the weight matrix of the neural network, which includes at least one weight value;

[0057] The data preprocessing module is also used to determine the size of the index matrix based on the size of the weight matrix;

[0058] The data preprocessing module is also used to determine the contents of the index matrix based on the position of at least one zero-value activation site in the point cloud data to be processed and the position of at least one non-zero activation site in the point cloud data. The index matrix includes at least one index value, and each index value is used to indicate whether the value of the site at the corresponding position in the point cloud data is zero.

[0059] The data preprocessing module is also used to rearrange at least one non-zero activation site according to the position of at least one non-zero activation site and the size of the weight matrix to determine the activation value matrix.

[0060] The accelerator is used to determine at least one pairing group based on an index matrix, an activation value matrix, and a weight matrix. Each pairing group includes at least one pairing, and each pairing includes a non-zero activation site in the activation value matrix and a weight value in the weight matrix corresponding to the non-zero activation site.

[0061] The accelerator is also used to perform convolution operations on at least one pair of data using a neural network to obtain the processing results of point cloud data.

[0062] Due to the sparsity of point cloud data, neural networks perform convolution operations only in activation regions where the center point is a non-zero activation point. Therefore, after acquiring and storing point cloud data, before performing convolution operations on the activation regions using a neural network, it is currently necessary to determine the positions of the non-zero activation points within those regions. Then, based on the position of each non-zero activation point and the weight matrix, a pairing group of activation regions is determined, and convolution operations are performed on the paired groups. For example, when running a submanifold sparse convolutional neural network on a CPU or GPU to process point cloud data, a hash table is used to store the non-zero activation points and their corresponding position information. Determining a pairing group requires finding the corresponding position information based on the non-zero activation points, and then pairing the non-zero activation points with the weights based on the position information. This process is cumbersome and slow, resulting in slow overall processing speed and high power consumption when running a submanifold sparse convolutional neural network on a CPU or GPU. Based on this, the above embodiments of this application determine the size of the index matrix by adjusting the size of the weight matrix, ensuring that the position of each index value corresponds to the position in the weight matrix. Furthermore, the content of the index matrix is ​​further determined by encoding based on the sparsity of the point cloud data. The activation value matrix, determined by rearranging at least one non-zero activation site, significantly saves storage space by discarding zero-value activation sites. Furthermore, since the size of the activation value matrix is ​​related to the size of the weight matrix, and the size of the index matrix is ​​also related to the size of the weight matrix, the position of each value in the activation and weight matrices can be correlated with the corresponding index value in the index matrix. Therefore, by using the index matrix, pairing groups can be determined from the activation and weight matrices in an orderly, efficient, and accurate manner, improving the speed of neural network processing of point cloud data and reducing the consumption of required hardware resources and processing time. Attached Figure Description

[0063] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of a pairing group of point cloud data provided in an exemplary embodiment of this application;

[0065] Figure 2 This is a schematic flowchart of a point cloud data processing method based on a neural network provided in an exemplary embodiment of this application;

[0066] Figure 3 This is a schematic diagram of an index matrix, weight matrix, and activation value matrix corresponding to an example point cloud data provided in an exemplary embodiment of this application;

[0067] Figure 4 This is an exemplary flowchart illustrating point cloud data pairing provided in an exemplary embodiment of this application;

[0068] Figure 5 This is a schematic diagram of an index matrix provided in an exemplary embodiment of this application;

[0069] Figure 6 This is a schematic flowchart illustrating an example of determining the weight values ​​of a weight matrix in a pairing group, provided by an exemplary embodiment of this application.

[0070] Figure 7 This is a schematic diagram of a point cloud data processing flow provided in an exemplary embodiment of this application;

[0071] Figure 8 This is a schematic diagram of a storage pairing group processing flow provided by an exemplary embodiment of this application;

[0072] Figure 9 This is another example of a point cloud data processing flow diagram provided by an exemplary embodiment of this application;

[0073] Figure 10 This is a schematic diagram of an example of a point cloud data processing system based on a neural network provided in an exemplary embodiment of this application;

[0074] Figure 11 This is a schematic diagram of an accelerator provided in an exemplary embodiment of this application;

[0075] Figure 12 This is another example of an accelerator schematic diagram provided in an exemplary embodiment of this application;

[0076] Figure 13 This is another example of a point cloud data processing flow diagram provided by an exemplary embodiment of this application;

[0077] Figure 14 This is another example of an accelerator schematic diagram provided in an exemplary embodiment of this application;

[0078] Figure 15 This is a schematic diagram of an example computing module provided in an exemplary embodiment of this application. Detailed Implementation

[0079] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0080] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0081] This application provides a method for processing point cloud data based on neural networks. This method can be applied to various types of electronic devices.

[0082] For example, this method can be applied to a three-dimensional time-of-flight camera, and the point cloud data acquired by the camera can be processed based on the method provided in this application, thereby quickly obtaining the point cloud data processing results.

[0083] In another example, the electronic device may be a terminal device (such as a computer) connected to a three-dimensional time-of-flight camera, and after receiving the point cloud data acquired by the camera, it processes the point cloud data based on the method provided in this application to efficiently obtain the processing results of the point cloud data.

[0084] Figure 2 This is a schematic flowchart of a method for processing point cloud data based on a neural network, provided in an exemplary embodiment of this application.

[0085] like Figure 2 As shown, the method for processing point cloud data based on neural networks includes the following steps:

[0086] S110, Obtain the weight matrix of the neural network, the weight matrix includes at least one weight value.

[0087] For example, such as Figure 1As shown, the weight matrix includes nine weight values, W0 to W8. It should be understood that, for clarity, Figure 1 The weight matrix is ​​exemplified by a two-dimensional matrix with a size of 3*3. The weight matrix of this application can also be a three-dimensional matrix; for example, the three-dimensional weight matrix includes 27 weight values ​​and has a size of 3*3*3.

[0088] For example, the neural network is a submanifold sparse convolutional neural network, and the weight matrix is ​​a pre-trained parameter matrix of the submanifold sparse convolutional neural network. The neural network can process point cloud data; this application does not limit the type of neural network.

[0089] S120, determine the size of the index matrix based on the size of the weight matrix.

[0090] For example, determining the size of the index matrix based on the size of the weight matrix includes:

[0091] Use the number of columns in the weight matrix as the number of columns in the index matrix;

[0092] Use the depth of the weight matrix as the depth of the index matrix;

[0093] The number of rows in the index matrix is ​​determined based on the number of rows in the point cloud data and the step size used to traverse the point cloud data when determining the content of the index matrix.

[0094] like Figure 3 As shown, determining the size of the index matrix can be divided into the following steps:

[0095] (1) Based on the number of columns of the weight matrix being 3, the number of columns of the index matrix is ​​determined to be 3.

[0096] (2) Based on the depth of the weight matrix being 1, the depth of the index matrix is ​​determined to be 1.

[0097] (3) During the determination of the index matrix content, the first traversal determines the data in rows 1 to 6 of the index matrix based on the data in columns 1 to 3 of the point cloud data. The second traversal determines the data in rows 7 to 12 of the index matrix based on the data in columns 4 to 6 of the point cloud data. The area traversed in the first traversal is shifted to one side by 3 columns to form the area traversed in the second traversal. The step size for traversing the point cloud data can be said to be 3. Since the step size for traversing the point cloud data is 3, the content of the index matrix is ​​determined after two traversals. Furthermore, since each traversal area includes 6 rows of data, the number of rows in the index matrix determined after two traversals is 2 * 6 = 12. It can also be seen that if the step size for traversing the point cloud data is less than 3, the number of rows in the index matrix will be greater than 12.

[0098] It should be understood that if both the weight matrix and the point cloud data are three-dimensional data, then the index matrix is ​​also three-dimensional data. For an example of how to determine the size of the index matrix, please refer to the above-mentioned methods. Figure 3 The explanation will not be repeated here.

[0099] In the example above, the size of the index matrix, determined by the size of the weight matrix and the point cloud data, allows each position of the index matrix to be associated with the corresponding position of the weight matrix and the point cloud data, so that the pairing group can be determined later based on the index matrix.

[0100] S130, determine the contents of the index matrix based on the position of at least one zero-value activation site in the point cloud data to be processed, and the position of at least one non-zero activation site in the point cloud data.

[0101] The index matrix includes at least one index value, each index value indicating whether the value of the corresponding point in the point cloud data is zero.

[0102] For example Figure 3 As shown, in the point cloud data, the gray A1~A5, B1~B5, and C1~C5 are all non-zero activation sites, while the other white squares represent zero-value activation sites.

[0103] First, based on the non-zero activation sites and zero-value activation sites in the first traversal region, determine the index value at the corresponding position between rows 1 and 6 of the index matrix. Here, index value 1 indicates that the point cloud data at the corresponding position is a non-zero activation site, and index value 0 indicates that the point cloud data at the corresponding position is a zero-value activation site.

[0104] Then, based on the non-zero activation sites and zero activation sites in the second traversal region, the index value is determined at the corresponding position between rows 7 and 12 of the index matrix.

[0105] The following examples illustrate the meaning of the corresponding positions: (Combined) Figure 3 It can be seen that the point cloud data and the data in the index matrix are related at corresponding positions. For example, an index value of 1 in the first row and first column of the index matrix indicates that the data in the first row and first column of the point cloud data is a non-zero activation site; an index value of 0 in the first row and second column of the index matrix indicates that the data in the first row and second column of the point cloud data is a zero-value activation site. Therefore, an index value of 1 in the first row and first column can be said to indicate that the value of site A1 at the corresponding position in the point cloud data is not zero. In other words, the index value in the first row and first column of the index matrix corresponds to the first row and first column in the point cloud data.

[0106] It should be understood that, for clarity, the position of point cloud data is represented by rows and columns in this application example. In practical applications, the position of point cloud data can also be represented by coordinates. The coordinates can be converted into rows and columns for subsequent calculations, or the correspondence between each coordinate and the corresponding position in the index matrix can be established directly based on the relationship between the coordinates of each activation point.

[0107] In the example above, since only two different index values ​​are used to indicate whether the value of the corresponding position in the point cloud data is zero, the index matrix occupies less storage resources and can represent the content of each active position in the point cloud data.

[0108] S140, based on the position of at least one non-zero activation site and the size of the weight matrix, rearrange at least one non-zero activation site to determine the activation value matrix.

[0109] In one example, the activation value matrix is ​​determined by rearranging at least one non-zero activation site based on its location and the size of the weight matrix.

[0110] (1) Determine the number of columns in the activation value matrix based on the number of columns in the weight matrix.

[0111] (2) Determine the depth of the activation value matrix based on the depth of the weight matrix.

[0112] (3) To determine the traversal method when determining the content of the index matrix, traverse the point cloud data, using the column number of the weight matrix as the number of columns contained in the region of the point cloud data traversed each time. During each traversal of the point cloud data, determine the content of the activation value matrix in the corresponding column by traversing each column of point cloud data. During the traversal of each column of point cloud data, obtain the non-zero activation sites of the current column, and then determine the row order of the non-zero activation sites of the current column in the activation value matrix according to the row order of the non-zero activation sites of the current column.

[0113] The following examples illustrate this. Figure 3 Explanation of the method for determining the activation value matrix:

[0114] (1) Based on the number of columns of the weight matrix being 3, the number of columns of the activation value matrix is ​​determined to be 3.

[0115] (2) Based on the depth of the weight matrix being 1, the depth of the activation value matrix is ​​determined to be 1.

[0116] (3)a, Based on the traversal step size of 3 corresponding to the index matrix, the traversal step size of the point cloud data is determined to be 3.

[0117] b. First, iterate through the point cloud data from the first to the third column. While iterating through the first column, obtain the non-zero activation points A1, A2, and A3. Based on the row order of A1, A2, and A3, determine their row order in the first column of the activation value matrix: row 1, row 2, row 3. Next, while iterating through the second column, obtain the non-zero activation points B1, B2, and B3. Based on the row order of B1, B2, and B3, determine their row order in the second column of the activation value matrix: row 1, row 2, row 3. Finally, while iterating through the third column, obtain the non-zero activation points C1, C2, and C3. Based on the row order of C1, C2, and C3, determine their row order in the third column of the activation value matrix: row 1, row 2, row 3.

[0118] c. Then, iterate through the point cloud data in columns 4 to 6. Using the same method as step b, determine the row order of A4 and A5 in the first column of the activation value matrix as row 5 and row 6, respectively; determine the row order of B4 and B5 in the second column of the activation value matrix as row 5 and row 6, respectively; and determine the row order of A4 and A5 in the third column of the activation value matrix as row 5 and row 6, respectively.

[0119] In the above example, the activation value size, determined based on the positions of non-zero activation sites and the size of the weight matrix, ensures that each position in the activation value matrix is ​​correlated with the corresponding positions in the weight matrix and the point cloud data, facilitating the subsequent determination of pairing groups based on the activation value matrix. Furthermore, considering that zero-value activation sites have no effect on the processing results of the point cloud data, this application determines the activation value matrix only based on the non-zero activation sites in the point cloud data, ensuring that the activation value matrix does not contain zero-value activation sites, thereby reducing the storage consumption of the activation value matrix.

[0120] S150, determine at least one pairing group based on the index matrix, activation value matrix and weight matrix, each pairing group includes at least one pair, each pair includes a non-zero activation site in the activation value matrix and a weight value in the weight matrix corresponding to the non-zero activation site.

[0121] In one example, at least one index value includes a first index value and a second index value. The first index value indicates that the value of the corresponding point in the point cloud data is zero, and the second index value indicates that the value of the corresponding point in the point cloud data is not zero. At least one pairing group is determined based on the index matrix, the activation value matrix, and the weight matrix, including:

[0122] Based on the size of the weight matrix, determine the size of the extraction region in the index matrix used to extract each pair of at least one pair;

[0123] The positions of at least one extraction region are determined sequentially in the index matrix from the first to the last column, wherein each determined extraction region is the target extraction region;

[0124] After determining the location of the target extraction region each time, the following processing is performed:

[0125] If the index value of the center position of the target extraction region is the second index value, then the pairing group corresponding to the position in the point cloud data corresponding to the center position is determined according to the index value of the target extraction region.

[0126] If the index value of the center position of the target extraction region is the first index value, the process ends.

[0127] The following example illustrates how to obtain the extraction region:

[0128] For example Figure 4 As shown, based on Figure 4 Point cloud data and Figure 3 The weight matrix in the matrix determines the index matrix as follows: Figure 5 As shown. Based on the weight matrix having 3 rows, 3 columns, and a depth of 1, each extraction region is determined to have 3 rows, 3 columns, and a depth of 1. At least one extraction region is sequentially determined on the index matrix in column order from beginning to end. The first extraction region is region 0, which includes the data from the first to third rows of the index matrix. The second extraction region is region 1, which includes the data from the second to fourth rows of the index matrix. The third extraction region is region 2, which includes the data from the third to fifth rows of the index matrix, and so on. The positions of other extraction regions are not detailed here.

[0129] In the example above, if the index value of the center position of the target extraction region is the second index value, it indicates that the point cloud data of the corresponding region is an active region, and it is necessary to further determine the pairing group corresponding to this active region. If the index value of the center position of the target extraction region is the first index value, it indicates that the point cloud data of the corresponding region is an inactive region, and there is no need to calculate the pairing group; this extraction region can be skipped and the process can continue to the next extraction region. By determining the index value of the center position of the target extraction region, it is determined whether to calculate the pairing group. If the pairing group calculation is not required, the current calculation process can be directly terminated, and the next step of the process can be performed, which can improve the processing efficiency of point cloud data.

[0130] In one example, based on the index value of the target extraction region, the pairing groups corresponding to the positions in the point cloud data corresponding to the center position are determined, including:

[0131] By traversing the values ​​of all columns in the target extraction area, at least one pairing is determined for each column in the point cloud data.

[0132] Based on at least one pairing corresponding to each column in the point cloud data, determine the pairing group corresponding to the position corresponding to the center position in the point cloud data.

[0133] During the process of iterating through the values ​​in each column:

[0134] The first state value is determined based on the number of the second index values ​​in the current column of the target extraction region;

[0135] Based on the number of new second index values ​​in the same column of the current column compared to the previous extraction region, and the second status value of the previous extraction region in the same column, determine the second status value corresponding to the current column in the target extraction region;

[0136] Based on the first and second state values ​​of the current column in the target extraction region, determine the non-zero activation sites in the point cloud data corresponding to the current column from the activation value matrix;

[0137] Based on the index value of the current column in the target extraction region, determine the weight value of the position corresponding to the current column from the weight matrix;

[0138] Based on the non-zero activation sites and corresponding weight values ​​of the current column, determine at least one pair in the point cloud data that corresponds to the current column.

[0139] For example, when the target extraction region is the first extraction region, the second state value is the same as the first state value.

[0140] The following is combined with Figure 4 Taking region 1 as an example, this explains how to determine the first and second state values.

[0141] First, iterate through the values ​​in the first column of region 1. The first state value is determined by the number of index values ​​(1) in the first column of region 1. The number of new second index values ​​added to the first column of region 1 compared to region 0 is 1. Therefore, the second state value of region 0 in its first column is 1, and the second state value of the first column of region 1 is 2. This is the sum of the second state value (1) of region 0 in its first column and the number of new second index values ​​(1) added to the first column of region 1 compared to region 0. The state value group corresponding to the first column of region 1 is represented as (second state value, first state value). Therefore, the state value group corresponding to the first column is (2,1). Similarly, the state value group corresponding to the second column is (2,2), and the state value group corresponding to the third column is (3,2).

[0142] Due to the sparsity of point cloud data, neural networks perform convolution operations only in activation regions where the center point is a non-zero activation point. Therefore, after acquiring and storing point cloud data, before performing convolution operations on the activation regions using a neural network, it's necessary to determine the positions of the non-zero activation points within those regions. Then, based on the position of each non-zero activation point and the weight matrix, pairing groups of activation regions are determined, and convolution operations are performed on these paired groups. For example, when running a submanifold sparse convolutional neural network on a CPU or GPU to process point cloud data, a hash table is used to store the non-zero activation points and their corresponding position information. Determining pairing groups requires finding the corresponding position information based on the non-zero activation points, and then pairing the non-zero activation points with their weights based on the position information. This process is cumbersome and slow, resulting in slow overall processing speed and high power consumption when running a submanifold sparse convolutional neural network on a CPU or GPU.

[0143] To address the aforementioned issues, in this embodiment, a first state value and a second state value are determined for each column based on the index value in the target extraction region. This allows for accurate determination of the corresponding activation value from the activation value matrix based on the first and second state values. Similarly, a corresponding weight value is determined from the weight matrix based on the index value in the target extraction region, enabling the pairing of activation values ​​and weight values ​​corresponding to the same index value. Compared to the complex and slow pairing methods currently used when processing point cloud data using submanifold sparse convolutional neural networks on CPUs or GPUs, the above embodiments of this application can accurately and efficiently determine each pair.

[0144] Based on the index matrix, activation value matrix, and weight matrix obtained in the above embodiments, this application also provides a storage method that can store the above matrices in an orderly manner, so as to efficiently and accurately determine the non-zero activation sites and weight values ​​in each pair according to the index matrix. Exemplarily, the method further includes:

[0145] Store each index value according to the number of rows and columns of each index value in the index matrix;

[0146] Store each activation value based on the number of rows and columns of each non-zero activation site in the activation value matrix;

[0147] Store each weight value according to the number of rows and columns of each weight value in the weight matrix.

[0148] Based on the aforementioned ordered storage method for the index matrix, activation value matrix, and weight matrix, this application also provides an example of a method for obtaining non-zero activation sites and weight values ​​in a pair. Exemplarily, based on the first and second state values ​​of the current column in the target extraction region, non-zero activation sites corresponding to the current column in the point cloud data are determined from the activation value matrix; and based on the index value of the current column in the target extraction region, weight values ​​corresponding to the current column are determined from the weight matrix, including:

[0149] Based on the first and second state values ​​of the current column in the target extraction region, determine the highest and lowest bit addresses of the non-zero activation sites in the point cloud data corresponding to the current column.

[0150] Based on the highest and lowest bit addresses, extract the non-zero activation points in the point cloud data corresponding to the current column from the activation value matrix;

[0151] Based on the row and column number of the index value of the current column in the target extraction region, determine the address where the weight value of the position corresponding to the current column is stored;

[0152] Based on the address where the weight value is stored, extract the weight value of the position corresponding to the current column from the weight matrix.

[0153] For example, the lowest bit address of the current column's activation value is represented by (second state value - first state value + 1), and the highest bit address of the current column's activation value is represented by (lowest bit address + first state value - 1).

[0154] Combination Figure 4 Taking region 1 as an example, the pairing of the third column of region 1 will be explained.

[0155] First, based on the state value group (3,2) corresponding to the third column, the lowest address of the activation value in the third column is determined to be 2 (i.e., the address of the second value C1 in the third column), and the highest address is 3 (i.e., the address of the third value C2 in the third column). Then, the activation values ​​corresponding to the lowest and highest addresses in the third column are extracted, namely C1 and C2. Finally, based on the index value of the third column, the corresponding weight values ​​are determined to be W5 and W8. Therefore, the pairings in the third column include (C1, W5) and (C2, W8).

[0156] Similarly, the pairs corresponding to the second column include (B0, W1) and (B1, W4), and the pairs corresponding to the first column include (A1, W6). Therefore, the pairing group B1 corresponding to region 1 includes {(A1, W6), (B0, W1), (B1, W4), (C1, W5), (C2, W8)}.

[0157] Combination Figure 6Taking region 0 as an example, this paper explains how to determine the weight value of the corresponding position based on the index value of region 0.

[0158] First, based on the index value 1 of the first row and first column, the address where the corresponding weight value is stored is determined to be the first row and first column, and the extracted effective weight value is W0. Then, based on the index value 0 of the second column and first column, the address where the corresponding weight value is stored is determined to be the second column and first row, and the extracted weight value is 0 (in...). Figure 6 The blank square in region 0 indicates that the weight value of 0 is invalid and does not participate in the calculation of subsequent pairing groups. Similarly, other valid weight values ​​extracted from region 0 include W4, W7, W2, and W8.

[0159] In the above example, based on the first and second state values ​​of the target extraction region, the highest and lowest bit addresses of the non-zero activation sites in the corresponding columns are determined. These addresses allow for efficient and accurate extraction of the non-zero activation sites included in the pairing group. Based on the position of the index value in the corresponding column of the index value matrix, the weight values ​​included in the pairing group can be extracted efficiently and accurately. Following the storage order of the non-zero activation sites and the weight values, the non-zero activation sites and weight values ​​can be accurately paired to obtain the pairing group.

[0160] In one example, the method further includes, before determining the non-zero activation sites in the point cloud data corresponding to the current column from the activation value matrix:

[0161] Determine the location of the next target extraction region, and determine whether the index value of the center location of the next target extraction region is the second index value.

[0162] To better understand the effect of the above example, such as Figure 7 As shown, the method of "determining the location of the target extraction region and determining whether the index value of the center position of the target extraction region is the second index value" in the above example can be simply referred to as the "judgment" step. The method of "determining the first state value based on the number of second index values ​​in the current column of the target extraction region; determining the second state value corresponding to the current column of the target extraction region based on the number of newly added second index values ​​in the same column compared to the previous extraction region, and the second state value in the same column of the previous extraction region" can be simply referred to as the "generation" step. The method of "determining the non-zero activation sites in the point cloud data corresponding to the current column from the activation value matrix; determining the weight values ​​of the positions corresponding to the current column from the weight matrix" can be simply referred to as the "reading" step.

[0163] For example, such as Figure 7 As shown, in the current target extraction area (i.e. Figure 7While generating region 0 in the middle, the next target region (i.e., Figure 7 The judgment step is performed in region 1).

[0164] In the example above, while the current target extraction region is being generated, the location of the next target extraction region is determined, and it is checked whether the next target extraction region is an active region. If it is an active region, subsequent processing is performed on the next target extraction region; otherwise, the location of the next target extraction region is determined, and its activation status is checked. This pipeline approach synchronizes the different steps between extraction regions, improving data throughput and thus increasing the efficiency of point cloud data processing.

[0165] In one example, before determining the at least one pair in the point cloud data corresponding to the current column, the method further includes:

[0166] If the index value of the center position of the next target extraction region is the second index value, before determining the non-zero activation sites in the point cloud data corresponding to the current column from the activation value matrix.

[0167] Determine the location of the next target extraction region and determine whether the index value of the center position of the next target extraction region is the second index value.

[0168] For example, such as Figure 7 As shown, while the current target extraction region reading step is being performed, a step to generate the next target extraction region is also being performed, and a step to generate the next target extraction region (i.e., the next target extraction region) is then generated. Figure 7 The judgment steps for region 2 in the text.

[0169] In the example above, the different steps between each extraction area are processed synchronously using a pipeline approach, which further improves the data throughput and thus enhances the efficiency of point cloud data processing.

[0170] S160: Perform convolution operations on at least one paired group using a neural network to obtain the processing results of the point cloud data.

[0171] In one example, before performing convolution operations on at least one paired group via a neural network, such as Figure 8As shown, each pair is stored in a first-in / first-out (FIFO) register according to its column position, with pairs from the same column stored in one FIFO. The number of FIFOs used to store each pair group is the same as the number of columns in the weight matrix. In this example, by storing pairs from different columns sequentially in multiple FIFOs, it is possible to quickly and orderly retrieve each pair from a pair group in column order.

[0172] In one example, a neural network is used to perform convolution operations on at least one paired group to obtain the processing results of point cloud data, including:

[0173] By traversing at least one pair of pairs, the convolution operation result of each pair of pairs is obtained;

[0174] Based on the convolution operation results of each pair of groups, obtain the processing results of the point cloud data;

[0175] During the traversal of each pair:

[0176] Obtain each pair in the pairing group. The non-zero activation sites of each pair include data from n+1 input channels, and the weight values ​​of each pair include data from n+1 input channels and m+1 output channels, where n and m are both integers greater than or equal to 0.

[0177] The data of each pair in the pairing group is input into the hardware that performs neural network calculations, so that the hardware performs convolution operations on n+1 input channels and m+1 output channels simultaneously to obtain the convolution operation results corresponding to each pair in the pairing group.

[0178] Based on the convolution operation result corresponding to each pair, obtain the convolution operation result of the paired group.

[0179] For example, based on the convolution operation result corresponding to each pair, the convolution operation result of the paired group is obtained, including:

[0180] The convolution results of each pair in the pairing group are summed to obtain a cumulative sum, which is then used as the convolution result of the pairing group.

[0181] The values ​​of m and n in the above example can be set as needed, and this application does not impose any restrictions on them. For example, n = m = 15.

[0182] The above example improves system throughput and computational efficiency by setting up parallel operations for input and output channels.

[0183] In one example, the method also includes the following steps before determining the contents of the index matrix:

[0184] First, the raw point cloud data is divided into multiple blocks. Then, it is determined whether each block contains non-zero activation sites. If so, the block is identified as point cloud data.

[0185] In the above example, due to the sparsity of point cloud data, the neural network only performs convolution on regions where the center point is a non-zero activation point. Therefore, redundant all-zero regions can be removed in advance before processing point cloud data, which can improve the computational efficiency of the point cloud data processing method in this application.

[0186] like Figure 9 As shown, the purpose is to remove all-zero regions from the raw input data and then process the remaining data. For example, the raw point cloud data is first divided into N×M×L blocks, where N, M, and L are configurable parameters representing the pixel size of the point cloud data blocks. These values ​​can be adjusted based on the actual input size; for the ShapeNet dataset, a feasible value is 8×8×8. Then, for each block, it is checked whether there are non-zero activation sites. If so, the block is considered an active block and retained; otherwise, it is considered a completely sparse block and removed from the raw point cloud data without further processing. The point cloud data optimized using this strategy consists only of active blocks, effectively alleviating the imbalance between data transmission volume and the data used for computation.

[0187] Due to the sparsity of point cloud data, neural networks perform convolution operations only in activation regions where the center point is a non-zero activation point. Therefore, after acquiring and storing point cloud data, before performing convolution operations on the activation regions using a neural network, it is currently necessary to determine the positions of the non-zero activation points within those regions. Then, based on the position of each non-zero activation point and the weight matrix, a pairing group of activation regions is determined, and convolution operations are performed on the paired groups. For example, when running a submanifold sparse convolutional neural network on a CPU or GPU to process point cloud data, a hash table is used to store the non-zero activation points and their corresponding position information. Determining a pairing group requires finding the corresponding position information based on the non-zero activation points, and then pairing the non-zero activation points with the weights based on the position information. This process is cumbersome and slow, resulting in slow overall processing speed and high power consumption when running a submanifold sparse convolutional neural network on a CPU or GPU. Based on this, the above embodiments of this application determine the size of the index matrix by adjusting the size of the weight matrix, ensuring that the position of each index value corresponds to the position in the weight matrix. Furthermore, the content of the index matrix is ​​further determined by encoding based on the sparsity of the point cloud data. The activation value matrix, determined by rearranging at least one non-zero activation site, significantly saves storage space by discarding zero-value activation sites. Furthermore, since the size of the activation value matrix is ​​related to the size of the weight matrix, and the size of the index matrix is ​​also related to the size of the weight matrix, the position of each value in the activation and weight matrices can be correlated with the corresponding index value in the index matrix. Therefore, by using the index matrix, pairing groups can be determined from the activation and weight matrices in an orderly, efficient, and accurate manner, improving the speed of neural network processing of point cloud data and reducing the consumption of required hardware resources and processing time.

[0188] In conjunction with the above embodiments of the point cloud data processing method based on neural networks, this application also provides a point cloud data processing system based on neural networks, including a data preprocessing module and an accelerator for deploying the neural network, such as... Figure 10 As shown;

[0189] The data preprocessing module is used to obtain the weight matrix of the neural network, which includes at least one weight value.

[0190] The data preprocessing module is also used to determine the size of the index matrix based on the size of the weight matrix;

[0191] The data preprocessing module is also used to determine the contents of the index matrix based on the position of at least one zero-value activation site in the point cloud data to be processed and the position of at least one non-zero activation site in the point cloud data. The index matrix includes at least one index value, and each index value is used to indicate whether the value of the site at the corresponding position in the point cloud data is zero.

[0192] The data preprocessing module is also used to rearrange at least one non-zero activation site according to the position of at least one non-zero activation site and the size of the weight matrix to determine the activation value matrix.

[0193] The data preprocessing module is also used to send the index matrix, weight matrix, and activation value matrix to the accelerator;

[0194] The accelerator is used to determine at least one pairing group based on an index matrix, an activation value matrix, and a weight matrix. Each pairing group includes at least one pairing, and each pairing includes a non-zero activation site in the activation value matrix and a weight value in the weight matrix corresponding to the non-zero activation site.

[0195] The accelerator is also used to perform convolution operations on at least one pair of data using a neural network to obtain the processing results of point cloud data.

[0196] For example, such as Figure 11 As shown, the accelerator includes a sparse pairing module and a computation module. The sparse pairing module implements the scheme whereby the accelerator determines at least one pairing group based on an index matrix, an activation value matrix, and a weight matrix. Each pairing group includes at least one pair, and each pair includes a non-zero activation point in the activation value matrix and a weight value in the weight matrix corresponding to the non-zero activation point. The sparse pairing module also transmits each pair of each pairing group to the computation module. The computation module implements the scheme whereby the accelerator performs convolution operations on at least one pairing group using a neural network to obtain the processing result of the point cloud data.

[0197] In one example, the data preprocessing module is further configured to divide the raw point cloud data into multiple blocks, and then determine whether each block contains non-zero activation sites. If so, the block is identified as part of the point cloud data.

[0198] In one example, such as Figure 11 As shown, the accelerator also includes a bus, a main controller, and an output buffer;

[0199] The main controller is used to control the logical order of operation between the various modules on the accelerator;

[0200] The bus is used for communication between the accelerator and modules on off-chip dynamic random access memory (DRAM), including data preprocessing modules.

[0201] The output buffer is used to store the processing results of the point cloud data.

[0202] In one example, such as Figure 11As shown, the sparse pairing module includes an input data buffer, a sparse decoding module, and a pairing data buffer. The input data buffer receives and stores the activation value matrix, weight matrix, and index matrix via a bus.

[0203] The sparse pairing module achieves the following: "Determine at least one pairing group based on the index matrix, activation value matrix, and weight matrix. Each pairing group includes at least one pairing, and each pairing includes a non-zero activation site in the activation value matrix and a weight value in the weight matrix corresponding to the non-zero activation site."

[0204] The sparse decoding submodule determines at least one pairing group based on the index matrix, activation value matrix and weight matrix in the data buffer. After determining each pairing group, it sends the non-zero activation site position and weight value position of each pairing group to the input data buffer.

[0205] The input data buffer reads the non-zero activation sites and weight values ​​of each pair based on the positions of the non-zero activation sites and weight values ​​of each pair, and sends each pair to the pairing data buffer. Accordingly, the pairing data buffer is used to store at least one pairing group.

[0206] In one example, the sparse pairing module transmits each pair of each pairing group to the computation module in the following way:

[0207] The sparse decoding submodule triggers the pairing data buffer to transmit each pair of each pairing group to the computing module. The method for transmitting each pair includes:

[0208] The sparse decoding submodule sends the location information of the pair to be transmitted to the pairing data buffer.

[0209] The pairing data buffer transmits the pairing to the computing module based on the location information of the pair to be transmitted.

[0210] In one example, such as Figure 12 As shown, the input data buffer includes a mask buffer, an activation value buffer, and a weight buffer. The input data buffer receives and stores the activation value matrix, weight matrix, and index matrix via the bus in the following ways:

[0211] The mask buffer receives and stores the index matrix via the bus, the activation value buffer receives and stores the activation value matrix via the bus, and the weight buffer receives and stores the weight matrix via the bus.

[0212] For example, a method for storing an index matrix in a mask cache includes:

[0213] Store each index value according to the number of rows and columns of each index value in the index matrix;

[0214] The methods for storing the activation value matrix in the activation value cache include:

[0215] The activation value buffer stores each activation value based on the number of rows and columns of each non-zero activation point in the activation value matrix;

[0216] Methods for storing the weight matrix in a weight buffer include:

[0217] The weight cache stores each weight value according to the number of rows and columns of each weight value in the weight matrix.

[0218] In one example, such as Figure 12 and Figure 13 As shown, the sparse decoding submodule also includes a mask discriminator, and the sparse pairing module determines the pairing group using the following method:

[0219] The Mask buffer determines the size of the extraction region in the index matrix used to extract each pair of at least one pair of pairs based on the size of the weight matrix.

[0220] The Mask cache determines the position of at least one extraction region sequentially in the index matrix from the beginning to the end along the column direction, wherein each determined extraction region is the target extraction region.

[0221] After determining the location of the target extraction region each time, the following processing is performed:

[0222] The Mask buffer sends the index value contained in the target extraction region to the Mask discriminator;

[0223] The Mask discriminator determines whether the index value of the center position of the target extraction region is the first index value or the second index value;

[0224] If the index value of the center position of the target extraction region is the second index value, the Mask discriminator determines that the region is an active region. The Mask discriminator sends the index values ​​contained in the target extraction region to the pairing group controller. The pairing group controller determines the position information of each pair in the pairing group corresponding to the center position in the point cloud data based on the index value of the target extraction region. The position information of each pair includes the position of the non-zero activation point and the position of the weight value. For the first pair in each pair, the pairing group controller sends the position of the non-zero activation point of the first pair to the activation value buffer and the position of the weight value of the first pair to the weight buffer. The input data buffer reads the non-zero activation point and weight value of each pair based on the position of the non-zero activation point and the position of the weight value of each pair, which is achieved in the following way:

[0225] The activation value buffer reads the non-zero activation sites of the first pair based on their positions;

[0226] The weight buffer reads the weight value of the first pair based on the position of the weight value of the first pair.

[0227] If the index value of the center position of the target extraction region is the first index value, the Mask discriminator determines that the region is an inactive region, the Mask discriminator ends the process, and continues to determine whether the index value of the next target extraction region sent by the Mask buffer is the first index value or the second index value.

[0228] For example, the pairing data buffer includes a FIFO group and a multiplexer (MUX). Taking the first pairing as an example, the method by which the input data buffer sends a pairing to the pairing data buffer is illustrated:

[0229] After the activation value buffer reads the first pair of non-zero activation sites, it sends the first pair of non-zero activation sites to the FIFO group;

[0230] After reading the weight value of the first pair, the weight buffer sends the weight value of the first pair to the FIFO group.

[0231] Taking the first pairing as an example, the pairing data cache stores the pairings in the following ways:

[0232] The FIFO group stores the first pair according to the column corresponding to the first pair.

[0233] Because the number of columns in each pair is fixed and they are stored sequentially in the FIFO group, the FIFO group can determine the FIFO location for each pair. For example... Figure 8 As shown, pairing group B0 is stored in FIFO0 to FIFO2 of the FIFO group. Since each FIFO only stores pairs for the corresponding column, when storing pairs for the corresponding column in the order they are stored, each FIFO iterates through all storage locations based on the maximum number of pairs each column can contain before iterating through the next FIFO to store the next column's pairs. Therefore, each FIFO can grasp the column corresponding to the first pair and store the first pair in column order.

[0234] Furthermore, the pairing group controller sends the location information of the pair to be transmitted to the pairing data buffer. The pairing data buffer then transmits the pairing to the computing module based on the location information. This can be achieved in the following way:

[0235] The pairing group controller sends the column order of the pairs to be transmitted, as well as the order of the pairing groups to which the pairs to be transmitted belong, to the multiplexer;

[0236] The multiplexer sends the pairs to be transmitted, stored in the FIFO group, to the computing module according to the column order of the pairs to be transmitted and the order of the pairing groups in which the pairs to be transmitted belong.

[0237] In one example, such as Figure 14 As shown, the pairing group controller includes a pairing controller, a status index generator, and an address generator. The pairing controller controls the status index generator, the address generator, and the multiplexer to perform operations in a logical order.

[0238] The pairing group controller determines the pairing group corresponding to the center position in the point cloud data based on the index value of the target extraction region. This can be achieved in the following ways:

[0239] The pairing group controller uses a state index generator and an address generator to traverse the values ​​of all columns in the target extraction area and determine at least one pairing corresponding to each column in the point cloud data.

[0240] The pairing group controller uses a state index generator and an address generator to determine the pairing group corresponding to the position corresponding to the center position in the point cloud data, based on at least one pairing corresponding to each column in the point cloud data.

[0241] During the process of iterating through the values ​​in each column:

[0242] The status index generator determines the first status value based on the number of second index values ​​in the current column of the target extraction region, where, Figure 13 In this context, B represents the first state value;

[0243] The status index generator determines the second status value corresponding to the current column in the target extraction region based on the number of new second index values ​​in the same column compared to the previous extraction region, and the second status value in the same column of the previous extraction region. Figure 13 In this context, A represents the second state value;

[0244] The address generator determines the position of the non-zero activation point in the point cloud data corresponding to the current column in the activation value buffer based on the first and second state values ​​of the current column in the target extraction region.

[0245] The address generator determines the position of the weight value of the location corresponding to the current column in the weight cache based on the index value of the current column in the target extraction area;

[0246] The address generator determines at least one pair in the point cloud data corresponding to the current column from the activation value cache and the weight cache, based on the position of the non-zero activation point of the current column and the position of the corresponding weight value.

[0247] In one example, the address generator determines the position of the non-zero activation point in the point cloud data corresponding to the current column in the activation value buffer based on the first and second state values ​​of the current column in the target extraction region; the address generator determines the position of the weight value of the position corresponding to the current column in the weight buffer based on the index value of the current column in the target extraction region, and achieves this in the following way:

[0248] The address generator determines the highest and lowest bit addresses of the non-zero activation sites in the point cloud data corresponding to the current column based on the first and second state values ​​of the current column in the target extraction region.

[0249] The address generator sends the highest and lowest bit addresses to the activation value buffer so that the activation value buffer can extract the non-zero activation points in the point cloud data corresponding to the current column from the activation value matrix and store them in the FIFO group.

[0250] The address generator determines the address where the weight value of the position corresponding to the current column is stored based on the number of rows and columns of the index value of the current column in the target extraction region.

[0251] The address generator sends the address where the weight value is stored to the weight buffer, so that the weight buffer can extract the weight value corresponding to the current column from the weight matrix and store it in the FIFO group.

[0252] For example, the address generator uses (A-B+1) to represent the lowest address of the active value of the current column, and (A) to represent the highest address of the active value of the current column, with the address segment being (A-B+1, A).

[0253] For example, the number of FIFOs is k^2, the number of first units used to determine the state value corresponding to each column is k^2, and the number of second units used to determine the paired address corresponding to each column is k^2.

[0254] For example, the main controller controls the address generator to perform the reading step for the current target area through the pairing controller, while controlling the state index generator to perform the generation step for the next target extraction area, and the main controller controls the mask discriminator to perform the judgment step for the next target extraction area of ​​the next target extraction area.

[0255] In one example, the pairing group controller sends the column order of the pairs to be transmitted, as well as the order of the pairing groups to which the pairs belong, to the multiplexer, which can be achieved in the following way:

[0256] The controller obtains the column order of the pairs to be transmitted and the order of the pairing groups to which the pairs to be transmitted belong through the status index generator and address generator, and sends them to the multiplexer.

[0257] In one example, such as Figure 12 or Figure 15 As shown: The computation module includes a computation array and an accumulator, which inputs the data of each pair into the hardware that performs neural network computation, so that the hardware simultaneously performs convolution operations on n+1 input channels and m+1 output channels to obtain the convolution operation result corresponding to each pair; the implementation method of obtaining the convolution operation result of the paired group based on the convolution operation result corresponding to each pair includes:

[0258] The data for each pair is input into a computation array that performs neural network computations. The computation array performs convolution calculations on the non-zero activation sites and weight values ​​of each pair in parallel with the input and output channels. The accumulator obtains the convolution calculation results of different output channels for each pair in the pairing group, performs an accumulation operation on the convolution calculation results of different pairs in the same output channel in the pairing group to obtain the convolution calculation result corresponding to the pairing group, and inputs it into the output buffer.

[0259] For example, the computation array includes m+1 computation units and m+1 output channels corresponding to the weight values. Each output channel acquires data from n+1 input channels of each paired non-zero activation site. Each computation unit calculates the convolution result of the n+1 input channel data of the weight values ​​and the n+1 input channel data of the non-zero activation sites in each pair, and inputs it into an accumulator. The accumulator performs an accumulation operation on the convolution results of different pairs in the same output channel to obtain the convolution operation result corresponding to the pair, and inputs it into an output buffer so as to obtain the processing result of the point cloud data based on the convolution operation result corresponding to each pair in the output buffer.

[0260] like Figure 15 As shown, the n+1 input channel data with non-zero activation sites include X [0] ~X [n] Calculation unit 0 is used to calculate X [0] ~X [n] With weight W [0][0] ~W [n][0] The convolution result OC[0] (P-sum indicates that the convolution result OC[0] of the current output channel is a part of the cumulative sum corresponding to the current output channel), and the data calculated by other computing units can be found in [reference]. Figure 14 As shown in (b) above. X is calculated in computational unit m. [0] ~X [n] With weight W [0][m] ~W [n][m] In the process of convolution results, such as Figure 14 As shown in (c), obtain X [0] ~X [n]The convolution calculation results of each input channel data and the corresponding weights, and in the form of an addition tree, the convolution calculation results of n + 1 input channels are accumulated pairwise.

[0261] As Figure 12 shown, the calculation module calculates the processing results of the point cloud data through the internal loop of the accumulator. Among them, the process of the internal loop is as Figure 15 shown in (a) in

[0262] for (activation block i; i = i + 1) is used to traverse at least one activation block of the original point cloud data. An activation block is a block with a non-zero activation value at the central site.

[0263] for (region j; j = j + 1) is used to traverse at least one extraction region of each activation block.

[0264] for (pair k in pairing group K) is used to traverse at least one pair in the pairing group K corresponding to the activated extraction region. An activated extraction region is an extraction region with a non-zero activation value at the central site.

[0265] for (N = 0; N < Ni; N = N + n) and for (M = 0; M < Mi; M = M + m) (multiply-accumulate operation) are used to perform an accumulation operation on the convolution calculation results of different pairs in the pairing group K on the same output channel.

[0266] Exemplarily, both m and n are set to 15, and the parallelism reaches 256. The values of the parallelism m and n of the calculation module can be changed, and thus convolution calculations with different parallelisms can be implemented using the same architecture.

[0267] In one example, since a field programmable gate array (FPGA) platform or an application specific integrated circuit (ASIC) platform can be designed according to requirements, the method for processing point cloud data based on a neural network in this application is deployed on an FPGA or an ASIC, so that the obtained accelerator can fully achieve the effects of the method in the embodiments of this application.

[0268] For other implementation manners and effects of the system, refer to the embodiments of the method for processing point cloud data based on a neural network described above, which will not be elaborated here.

[0269] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0270] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0271] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0272] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0273] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0274] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for processing point cloud data based on a neural network, characterized in that, The method comprises the following steps: obtaining a weight matrix of a neural network, the weight matrix comprising at least one weight value; determining the size of an index matrix according to the size of the weight matrix; determining the content of the index matrix according to the position of at least one zero-activated site in the point cloud data to be processed and the position of at least one non-zero-activated site in the point cloud data, the index matrix comprising at least one index value, each index value being used to represent whether the value of the site at the corresponding position in the point cloud data is zero; rearranging at least one non-zero-activated site according to the position of the at least one non-zero-activated site and the size of the weight matrix to determine an activated value matrix; determining at least one matched group according to the index matrix, the activated value matrix and the weight matrix, each matched group comprising at least one pair, and each pair comprising one non-zero-activated site in the activated value matrix and one weight value in the weight matrix corresponding to the non-zero-activated site, wherein the at least one index value comprises a first index value and a second index value, the first index value being used to represent that the value of the site at the corresponding position in the point cloud data is zero, and the second index value being used to represent that the value of the site at the corresponding position in the point cloud data is not zero, and the determining at least one matched group according to the index matrix, the activated value matrix and the weight matrix comprises the following steps: determining the size of an extraction area in the index matrix for extracting each matched group in the at least one matched group according to the size of the weight matrix; determining the position of at least one extraction area in the index matrix in the order from the first to the last in the column direction, wherein the extraction area determined each time is a target extraction area; after determining the position of the target extraction area each time, performing the following processing: if the index value of the center position of the target extraction area is the second index value, determining the matched group corresponding to the position in the point cloud data corresponding to the center position according to the index value of the target extraction area; if the index value of the center position of the target extraction area is the first index value, ending the processing process for the target extraction area; performing convolution operation on the at least one matched group through the neural network to obtain the processing result of the point cloud data.

2. The method of claim 1, wherein, The determining the matched group corresponding to the position in the point cloud data corresponding to the center position according to the index value of the target extraction area comprises the following steps: determining at least one pair corresponding to each column in the point cloud data by traversing the values of all columns in the target extraction area; determining the matched group corresponding to the position in the point cloud data corresponding to the center position according to at least one pair corresponding to each column in the point cloud data; in the process of traversing the values of each column: determining a first state value according to the number of the second index values of the current column in the target extraction area; determining a second state value corresponding to the current column in the target extraction region according to the number of the second index values newly added in the same column of the current column compared to the last extraction region and a second state value in the same column of the last extraction region; determining a non-zero active site of a position corresponding to the current column in the point cloud data from the activation value matrix according to the first state value and the second state value of the current column in the target extraction region; determining a weight value of a position corresponding to the current column from the weight matrix according to the index value of the current column in the target extraction region; determining the at least one pair corresponding to the current column in the point cloud data according to the non-zero active site and the corresponding weight value of the current column.

3. The method of claim 2, wherein, The method further comprises: storing each index value according to the row number and the column number of each index value in the index matrix; storing each non-zero active site according to the row number and the column number of each non-zero active site in the activation value matrix; storing each weight value according to the row number and the column number of each weight value in the weight matrix.

4. The method of claim 3, wherein, The method further comprises: determining a highest bit address and a lowest bit address of the non-zero active site of the position corresponding to the current column in the point cloud data stored according to the first state value and the second state value of the current column in the target extraction region; extracting the non-zero active site of the position corresponding to the current column in the point cloud data from the activation value matrix according to the highest bit address and the lowest bit address; determining an address of the weight value of the position corresponding to the current column stored according to the row number and the column number of the index value of the current column in the target extraction region; extracting the weight value of the position corresponding to the current column from the weight matrix according to the address of the weight value stored.

5. The method according to any one of claims 2-4, characterized in that, Before determining the non-zero active site of the position corresponding to the current column in the point cloud data from the activation value matrix, the method further comprises: determining a position of a next target extraction region and determining whether an index value of a center position of the next target extraction region is a second index value.

6. The method according to any one of claims 2-4, characterized in that, Before determining the at least one pair corresponding to the current column in the point cloud data, the method further comprises: if the index value of the center position of the next target extraction region is the second index value, before determining the non-zero active site of the position corresponding to the current column in the point cloud data from the activation value matrix, determining a position of a next target extraction region of the next target extraction region and determining whether an index value of a center position of the next target extraction region of the next target extraction region is a second index value.

7. The method according to any one of claims 1-4, characterized in that, The convolution operation on the at least one pair group through the neural network obtains a processing result of the point cloud data, and the convolution operation on the at least one pair group through the neural network comprises: obtaining a convolution operation result of each pair group by traversing the at least one pair group; obtaining a processing result of the point cloud data according to the convolution operation result of each pair group; in the process of traversing each pair group: obtaining each pair of the pair group, the non-zero active site of each pair comprising data of n+1 input channels, and the weight value of each pair comprising data of n+1 input channels and m+1 output channels, wherein n and m are both integers greater than or equal to 0; inputting the data of each pair into hardware for performing neural network calculation, so that the hardware simultaneously performs convolution operation on the n+1 input channels and the m+1 output channels to obtain the corresponding convolution operation result of each pair; obtaining the convolution operation result of the pair group according to the corresponding convolution operation result of each pair.

8. The method according to any one of claims 1-4, characterized in that, Before determining the content of the index matrix, the method further comprises: dividing the original point cloud data into multiple blocks; determining whether there is a non-zero active site in each block; if there is, determining that the block is the point cloud data.

9. A neural network-based point cloud data processing system, characterized by, comprising a data preprocessing module and an accelerator: the data preprocessing module is used to obtain a weight matrix of a neural network, the weight matrix comprising at least one weight value; the data preprocessing module is further used to determine the size of an index matrix according to the size of the weight matrix; the data preprocessing module is further used to determine the content of the index matrix according to the position of at least one zero active site in the point cloud data to be processed and the position of at least one non-zero active site in the point cloud data, the index matrix comprising at least one index value, each index value being used to represent whether the value of the site at the corresponding position in the point cloud data is zero; the data preprocessing module is further used to rearrange at least one non-zero active site according to the position of the at least one non-zero active site and the size of the weight matrix to determine an active value matrix; the accelerator is used to determine at least one pair group according to the index matrix, the active value matrix and the weight matrix, each pair group comprising at least one pair, each pair comprising one non-zero active site in the active value matrix and one weight value in the weight matrix corresponding to the non-zero active site, wherein the at least one index value comprises a first index value and a second index value, the first index value being used to represent that the value of the site at the corresponding position in the point cloud data is zero, and the second index value being used to represent that the value of the site at the corresponding position in the point cloud data is not zero, and the determination of the at least one pair group according to the index matrix, the active value matrix and the weight matrix comprises: determining the size of an extraction area in the index matrix for extracting each pair group in the at least one pair group according to the size of the weight matrix; The position of at least one of the extraction regions is determined in sequence from the beginning to the end in the column direction of the index matrix, wherein each determined extraction region is a target extraction region; After the position of the target extraction region is determined each time, the following processing is performed: If the index value of the center position of the target extraction region is the second index value, a pair group corresponding to a position corresponding to the center position in the point cloud data is determined according to the index value of the target extraction region; If the index value of the center position of the target extraction region is the first index value, the processing process for the target extraction region is ended; The accelerator is further configured to perform convolution operation on the at least one pair group through a neural network to obtain a processing result of the point cloud data.

Images