A weight matrix acceleration system and method

Abstract

The present application provides a weight matrix acceleration system and method, which relate to the field of accelerator technology. The system includes: a data preprocessing module and a sparse feature extraction module; the data preprocessing module is configured to: obtain point cloud data; map data points in the point cloud data to a voxel grid; filter the point cloud data to obtain sparse point cloud data; the sparse feature extraction module is configured to: use sparse convolution operations to extract features of the sparse point cloud data to obtain data dimensions; obtain an initialized weight matrix; generate a mask matrix; generate a sparse weight matrix; based on the sparse weight matrix and the data dimension, retain the TopK weights of each row in the sparse weight matrix to obtain a QKV matrix; this is to solve the problem that the current point cloud Transformer architecture needs to perform intensive calculations on the high-dimensional weight matrix when generating the QKV matrix, resulting in a sharp increase in computing power requirements in large-scale point cloud data and difficulty in meeting real-time requirements.

Description

Technical Field

[0001] The present application relates to the field of accelerator technology, and in particular to a weight matrix acceleration system and method. Background Art

[0002] With the widespread application of 3D point cloud data in fields such as autonomous driving, robotic perception, and 3D reconstruction, efficient and accurate point cloud feature extraction and understanding have become core requirements. For example, in autonomous driving scenarios, real-time 3D object detection relies on efficient processing of large-scale point clouds, ensuring accuracy while reducing computational latency. Robotic navigation requires rapid analysis of the geometric structure of complex scenes for obstacle avoidance and path planning. However, current point cloud processing algorithms are limited in their ability to express features, making it difficult to capture the long-range dependencies and global context of point cloud data.

[0003] To address the problem of global feature modeling in point clouds, the existing technology proposes a point cloud Transformer architecture, the core idea of ​​which is to achieve global interaction of point cloud features through a self-attention mechanism. Specifically, this type of method first projects the point cloud features into query (Q), key (K), and value (V) matrices, then calculates feature weights through dot product attention, and finally integrates global information. Some methods further introduce local window attention, which divides the point cloud into local regions of fixed size and only calculates attention within the region to reduce computational complexity. In addition, sparsification techniques (such as weight pruning and low-rank decomposition) are currently being applied to compress the number of parameters in the QKV fully connected layer to reduce memory usage.

[0004] However, the current point cloud Transformer architecture requires intensive calculations of high-dimensional weight matrices when generating Q, K, and V matrices, resulting in a sharp increase in computing power requirements in large-scale point cloud data (such as autonomous driving scenarios), making it difficult to meet real-time requirements; the number of parameters in the QKV fully connected layer is quadratically related to the feature dimension, resulting in a large model size and difficulty in deployment on edge computing devices; although sparsification technology can reduce the amount of calculation, it will destroy the local geometric structure characteristics of the point cloud data, resulting in a significant reduction in the number of key indicators (such as mIoU). Summary of the Invention

[0005] This application provides a weight matrix acceleration system and method to solve the technical problem that the current point cloud Transformer architecture needs to perform intensive calculations on high-dimensional weight matrices when generating Q, K, and V matrices, resulting in a sharp increase in computing power requirements in large-scale point cloud data and difficulty in meeting real-time requirements.

[0006] The first aspect of the present application provides a weight matrix acceleration system, comprising:

[0007] Data preprocessing module and sparse feature extraction module;

[0008] The data preprocessing module is configured to:

[0009] Get point cloud data;

[0010] Mapping data points in the point cloud data into a voxel grid;

[0011] Based on the point cloud data, the point cloud data is filtered using a preset model to obtain sparse point cloud data; the preset model includes: a TopK model, a threshold model, and a zero value model;

[0012] The sparse feature extraction module is configured to:

[0013] Using a 3D sparse convolution operation, extracting features of the sparse point cloud data to obtain data dimensions;

[0014] Get the initialized weight matrix;

[0015] Based on the initialization weight matrix, generating a mask matrix;

[0016] Generating a sparse weight matrix based on the initialization weight matrix and the mask matrix;

[0017] Based on the sparse weight matrix and the data dimension, a TopK weight model is used to retain the TopK weight of each row in the sparse weight matrix to obtain a QKV matrix; the TopK weight model is configured to obtain the TopK weight of each row in the sparse weight matrix using a TopK weight formula; the TopK weight formula is:

[0018] Q / K / V = ;

[0019] Where, Q is the query matrix; K is the bond matrix; V is the value matrix; W sparse is a sparse weight matrix; F emb is the data dimension.

[0020] In some embodiments, after the step of obtaining point cloud data, the following steps are included:

[0021] The data point coordinates of the point cloud data are mapped to a normalized space.

[0022] In some embodiments, the step of mapping the data points in the point cloud data into a voxel grid comprises:

[0023] Determining a voxel grid size based on the maximum data point coordinates and the minimum data point coordinates of the point cloud data;

[0024] Establishing a voxel grid according to the voxel grid size;

[0025] Calculate a voxel index based on the point cloud data and the voxel grid; the voxel index is:

[0026] i = , j = , k = ;

[0027] Where, the data point coordinates of the point cloud data are (x, y, z); v is the boundary length of the voxel grid;

[0028] Data points in the point cloud data are mapped to the voxel grid according to the voxel index.

[0029] In some embodiments, the step of filtering the point cloud data using a preset model based on the point cloud data to obtain sparse point cloud data includes:

[0030] Repeatedly obtain a preset number of data points in the point cloud data, and input the data points into the TopK model, threshold model, and zero value model in turn for filtering until all data points in the point cloud data are filtered to obtain sparse point cloud data.

[0031] In some embodiments, the TopK model is configured to:

[0032] Obtaining a feature vector of each data point in the point cloud data;

[0033] Calculating the L2 norm of each data point in the point cloud data using the eigenvector;

[0034] Sort the data points in the point cloud data from large to small according to the L2 norm, and retain the top 30% of the data points in the point cloud data.

[0035] In some embodiments, the threshold model is configured to:

[0036] Obtaining a feature vector of each data point in the point cloud data;

[0037] Calculating the L2 norm of each data point in the point cloud data using the eigenvector;

[0038] Data points in the point cloud data whose L2 norm is greater than 0.1 are retained.

[0039] In some embodiments, the zero-value model is configured to:

[0040] Obtaining a feature vector of each data point in the point cloud data;

[0041] Determine whether there are data points with the eigenvector being 0 in the point cloud data; if so, remove the data points with the eigenvector being 0.

[0042] In some embodiments, a plurality of windows are provided in the QKV matrix, and the sequence lengths of the QKV matrices in the windows are the same; a computing unit is provided in the window; and the computing unit is configured to perform sparse attention calculations in the window.

[0043] In some embodiments, the system further comprises:

[0044] MLP feature enhancement module, the MLP feature enhancement module is configured to:

[0045] Based on the data dimension, determining the original number of channels;

[0046] Expand the number of input feature channels of the hidden layer to 4 times the original number of channels;

[0047] The number of input feature channels of the activation layer is restored to the original number of channels.

[0048] A second aspect of the present application provides a weight matrix acceleration method, which is applied to a weight matrix acceleration system described in any one of the first aspects above, comprising:

[0049] Get point cloud data;

[0050] Mapping data points in the point cloud data into a voxel grid;

[0051] Based on the point cloud data, the point cloud data is filtered using a preset model to obtain sparse point cloud data; the preset model includes: a TopK model, a threshold model, and a zero value model;

[0052] Using a 3D sparse convolution operation, extracting features of the sparse point cloud data to obtain data dimensions;

[0053] Get the initialized weight matrix;

[0054] Based on the initialization weight matrix, generating a mask matrix;

[0055] Generating a sparse weight matrix based on the initialization weight matrix and the mask matrix;

[0056] Based on the sparse weight matrix and the data dimension, a TopK weight model is used to retain the TopK weight of each row in the sparse weight matrix to obtain a QKV matrix; the TopK weight model is configured to obtain the TopK weight of each row in the sparse weight matrix using a TopK weight formula; the TopK weight formula is:

[0057] Q / K / V = ;

[0058] Where, Q is the query matrix; K is the bond matrix; V is the value matrix; W sparse is a sparse weight matrix; F emb is the data dimension.

[0059] The present application provides a weight matrix acceleration system and method, the system comprising: a data preprocessing module and a sparse feature extraction module; the data preprocessing module is configured to: obtain point cloud data; map data points in the point cloud data to a voxel grid; based on the point cloud data, filter the point cloud data using a preset model to obtain sparse point cloud data; the preset model comprises: a TopK model, a threshold model, and a zero-value model; the sparse feature extraction module is configured to: extract features of the sparse point cloud data using a 3D sparse convolution operation to obtain data dimensions; obtain an initialized weight matrix; generate a mask matrix based on the initialized weight matrix; generate a sparse weight matrix based on the initialized weight matrix and the mask matrix; based on the sparse weight matrix and the data dimensions, retain the TopK weight of each row in the sparse weight matrix using a TopK weight model to obtain a QKV matrix; the TopK weight model is configured to obtain the TopK weight of each row in the sparse weight matrix using a TopK weight formula; the TopK weight formula is:

[0060] Q / K / V = ;

[0061] Where, Q is the query matrix; K is the bond matrix; V is the value matrix; W sparse is a sparse weight matrix; F embThe data dimension is used to solve the problem that the current point cloud Transformer architecture needs to perform intensive calculations on high-dimensional weight matrices when generating Q, K, and V matrices, resulting in a sharp increase in computing power requirements in large-scale point cloud data and difficulty in meeting real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 This is a schematic diagram of the structure of the weight matrix acceleration system in this application;

[0064] Figure 2 This is a flowchart of the data preprocessing module in this application when it is running;

[0065] Figure 3 This is a flowchart of the sparse feature extraction module in this application when it is running;

[0066] Figure 4 This is a comparison curve of the effective value density under the original point cloud Transformer architecture and the point cloud Transformer architecture in this application.

[0067] Description of reference numerals:

[0068] 1-Data preprocessing module; 2-Sparse feature extraction module; 3-MLP feature enhancement module. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0070] For example, point cloud transformers (such as PointCloudTransformerV3) perform well in tasks such as 3D object detection and point cloud segmentation, but their self-attention mechanism needs to calculate the global relationship between all points, resulting in a computational complexity of O(N 2 ) (N is the number of data points), which is difficult to meet real-time requirements.

[0071] For example, 1. The computational efficiency of the point cloud Transformer network is low and the computational complexity is high: the current point cloud Transformer (taking PointCloudTransformerV3 as an example) needs to perform intensive calculations on the high-dimensional weight matrix when generating the query (Q), key (K), and value (V) matrices, resulting in a sharp increase in computing power requirements, especially in large-scale point cloud data (such as 3D target detection for autonomous driving). It is difficult to deploy in real time. 2. Large memory usage: The QKV fully connected layer has a large number of parameters, which makes it difficult to deploy edge devices. 3. Invalid feature redundancy: There are a large number of low-information points in the original point cloud data (such as ground points, sensor noise), which cannot be dynamically filtered. 4. The current sparsification method has a significant loss of accuracy: although the current weight pruning or low-rank approximation method can reduce the amount of calculation, it will destroy the local geometric structure characteristics of the point cloud data, resulting in a decrease in key indicators such as mIoU.

[0072] In order to solve this technical problem, the present application provides a weight matrix acceleration system and method, which are described below:

[0073] like Figure 1 As shown, it is a structural diagram of the weight matrix acceleration system in this application.

[0074] The first aspect of the present application provides a weight matrix acceleration system, comprising:

[0075] Data preprocessing module 1 and sparse feature extraction module 2.

[0076] like Figure 2 As shown, this is a flow chart of the data preprocessing module in this application when it is running.

[0077] The data preprocessing module 1 is configured as follows:

[0078] Acquire point cloud data; the point cloud data P For: R N×3 ; R is the real number set in the point cloud data, and N is the three-dimensional coordinate of the data point in the point cloud data.

[0079] After the step of obtaining point cloud data, the following steps are included:

[0080] The data point coordinates of the point cloud data are mapped to a normalized space.

[0081] Specifically, the normalization operation of the data point coordinates of the point cloud data is as follows: first translate the coordinates to the origin (minus the minimum value), then scale to the grid unit (divided by grid_size=x max -x min ), and finally rounded to get the discrete grid coordinates.

[0082] Mapping the data point coordinates of the point cloud data to a normalized space has the following effects: 1. Eliminating scale differences: Point cloud data may come from different sensors or scenes, and their coordinate ranges may vary greatly. Normalization maps all coordinates to a unified scale, allowing the model to maintain a consistent response to data from different sources. 2. Accelerating convergence: Neural networks learn more easily when processing normalized data because the range of input values ​​is constrained, making gradient updates more stable, thereby accelerating the training process.

[0083] Data points in the point cloud data are mapped into a voxel grid.

[0084] The step of mapping the data points in the point cloud data into a voxel grid comprises the following sub-steps:

[0085] Determine the voxel grid size based on the maximum data point coordinate and the minimum data point coordinate of the point cloud data; max and the minimum data point coordinate x min , the boundary of the voxel grid, ie the size of the voxel grid, can be determined.

[0086] A voxel grid is established according to the voxel grid size; the voxel grid size can be established by the size of the voxel grid, and the voxel grid is in a cubic shape.

[0087] Calculate a voxel index based on the point cloud data and the voxel grid; the voxel index is:

[0088] i = , j = , k = ;

[0089] Where, the data point coordinates of the point cloud data are x, y, z; v is the boundary length of the voxel grid; Indicates rounding down.

[0090] Based on the voxel index, the data points in the point cloud data are mapped to the voxel grid. Voxelization is used to reduce the dimensionality and sparsely represent point cloud data: point cloud data is irregular three-dimensional data, and direct processing has high computational overhead; voxelization divides the point cloud data into a regular three-dimensional grid, with each voxel represented by a numerical value, significantly reducing the amount of data. Preserving spatial structure: Voxelization preserves the spatial relationship between points through grid indexing, allowing subsequent convolution operations to effectively extract local features.

[0091] Based on the point cloud data, the point cloud data is filtered using a preset model to obtain sparse point cloud data; the preset model includes: TopK model, threshold model, zero value model. The sparse point cloud data P sparse For: R M×3 ; M is the number of data points in the point cloud data after filtering by the preset model.

[0092] The step of filtering the point cloud data using a preset model based on the point cloud data to obtain sparse point cloud data further includes the following steps:

[0093] Repeatedly obtain a preset number of data points from the point cloud data, and sequentially input the data points into the TopK model, threshold model, and zero-value model for filtering until all data points in the point cloud data are filtered, thereby obtaining sparse point cloud data. This application adopts a dynamic filtering mode to filter the data points in the point cloud data. By obtaining a preset number of data points from the point cloud data each time, the data points are sequentially input into the TopK model, threshold model, and zero-value model for filtering. The retained data points are dynamically selected based on their weight importance to avoid the loss of accuracy caused by a fixed pruning rate.

[0094] In this embodiment, the TopK model is configured as follows:

[0095] Obtain the feature vector of each data point in the point cloud data; for example, the feature vector of data point P is (x, y, z).

[0096] Using the eigenvectors, calculate the L2 norm of each data point in the point cloud data. The L2 norm (Euclidean norm) is the square root of the sum of the squares of the vector elements and is used to measure the length or distance of the vector. For example, the L2 norm of the data point P is:

[0097] = .

[0098] Sort the data points in the point cloud data from large to small according to the L2 norm, and retain the top 30% of the data points in the point cloud data.

[0099] Specifically, in the implementation of the TopK model, filter_k=0.3 means retaining the top 30% of the most important points in the TopK model. For example, if there are 1000 data points, then the 300 data points with the largest L2 norm will be retained. The corresponding code is:

[0100] point_cloud.sparsify(filter_mode="topk", filter_k=0.3) .

[0101] In this embodiment, the threshold model is configured as follows:

[0102] Obtain a feature vector for each data point in the point cloud data; calculate an L2 norm for each data point in the point cloud data using the feature vector; and retain data points in the point cloud data whose L2 norm is greater than 0.1.

[0103] Specifically, the implementation of the threshold mode: also using the L2 norm, filter_threshold=0.1 means retaining the data points with L2 norm greater than 0.1 in the point cloud data. The corresponding code is expressed as:

[0104] point_cloud.sparsify(filter_mode="threshold", filter_threshold=0.1).

[0105] In this embodiment, the zero-value model is configured as follows:

[0106] Obtain a feature vector for each data point in the point cloud data; determine whether there is a data point in the point cloud data whose feature vector is 0, and if so, remove the data point whose feature vector is 0.

[0107] Specifically, the zero-value mode is implemented by checking whether the feature vector of each data point is all zero. If all dimensions are 0, the point is considered invalid. The code generates a Boolean mask by calling torch.sum(self.feat != 0, dim=1) = 0 to mark all zero feature points. In this mode, the filter_k parameter is invalid, and all non-zero feature points are retained. The corresponding code is:

[0108] point_cloud.sparsify(filter_mode="zero_feat") .

[0109] like Figure 3 As shown in FIG, it is a flowchart of the sparse feature extraction module during operation in this application.

[0110] The sparse feature extraction module 2, i.e., the first K weighted sparse linear layer, is configured as follows:

[0111] Utilize 3D sparse convolution operation to extract the features of the sparse point cloud data and obtain data dimensions; the data dimensions F embR M×C ; C is the number of channels of output features.

[0112] Obtain an initialization weight matrix; the initialization weight matrix is ​​randomly generated by the system.

[0113] Based on the initialization weight matrix, a mask matrix is ​​generated; the mask matrix has the same shape as the initialization weight matrix, and the weight of the mask matrix is ​​0 or 1, wherein 30% of the values ​​in the mask matrix are assigned to 1 and the rest are assigned to 0, and the assignment of 1 and assignment of 0 is determined according to the k value of filter_k in the TopK mode.

[0114] Based on the initialization weight matrix and the mask matrix, a sparse weight matrix is ​​generated; the sparse weight matrix is:

[0115] W sparse = ;

[0116] Where, W To initialize the weight matrix; M is the mask matrix.

[0117] Based on the sparse weight matrix and the data dimension, a TopK weight model is used to retain the TopK weight of each row in the sparse weight matrix to obtain a QKV matrix; the TopK weight model is configured to obtain the TopK weight of each row in the sparse weight matrix using a TopK weight formula; the TopK weight formula is:

[0118] Q / K / V = ;

[0119] Where, Q is the query matrix; K is the bond matrix; V is the value matrix; W sparse is a sparse weight matrix; F emb is the data dimension.

[0120] In this embodiment, the QKV matrix is ​​provided with a plurality of windows, wherein the sequences of the QKV matrix in the windows are of the same length; a computing unit is provided in the windows; and the computing unit is configured to perform sparse attention calculation in the window. By performing attention calculation in a local window, the computational complexity is reduced from O(M 2 ) is reduced to O(MK).

[0121] Specifically, in the local window attention mechanism, the QKV matrix is ​​divided into multiple non-overlapping windows, and attention is calculated independently within each window. Window division is based on position order rather than attention strength, so data points with the same attention may not necessarily be assigned to the same window. The specific steps are as follows:

[0122] 1. Window division:

[0123] Each window contains 48 tokens. If the sequence length is not an integer multiple of the window size, padding is applied. The sequence is divided into multiple consecutive windows in sequence, and the tokens in each window calculate attention to each other.

[0124] 2. Calculation of attention within the window:

[0125] Within each window, attention calculation is identical to standard self-attention. Tokens within each window only pay attention to other tokens within the same window; there is no direct interaction between windows. Attention calculations across all windows can be processed in parallel, improving efficiency.

[0126] Although attention is calculated independently in each window, subsequent layers will mix information between windows through residual connections and MLP layers. The computational complexity is reduced from O(M 2 ) is reduced to O(MK), where M is the sequence length and K is the window size, K=48.

[0127] In this embodiment, the system further comprises:

[0128] MLP (Multi-layer Perceptron) feature enhancement module, the MLP feature enhancement module 3 is configured as follows:

[0129] Based on the data dimension, the original number of channels C is determined; the number of input feature channels of the hidden layer is expanded to four times the original number of channels; and the number of input feature channels of the activation layer is restored to the original number of channels. Specifically, a dimensionality increase (C → 4C) is first employed to expand feature representations using a higher dimension and refine nonlinear relationships. Specifically, the number of input feature channels of the hidden layer is increased to four times the original number of channels. Subsequently, a dimensionality reduction (4C → C) is employed to output features learned in the high-dimensional space in the form of the original number of channels C, preserving the information after high-dimensional transformation and adapting to the dimensionality requirements of subsequent residual connections and feature fusion. Specifically, the number of input feature channels of the activation layer is restored to the original number of channels.

[0130] Specifically, the MLP feature enhancement module 3 includes two cascaded fully connected layers (the hidden layer dimension is 4C) and a GELU activation function.

[0131] Two cascaded fully connected layers quadruple the number of channels, C, in the input features, increasing nonlinear expression capabilities. The GELU activation layer restores the number of channels to the original dimension, C, compressing feature information. This allows the model to transform features in a high-dimensional space and then return to the original feature space. The MLP feature enhancement module 3 concatenates the residuals, fusing the independent attention information of each QKV window and enabling inter-layer feature transfer.

[0132] like Figure 4 As shown, it is a comparison curve of the effective value density under the original point cloud Transformer architecture and the point cloud Transformer architecture in this application.

[0133] For example, by adopting the weight matrix acceleration system provided by this application, the K elements with the largest absolute values ​​in the weight matrix are retained (the rest are set to zero), which significantly reduces the effective value density of the model (i.e., the proportion of non-zero weights), such as Figure 4 As shown in the figure, encoder 0 (module layer × 2); encoder 1 (pooling + module layer × 2); encoder 2 (pooling + module layer × 2); encoder 3 (pooling + module layer × 6); encoder 4 (pooling + module layer × 2); decoder 3 (unpooling + module layer × 2); decoder 2 (unpooling + module layer × 2); decoder 1 (unpooling + module layer × 2); decoder 0 (unpooling + module layer × 2). This sparsification not only reduces computational and storage overhead but also improves hardware deployment efficiency.

[0134] For example, current optimization solutions for point cloud transformers focus on:

[0135] The traditional point cloud Transformer architecture ("Point Transformer V2: Grouped Vector Attention and Partition-based Pooling." NeurIPS, 2021; "PCT: Point Cloud Transformer." Computational Visual Media, 2021) uses K-nearest neighbor (KNN) for neighborhood search and relies on relative position encoding (RPE) to capture spatial relationships, resulting in high computational complexity and high memory consumption. The weight matrix acceleration system proposed in this application, when applied to the point cloud Transformer architecture, eliminates the need for KNN neighborhood search, avoiding numerous distance calculations and sorting operations. By directly sparsifying the weight matrix, unnecessary computational connections are reduced.

[0136] This application provides a weight matrix acceleration system, which has the following beneficial effects:

[0137] 1. Improved computational efficiency: Through the TopK sparsification strategy, only the non-zero values ​​of the TopK in the QKV weight matrix (for example, K = 30%) are retained. By skipping the calculation of zero values, the computational complexity is reduced by more than 50%.

[0138] 2. Lossless Precision Optimization: Based on a dynamic sparsity strategy and gradient reparameterization technology, the mIoU indicator of the point cloud segmentation task decreases by less than 0.5% at a sparsity of K=30%.

[0139] A second aspect of the present application provides a weight matrix acceleration method, which is applied to a weight matrix acceleration system described in any of the above embodiments, including:

[0140] Get point cloud data;

[0141] Mapping data points in the point cloud data into a voxel grid;

[0142] Based on the point cloud data, the point cloud data is filtered using a preset model to obtain sparse point cloud data; the preset model includes: a TopK model, a threshold model, and a zero value model;

[0143] Using a 3D sparse convolution operation, extracting features of the sparse point cloud data to obtain data dimensions;

[0144] Get the initialized weight matrix;

[0145] Based on the initialization weight matrix, generating a mask matrix;

[0146] Generating a sparse weight matrix based on the initialization weight matrix and the mask matrix;

[0147] Based on the sparse weight matrix and the data dimension, a TopK weight model is used to retain the TopK weight of each row in the sparse weight matrix to obtain a QKV matrix; the TopK weight model is configured to obtain the TopK weight of each row in the sparse weight matrix using a TopK weight formula; the TopK weight formula is:

[0148] Q / K / V = ;

[0149] Where, Q is the query matrix; K is the bond matrix; V is the value matrix; W sparse is a sparse weight matrix; Femb is the data dimension.

[0150] It is worth noting that the effects of the above method embodiments can be found in the effects of the above system embodiments, which will not be described in detail here.

[0151] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not intended to limit the scope of protection of the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the scope of protection of the embodiments of the present application.

Claims

1. A weight matrix acceleration system, characterized in that: include: Data preprocessing module (1) and sparse feature extraction module (2); The data preprocessing module (1) is configured to: Get point cloud data; Mapping data points in the point cloud data into a voxel grid; Based on the point cloud data, filtering the point cloud data using a preset model to obtain sparse point cloud data; The preset models include: TopK model, threshold model and zero value model; The sparse feature extraction module (2) is configured as follows: Using a 3D sparse convolution operation, extracting features of the sparse point cloud data to obtain data dimensions; Get the initialized weight matrix; Generating a mask matrix based on the initialized weight matrix; Generate a sparse weight matrix based on the initialization weight matrix and the mask matrix; Based on the sparse weight matrix and the data dimension, a TopK weight model is used to retain the TopK weight of each row in the sparse weight matrix to obtain a QKV matrix; the TopK weight model is configured to obtain the TopK weight of each row in the sparse weight matrix using a TopK weight formula; the TopK weight formula is: Q / K / V = ； Where, Q is the query matrix; K is the bond matrix; V is the value matrix; W sparse is a sparse weight matrix; F emb is the data dimension; The TopK model is configured as follows: Obtaining a feature vector of each data point in the point cloud data; Calculating the L2 norm of each data point in the point cloud data using the eigenvector; Sort the data points in the point cloud data from large to small according to the L2 norm, and retain the top 30% of the data points in the point cloud data; The threshold model is configured as: Obtaining a feature vector of each data point in the point cloud data; Calculating the L2 norm of each data point in the point cloud data using the eigenvector; Retaining data points in the point cloud data whose L2 norm is greater than 0.1; The zero-value model is configured as: Obtaining a feature vector of each data point in the point cloud data; Determine whether there are data points with the eigenvector being 0 in the point cloud data; if so, remove the data points with the eigenvector being 0.

2. A weight matrix acceleration system according to claim 1, characterized in that: After the step of obtaining point cloud data, the following steps are included: The data point coordinates of the point cloud data are mapped to a normalized space.

3. A weight matrix acceleration system according to claim 1, characterized in that: The step of mapping the data points in the point cloud data into a voxel grid comprises: Determining a voxel grid size based on the maximum data point coordinates and the minimum data point coordinates of the point cloud data; Establishing a voxel grid according to the voxel grid size; Calculate a voxel index based on the point cloud data and the voxel grid; the voxel index is: i = ， j = ， k = ； Where, the data point coordinates of the point cloud data are (x, y, z); v is the boundary length of the voxel grid; is the minimum value of the x-coordinate of all points in the point cloud data. is the minimum value of the y coordinates of all points in the point cloud data, It is the minimum value of the z coordinate of all points in the point cloud data; Data points in the point cloud data are mapped to the voxel grid according to the voxel index.

4. A weight matrix acceleration system according to claim 1, characterized in that: The step of filtering the point cloud data using a preset model based on the point cloud data to obtain sparse point cloud data includes: Repeatedly obtain a preset number of data points in the point cloud data, and input the data points into the TopK model, threshold model, and zero value model in turn for filtering until all data points in the point cloud data are filtered to obtain sparse point cloud data.

5. A weight matrix acceleration system according to claim 1, characterized in that: Several windows are set in the QKV matrix, and the sequence lengths of the QKV matrix in the windows are the same; a computing unit is set in the window; and the computing unit is configured to perform sparse attention calculation in the window.

6. A weight matrix acceleration system according to claim 1, characterized in that: The system further comprises: The MLP feature enhancement module (3) is configured to: Based on the data dimension, determining the original number of channels; Expand the number of input feature channels of the hidden layer to 4 times the original number of channels; The number of input feature channels of the activation layer is restored to the original number of channels.

7. A weight matrix acceleration method, applied to a weight matrix acceleration system according to any one of claims 1 to 6, characterized in that: include: Get point cloud data; Mapping data points in the point cloud data into a voxel grid; Based on the point cloud data, filtering the point cloud data using a preset model to obtain sparse point cloud data; The preset models include: TopK model, threshold model and zero value model; Using a 3D sparse convolution operation, extracting features of the sparse point cloud data to obtain data dimensions; Get the initialized weight matrix; Generating a mask matrix based on the initialized weight matrix; Generate a sparse weight matrix based on the initialization weight matrix and the mask matrix; Based on the sparse weight matrix and the data dimension, a TopK weight model is used to retain the TopK weight of each row in the sparse weight matrix to obtain a QKV matrix; the TopK weight model is configured to obtain the TopK weight of each row in the sparse weight matrix using a TopK weight formula; the TopK weight formula is: Q / K / V = ； Where, Q is the query matrix; K is the bond matrix; V is the value matrix; W sparse is a sparse weight matrix; F emb is the data dimension; The TopK model is configured as follows: Obtaining a feature vector of each data point in the point cloud data; Calculating the L2 norm of each data point in the point cloud data using the eigenvector; Sort the data points in the point cloud data from large to small according to the L2 norm, and retain the top 30% of the data points in the point cloud data; The threshold model is configured as: Obtaining a feature vector of each data point in the point cloud data; Calculating the L2 norm of each data point in the point cloud data using the eigenvector; Retaining data points in the point cloud data whose L2 norm is greater than 0.1; The zero-value model is configured as: Obtaining a feature vector of each data point in the point cloud data; Determine whether there are data points with the eigenvector being 0 in the point cloud data; if so, remove the data points with the eigenvector being 0.

Images