Table convolution and acceleration

Abstract

Certain aspects of the present disclosure provide techniques for performing table convolution, including performing a tabularization operation on input data to generate a tabularized representation of the input data, and performing a convolution operation using the tabularized representation of the input data to generate a convolution output.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 407,046, filed August 19, 2021, and U.S. Provisional Patent Application No. 63 / 068,891, filed August 21, 2020, the entire contents of which are incorporated herein by reference.

[0003] introduction

[0004] Various aspects of this disclosure relate to improved machine learning processing.

[0005] Machine learning is generally the process of generating trained models (e.g., artificial neural networks, trees, or other structures) that represent a generalized fit to a previously known set of training data. Applying the trained model to new data produces inferences, which can be used to gain insights about the new data. In some contexts, applying a model to new data is described as “running inference” on the new data.

[0006] The surge in use of machine learning for various machine learning (or artificial intelligence) tasks has created a need for more efficient processing of machine learning model data. For example, "edge processing" devices (such as mobile devices, always-on devices, Internet of Things (IoT) devices, etc.) must balance processing power with power and packaging constraints.

[0007] One aspect of machine learning processing that is typically inefficient is handling sparse data in machine learning models. Machine learning often requires performing numerous mathematical operations on the input data, including multiplication and addition. Normally, these operations are performed regardless of whether they produce a meaningful output. For example, even when the input data is very sparse (e.g., with many zero values), conventional machine learning processing still performs multiplication and addition on zero-valued inputs. This can lead to inefficiency because operations performed with zero input values ​​may not change or otherwise lead to a meaningful output. For example, adding zero to any number will result in the same number.

[0008] Accordingly, there is a need for systems and methods to improve the efficiency of machine learning processing for sparse input data.

[0009] Brief Overview

[0010] Some aspects provide methods for performing tabular convolution, including: performing a tabular operation on input data to generate a tabular representation of the input data; and using the tabular representation of the input data to perform a convolution operation to generate a convolution output.

[0011] Other aspects include: a processing system configured to perform the foregoing methods and those methods described herein; a non-transient computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product implemented on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including means for performing the foregoing methods and those methods further described herein.

[0012] The following description and related figures illustrate certain illustrative features of one or more embodiments in detail. Brief description of the attached diagram

[0014] The accompanying drawings illustrate certain aspects of one or more embodiments and are therefore not intended to limit the scope of this disclosure.

[0015] Figure 1A and 1B An example of a weight kernel that can be used in tabular convolution is depicted.

[0016] Figure 2A and 2B An example aspect of tabularizing sparse input data is described.

[0017] Figure 3 An example method for performing a table convolution is described.

[0018] Figures 4A-4C Example results of table convolution are depicted.

[0019] Figure 5 An example method for performing table convolution using skip connections is described.

[0020] Figure 6 An example of hashing tabular input data is depicted.

[0021] Figure 7 An example method for performing a table convolution is described.

[0022] Figure 8 An example processing system configured to perform table convolutions is described.

[0023] To facilitate understanding, the same reference numerals are used wherever possible to designate common elements in the various figures. Elements and features of one embodiment are contemplated and may be beneficially incorporated into other embodiments without further elaboration.

[0024] Detailed description

[0025] This disclosure provides apparatus, methods, processing systems, and computer-readable media for efficiently performing machine learning processing on sparse input data, and in particular for performing tabular convolutions on sparse data to improve the efficiency of convolution processing.

[0026] Sparse data generally refers to data with a relatively high proportion of zero (or near-zero) value entries, such as vectors, matrices, or tensors. For example, a simple example vector with entries [0,0,0,1] can be considered 75% sparse (and 25% dense). When performing mathematical operations (such as element-wise addition) on this example vector, it is clear that the sparse entries do not change the output value for 75% of the vector elements. Similarly, when performing multiplication, it is clear that all successive multiplications based on zero-valued elements will also take zero values. When considering a large number of mathematical operations in machine learning processing, these sparse entries represent opportunities for efficiency enhancements, which can beneficially lead to faster machine learning processing, lower power consumption, reduced processing complexity, reduced data storage (e.g., memory) requirements, lower cost hardware requirements, and other benefits.

[0027] In contrast to conventional dense representations, sparse representations of data are becoming increasingly appropriate in many technological fields. For example, modern sensor systems used in various types of end-user products typically generate sparse data. One example is the three-dimensional (3D) point clouds generated by light detection and ranging (LiDAR) sensors used in automotive applications (e.g., autonomous driving systems) and mobile device applications (e.g., for biometric user authentication). Such 3D point clouds are typically very sparse.

[0028] Conventional machine learning approaches (such as deep learning) do not leverage sparsity to improve performance. For example, conventional convolutional processing of sparse image data may rely on multilayer perceptual (MLP) operations (e.g., PointNet, PointNet++, and PointConv), which sacrifice efficiency due to the fully connected nature of their operations.

[0029] Other approaches may involve transforming sparse input data into other representations, such as voxelization for 3D convolutional neural network processing, projection / rendering for 2D convolutional neural network processing, and feature extraction for fully connected operations. It is noteworthy that none of these transformation-based approaches directly perform deep learning on the sparse data representation, and thus, each of these approaches introduces additional complexity and inefficiency based on the required transformation.

[0030] In some cases, such as when processing event images from event cameras or sensors, deep learning methods are completely avoided due to the lack of efficient methods for performing convolutions on sparse input data.

[0031] The embodiments described herein overcome these conventional technical problems by implementing sparse-aware convolution (which may be alternatively referred to as tabular convolution). Tabular convolution advantageously enables fast and efficient convolution processing directly on sparse data without the need for expensive and complex transformations of input data to dense representations. Overall, the tabular convolution methods described herein can be applied to 2D convolutions as well as higher-dimensional convolutions. Additional embodiments described herein further enhance tabular convolution through various acceleration steps.

[0032] Table Convolution

[0033] The embodiments of tabular convolution described herein generally include performing tabular operations on input data, followed by specialized convolutions utilizing the tabular input data. Tabular operations typically acquire sparse data sequences and convert them into a tabular data format with one column of active points and multiple columns indicating the relationship between adjacent points and active points. Specialized convolutions acquire tabular input data to perform arithmetic operations equivalent to regular convolutions but with significantly reduced complexity. Thus, tabular convolutions allow operations to be performed directly on sparse data representations (e.g., without intermediate conversion to dense data representations) while performing mathematical operations in a dense manner (e.g., without processing sparse data elements). As mentioned above, sparse data elements in the data representation can typically be zero-valued elements. In some cases, a sparse element threshold can be used to determine whether an element should be considered a sparse element.

[0034] As an example, sparse input data representations (e.g., images) can be represented as having shape of Where C is the number of channels in the input data, and A represents the number of "active" (non-zero) elements in the input data, and thus C×A represents the real space. Dimensions.

[0035] thus, Represents an arbitrary set of active points (or elements) in "raw" format, depending on the sensor and device interface. It can then be transformed into a shape Deterministic dictionary representation (X) active ), so that in Indicate the coordinates of element a in channel c and in dimension j, and The element value of element a is indicated. Since the original input data has an arbitrary format, it may not always have D dimensions. Here, data format "normalization" can be performed by introducing a D factor to explicitly represent the D component coordinates of each active point. Accordingly, X activeThe format is a "dictionary" format for active elements, where each element is represented by a tuple of D-component coordinates and the quantity at that coordinate (position).

[0036] Then assume a depth-separable set of weights. It has K entries in each channel c (including the center entry of channel c). ), where K represents the kernel size for each channel. In the example described herein, for simplicity, K∈{5,9}, but other embodiments may use different values ​​for K.

[0037] Now, tabular operations can be defined as:

[0038]

[0039] It will X active Convert to a table (or tabular representation) X in each channel c T Labeled as c∈{0,1,…,C-1}, and where for each c:

[0040] X T =T K (X active )={x a,k :a=0,1,…,A-1;k=0,1,…,K-1},

[0041] And among them:

[0042] as well as

[0043] like

[0044] Note that in the expression above, 'a' indicates the active point, and the tabulated value of the active point at column k=0 is equal to (before tabulation) x. active Active points in the format.

[0045] Given the above definition of tabular operations, we can define an index rule k = Λ, where Λ is the set of indices of the neighbors of an active point. For example, if K = 5, then Λ = {1, 2, 3, 4}, because k = 0 is the index of the active point itself.

[0046] For example, Figure 1A and 1B It describes the weights (W) used for channel c. c Two example kernels, 100A and 100B, are given, where K=9 for kernel 100A and K=5 for kernel 100B. In this example, Figure 1AKernel 100A is a "square kernel" with dimensions 3×3, therefore it has K=9 entries, and Figure 1B The kernel 100B is a non-square cross-shaped kernel with K=5 entries.

[0047] In these examples, each entry of kernel weight matrices 100A and 100B (e.g., The intermediate weights are referenced sequentially, and they are always k=0.

[0048] Therefore, considering Figure 1A Where D = 2 (dimensions) and K = 9 (kernel entries), then for each k = Λ∈{1,…,K-1}, an indexing rule can be derived such that:

[0049] If k=1, if

[0050] If k=2, then

[0051] If k=3, then

[0052] If k=4, then

[0053] If k=5, then

[0054] If k=6, then

[0055] If k=7, then or

[0056] If k=8, then

[0057] As another example, consider Figure 1B Where D = 2 (dimensions) and K = 5 (kernel entries), then for each k = Λ∈{1,…,K-1}, an indexing rule can be derived such that:

[0058] If k=1, if

[0059] If k=2, then

[0060] If k=3, then or

[0061] If k=4, then

[0062] In these examples, where the index values ​​are in the form of (x, y), a negative x index value indicates a movement to the left relative to the center k=0, a negative y index value indicates a movement downwards relative to the center, a positive x index value indicates a movement to the right relative to the center, and a positive y index value indicates a movement upwards relative to the center.

[0063] Given the above definition of sparse input data representation and indexing rules, Figure 2A and 2B An example of how to use a weighted kernel to form a tabular output based on sparse data input is described.

[0064] Specifically, Figure 2A Depicting in D=2 dimensions (d 0 and d 1 The active elements {x0,…,x9} are plotted in the diagram. Therefore, in this example, A = 10 active elements.

[0065] Further in Figure 2A The text describes the weighted kernel coverage of 202 to provide... Figure 2B The context of the indexing scheme in the context. For example, from the perspective of active element x0, active element x1 is indexed as (0, -1) because it is in d 1 In the dimension from x0 to the next unit (-1) and in d 0 They are located in the same unit in terms of dimension.

[0066] Based on a weighted kernel with K=5 (e.g.) Figure 1B (in Chinese), tabular output of a single channel c of sparse data exist Figure 2B Described in the text.

[0067] exist Figure 2B In the table, the leftmost column (e.g., the center point) with index (0,0) is the active point vector X. active , which means Figure 2A The active elements {x0,…,x9} are depicted in the table. The remaining columns, indexed as (1,0), (-1,0), (0,1), and (0,-1), represent the tabular relationships between the active elements based on the indexing rules described above.

[0068] It is worth noting that the generation Figure 2B The tabular operation T described in the text K It does not involve multiplication and only requires bitwise logic to reorganize sparse data (which is much smaller than the amount of dense data).

[0069] Given the previous weight kernel definition For each 'c' in C, there exists a weight column vector (tensor) in the neural network. It can be trained and applied in tests to derive tabular convolution outputs based on tabular operations defined by the following formula:

[0070]

[0071] in It is a vector of channels c in C. Alternatively, for all channels, the operation can be defined by the following formula:

[0072]

[0073] Figure 3 An example tabular convolution operation 300 is described, which begins with input data X in step 302 (as described above). active The data is tabulated to generate tabular output data X. T Then, in step 304, the tabular output data X is... T Convolve with kernel weights W to generate tabular convolution output data Y=X. T W.

[0074] Notice, Figure 3 The example described assumes that convolution is performed on all or one or more channels c of the data C at once, but this process can be repeated iteratively for each channel of the data, i.e., generating convolutions for all channels c of C. In such cases, partial (channel-specific) convolutional outputs can be accumulated during each channel-specific iteration to generate the final tabular convolutional output.

[0075] Compared to regular dense convolution, tabular convolution achieves mathematically equivalent convolution results, but with a sparse representation that enables efficient matrix multiplication. Accordingly, if K is fixed, the complexity required to perform tabular convolution is CAK or O(CA), where C is the number of channels in the input data, A represents the number of active (non-zero) elements in the input data, and K is the number of kernel entries per layer. In the case of N output channels in a convolutional layer, the complexity thus becomes CAKN or O(CAN).

[0076] Furthermore, according to the above definition, sparsity on the sparse representation is maintained after one or more layers of tabular convolution. In other words, the set of active points remains unaffected, meaning that the convolution output is neither expanded nor blurred, because the set of target points strictly follows the active points.

[0077] For example, Figures 4A-4C An example of regular convolution relative to tabular convolution is depicted for sparse input data.

[0078] Specifically, Figure 4AThe sparse input data 400 is depicted, where pixels (squares) indicate active elements with non-zero values.

[0079] Figure 4B The result 410 of a regular convolution on sparse input data 400 is depicted. Notably, this convolution causes an expansion of the original active elements, resulting in many additional active elements with non-zero values, such as those labeled with pixels that are shaded differently around the pixels of the original input data.

[0080] on the other hand, Figure 4C The result of the table convolution operation is depicted in diagram 420. It is worth noting that, compared to... Figure 4A In contrast, the values ​​of active elements have changed, as depicted by the pattern variations in each pixel, but the number and position of pixels with non-zero values ​​(active elements) remain unchanged.

[0081] Table convolution with skip connections

[0082] In some embodiments, table convolution can be implemented using skip connections, such as Figure 5 As depicted. For such embodiments, X can be described as follows: T Redefining it as X T′ :

[0083] X T′ =T K (X active )={x a,k :a=0,1,…,A-1;k=1,…,K-1},

[0084] like

[0085] It is worth noting that, compared to the definition of tabular convolution without skip connections discussed above where k ranges from 0 to K-1, the redefinition above includes k ranging from 1 to K-1. Therefore, this redefinition removes the X values ​​of column 0 from the tabular result before applying kernel weights. T (with active element (X) active Dense vector components, such as Figure 2B (as depicted in the text) to form X T′ Beneficially, removing active elements increases the remaining tabular representation X. T′ The sparsity of the data further improves the efficiency gain of tabular convolution.

[0086] To account for active elements (X) before convolution active The removal of ) added a hop connection 502. Additionally, the kernel's central entry for all channels (e.g., Figure 1A and 1BIn the equation, the weights corresponding to k=0 become 1 for all active points, and the non-center entries of the kernel for all channels remain trainable and related to the center entries (which are 1), thus forming W′. Therefore, the resulting tabular convolution with skip connections can be represented as:

[0087] Y = X T′ W′+X active .

[0088] Accordingly, in Figure 5 In Example 500, X active It is provided as input to the tabulation step 504 and provided to the element-wise summation (or addition) operator 508 on the jump connection 502.

[0089] Then in step 506, the tabular data representation X is... T′ (Missing active elements at k=0) is convolved with kernel weights W′ to generate X. T′ W′. Finally, the convolution output X. T′ W′ connects to jump 502 (carrying value X) active Sum the results to generate a tabular convolution output Y.

[0090] Add jump connections and redefine the table representation X T′ Provided better than (e.g., regarding) Figure 3 The additional benefits of (described) skip-connected table convolutions include further reductions in computational complexity (especially for non-square weight kernels, such as...) Figure 1B Kernel 100B and Figure 2A In addition to reducing the weight memory and intermediate activation memory (especially for sparse weight kernels), each of these processing efficiency improvements can save time and power in the processing system, which can be particularly beneficial for mobile edge processing devices. Furthermore, further optimizations are possible when the intermediate memory contains fully sparse data, as described below.

[0091] Hash table convolution

[0092] It is possible to further compress the sparse data representation of the table by creating a one-dimensional hash for each dimension of the weight kernel.

[0093] like Figure 6 As described in the text, without using zero entries to fill in... In the case of each complete column, it is possible to represent each dimension in the data (e.g., d in this example). 0 and d 1 This generates a one-dimensional list of hash pointers (e.g., 602 and 604). Note that "list of hash pointers" is used here to describe a general data structure representing one-dimensional or multi-dimensional structures.

[0094] use Figure 1B , 2A In the example in 2B, where K=5 and D=2, a list of hash pointers {L} can be defined. o ,L1}, ​​where when D=2, each list points to dimension The value. In this example, the two hash pointer lists {L o First, construct L1. o The row pointers and column pointers of L1. After column-by-column folding and multiplying each column by the corresponding kernel value, in L... o Perform line-by-line folding to remove all empty entries. In general, this can be {L o ,L1,…,L D-1 Generates a list of hash pointers, which means that hash table convolutions can be applied to data with higher dimensions than 2D data, such as in... Figure 6 In the example.

[0095] Then, to process the tabular data representation, L1 is multiplied by the kernel index k. The corresponding scalar entries. Thus, for example:

[0096]

[0097] Because {L o L1} is a list of pointers, therefore other lists of pointers L o The dereferenced value is automatically updated with the corresponding kernel weight. For example, Finally, all the pointer entries in the list of hash pointers in L0 are summed, for example, according to the following formula, to generate the tabular convolution output:

[0098]

[0099] Where i is L 0,k The index of each element.

[0100] Therefore, using a list of hash pointers further reduces computational complexity, which in turn reduces processing time and memory requirements.

[0101] Example test results and efficiency improvements

[0102] In testing, table convolutions implemented with known convolutional architectures (such as ResNet) beneficially reduced computational complexity and model size by at least 20%, without compromising accuracy. In testing, all kernel weights were learned via stochastic gradient descent relative to k=0 kernel weight elements with a value of 1.

[0103] Furthermore, tests show that compared to a square kernel with the same width (e.g., 3 pixels wide and high, such as...) Figure 1A Compared to the kernel 100A in the middle, non-square kernels (such as...) Figure 1B The kernel 100B in the example leads to further improvements in complexity and memory utilization. Thus, for example, kernels with non-square kernels (such as...) Figure 1B A table convolution (in the middle) can outperform a square kernel (such as...) with the same outer width (e.g., the widest row is three units wide and the highest column is three units high). Figure 1A (in Chinese) table convolution.

[0104] Table 1 below provides a further comparison of computational complexity based on the methods described in this paper:

[0105]

[0106] Table 1: Comparison of Complexity

[0107] In Table 1, C refers to the number of channels and S = HW, where H and W are the height and width of the input data (e.g., a tensor), respectively. Furthermore, in Table 1, A = S / 100 is used as an example, where a density of 1 / 10 is assumed in each of the two dimensions, such as for sparse data from various types of sensors (including event cameras). Additionally, R is defined as the average (ratio) number of neighbors covered by the kernel relative to the central "active" element. Moreover, when using hash table convolutions with skip connections, the kernel size K does not directly affect the number of multiplications; instead, R becomes the determining factor for the number of multiplications (or the size of the activation memory).

[0108] As indicated in Table 1, using tabular convolutions advantageously reduces the number of multiplications and model size compared to conventional depthwise separable dense convolutions, since S = HW >> A, where H is the height of the input data (e.g., in pixels) and W is the width of the input data (e.g., in pixels).

[0109] Example methods for performing table convolution

[0110] Figure 7 Example method 700 for performing table convolution is described.

[0111] Method 700 begins in step 702, performing a tabular operation on the input data to generate a tabular representation of the input data (e.g., as shown in the table). Figure 3 Step 302 and Figure 5 (504 in the middle).

[0112] Method 700 then proceeds to step 704, using a tabular representation of the input data to perform a convolution operation to generate a convolution output (e.g., as shown in the image). Figure 3Step 304 and Figure 5 (506 in the middle).

[0113] In some embodiments, method 700 further includes: determining that the sparsity of the input data for the convolutional layer of the machine learning model is greater than a threshold sparsity value.

[0114] In some embodiments of method 700, performing the convolution operation includes: performing a matrix multiplication between the weight tensor and a tabular representation of the input data to generate a convolution output, such as... Figure 3 The (X) depicted T W) and Figure 5 The (X) depicted T′ W′).

[0115] In some embodiments of method 700, performing the tabulation operation includes: using an indexing rule to populate a tabular representation of the input data, wherein the indexing rule defines multiple index values ​​based on the relationship between active elements of the input data and multiple elements in the input data adjacent to the active elements.

[0116] In some implementations of method 700, the sparsity of the convolutional output of the convolutional layer is the same as the sparsity of the input data of the convolutional layer, such as... Figure 4A and 4C As depicted in the example.

[0117] In some embodiments, method 700 further includes: removing active point vector components from a tabular representation of the input data before performing the convolution operation, and summing the active point vector components with the convolution output to generate the convolutional layer output. For example, as described above regarding Figure 5 As described, X can be removed from the tabular representation. active To form X T′ And X can be connected via a jump link 502. active Add it back.

[0118] In some embodiments, method 700 further includes: removing active point vector components from a tabular representation of the input data before performing the convolution operation; and generating a plurality of one-dimensional hash pointer lists based on the tabular representation, such as those described above. Figure 6 As described. In such cases, the convolution operation includes: multiplying the input value associated with each pointer in a first list of a plurality of one-dimensional hash pointers by an associated scalar weight value based on the kernel index of that value; and summing the weighted input values ​​associated with each pointer in a second list of a plurality of one-dimensional hash pointers to generate a convolution output.

[0119] In some embodiments, method 700 further includes: determining a loss value associated with the convolutional output; and updating a plurality of weights associated with the convolutional layer of the machine learning model based on the loss value.

[0120] In some embodiments of method 700, the convolution operation includes a depthwise separable convolution operation.

[0121] In some embodiments of method 700, the input data includes sparse image sensor data. In some embodiments, the input data includes point clouds, such as sparse light detection and ranging (LiDAR) sensor data.

[0122] Example processing system for performing hardware-based speech activity detection

[0123] Figure 8 An example processing system 800 configured to perform table convolutions (such as the table convolutions described herein, for example, with respect to Figures 2-6) is depicted.

[0124] The processing system 800 includes a central processing unit (CPU) 802, which in some examples may be a multi-core CPU. Instructions executed at the CPU 802 may be loaded, for example, from the program memory associated with the CPU 802 or from memory partition 824.

[0125] The processing system 800 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, a multimedia processing unit 810, and a wireless connectivity component 812.

[0126] NPUs (such as the 808) are generally dedicated circuits configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0127] NPUs (such as the 808) can be configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs can be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, multiple NPUs can be part of a dedicated neural network accelerator.

[0128] An NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0129] NPUs designed to accelerate training are typically configured to speed up the training of new models. This involves a highly computationally intensive process: taking an existing dataset (often labeled or sublabeled), iterating over the dataset, and then tuning model parameters (such as weights and biases) to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through the layers of the model and determining gradients to reduce prediction errors.

[0130] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data segments as input and rapidly process them through an already trained model to generate model outputs (e.g., inference).

[0131] In one implementation, the NPU 808 is part of one or more of the CPU 802, GPU 804, and / or DSP 806.

[0132] In some examples, the wireless connectivity component 812 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 812 is further connected to one or more antennas 814.

[0133] The processing system 800 may also include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor, and / or a navigation processor 820, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0134] The processing system 800 may also include one or more input and / or output devices 822, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0135] In some examples, one or more processors of the processing system 800 may be based on the ARM or RISC-V instruction set.

[0136] The processing system 800 also includes a memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 824 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 800.

[0137] Specifically, in this example, the memory 824 includes a determination component 824A, a tabulation component 824B, a convolution component 824C, a vector-matrix multiplication component 824D, a hashing component 824E, an indexing component 824F, a training component 824G, and model parameters 824G. The depicted components, as well as other undepicted components, can be configured to perform various aspects of the methods described herein.

[0138] Generally, the processing system 800 and / or its components can be configured to perform the methods described herein. It is worth noting that when the processing system is configured primarily for training machine learning models using tabular convolutions, certain aspects such as multimedia 810, wireless connectivity 812, antenna 814, sensor 816, ISP 818, and navigation 820 may be omitted.

[0139] Example Terms

[0140] Examples of implementations are described in the following numbered clauses.

[0141] Clause 1: A method comprising: performing a tabular operation on input data to generate a tabular representation of the input data; and using the tabular representation of the input data to perform a convolution operation to generate a convolution output.

[0142] Clause 2: The method of Clause 1 further includes performing a tabulation operation based on determining that the sparsity of the input data is greater than a threshold sparsity value.

[0143] Clause 3: The method of any of Clauses 1-2, wherein performing a convolution operation includes performing matrix multiplication between a weight tensor and a tabular representation of the input data to generate a convolution output.

[0144] Clause 4: The method of any of Clauses 1-3, wherein performing the tabular operation includes: populating a tabular representation of the input data according to an indexing rule, wherein the indexing rule defines multiple index values ​​based on the relationship between an active element of the input data and multiple elements in the input data adjacent to that active element.

[0145] Clause 5: The method of any of Clauses 1-4, wherein the sparsity of the convolution output is the same as the sparsity of the input data.

[0146] Clause 6: The method of any one of Clauses 1-5 further includes: removing active point vector components from the tabular representation of the input data before performing the convolution operation; and summing the active point vector components with the convolution output to generate the convolutional layer output.

[0147] Clause 7: The method of Clause 4 further includes: removing active point vector components from a tabular representation of the input data before performing the convolution operation; and generating a plurality of one-dimensional hash pointer lists based on the tabular representation, wherein the convolution operation includes: multiplying an input value associated with each pointer in a first hash pointer list of the plurality of one-dimensional hash pointer lists by a scalar weight value based on a kernel index of the input value; and summing weighted input values ​​associated with each pointer in a second hash pointer list of the plurality of one-dimensional hash pointer lists to generate a convolution output.

[0148] Clause 8: The method of any of Clauses 1-7 further includes: determining a loss value associated with the convolutional output; and updating multiple weights associated with the convolutional layers of the machine learning model based on the loss value.

[0149] Clause 9: The method of any one of Clauses 1-8, wherein the convolution operation includes a depthwise separable convolution operation.

[0150] Clause 10: The method of any of Clauses 1-9, wherein the input data includes sparse image sensor data.

[0151] Clause 11: The method of any of Clauses 1-10, wherein the input data includes sparse light detection and ranging (LiDAR) sensor data.

[0152] Clause 12: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method pursuant to any one of Clauses 1-11.

[0153] Clause 13: A processing system comprising means for performing a method according to any one of Clauses 1-11.

[0154] Clause 14: A non-transient computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of Clauses 1-11.

[0155] Clause 15: A computer program product implemented on a computer-readable storage medium, comprising code for performing a method pursuant to any of Clauses 1-11.

[0156] Additional considerations

[0157] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein are not intended to limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made to the function and arrangement of the elements discussed without departing from the scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Moreover, features described with reference to some examples may be combined in others. For example, any number of aspects set forth herein may be used to implement an apparatus or practice a method. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that supplement or differ from the aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be implemented by one or more elements of the claims.

[0158] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” should not be construed as superior to or better than the others.

[0159] As used herein, the phrase “at least one of” a list of items refers to any combination of those items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0160] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertaining, and the like. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and the like. Moreover, "determine" can include parsing, selecting, choosing, building, and the like.

[0161] The methods disclosed herein include one or more steps or actions for implementing the method. These method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the above methods can be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations illustrated in the drawings are present, these operations may have corresponding paired means with similar numbers plus functional components.

[0162] The following claims are not intended to be limited to the embodiments shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean “one and only one” (unless specifically stated so), but rather “one or more.” Unless specifically stated otherwise, the term “some / a” refers to one or more. No element of the claims should be interpreted in accordance with the provisions of 35 U.S.SC §112(f) unless the element is expressly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “steps for…”. Elements of all aspects described throughout this disclosure that are now or hereafter known to a person skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be donated to the public, whether or not such disclosure is expressly stated in the claims.

Claims

1. A method comprising: Perform a tabular operation on the input data to generate a tabular representation of the input data, wherein performing the tabular operation includes: populating the tabular representation of the input data according to an index rule, wherein the index rule defines multiple index values ​​based on the relationship between active elements of the input data and multiple elements in the input data adjacent to the active elements; Before performing the convolution operation, the active point vector components are removed from the tabular representation of the input data; The convolution operation is performed using the tabular representation of the input data to generate a convolution output; and The active point vector components are summed with the convolution output to generate the convolutional layer output. The input data includes sparse image sensor data or sparse light detection and ranging LiDAR sensor data.

2. The method of claim 1, further comprising: The tabulation operation is performed based on the determination that the sparsity of the input data is greater than a threshold sparsity value.

3. The method of claim 1, wherein performing the convolution operation comprises: Perform matrix multiplication between the weight tensor and the tabular representation of the input data to generate the convolution output.

4. The method of claim 1, wherein, The sparsity of the convolution output is the same as the sparsity of the input data.

5. The method of claim 1, further comprising: Multiple lists of one-dimensional hash pointers are generated based on the tabular representation. The convolution operation includes: Multiply the input value associated with each pointer in the first hash pointer list of the plurality of one-dimensional hash pointer lists by a scalar weight value based on the kernel index of the input value; and The weighted input values ​​associated with each pointer in the second list of the plurality of one-dimensional hash pointer lists are summed to generate the convolution output.

6. The method of claim 1, further comprising: Determine the loss value associated with the convolution output; as well as The loss value is used to update multiple weights associated with the convolutional layers of the machine learning model.

7. The method of claim 1, wherein the convolution operation includes a depthwise separable convolution operation.

8. A processing system, comprising: Memory including computer-executable instructions; as well as One or more processors, the one or more processors being configured to execute the computer-executable instructions and enable the processing system to: A tabular operation is performed on the input data to generate a tabular representation of the input data, wherein, in order to perform the tabular operation, the one or more processors are further configured to populate the tabular representation of the input data according to an indexing rule, wherein the indexing rule defines a plurality of index values ​​based on the relationship between active elements of the input data and a plurality of elements in the input data adjacent to the active elements; Before performing the convolution operation, the active point vector components are removed from the tabular representation of the input data; The convolution operation is performed using the tabular representation of the input data to generate a convolution output; as well as The active point vector components are summed with the convolution output to generate the convolutional layer output. The input data includes sparse image sensor data or sparse light detection and ranging LiDAR sensor data.

9. The processing system of claim 8, wherein the one or more processors are further configured to perform the tabulation operation based on determining that the sparsity of the input data is greater than a threshold sparsity value.

10. The processing system of claim 8, wherein, in order to perform the convolution operation, the one or more processors are further configured to: perform matrix multiplication between the weight tensor and the tabular representation of the input data to generate the convolution output.

11. The processing system of claim 8, wherein, The sparsity of the convolution output is the same as the sparsity of the input data.

12. The processing system of claim 8, wherein, The one or more processors are further configured to: Multiple lists of one-dimensional hash pointers are generated based on the tabular representation. In order to perform the convolution operation, the one or more processors are further configured to: The input value associated with each pointer in the first hash pointer list of the plurality of one-dimensional hash pointer lists is multiplied by a scalar weight value based on the kernel index of the input value; as well as The weighted input values ​​associated with each pointer in the second list of the plurality of one-dimensional hash pointer lists are summed to generate the convolution output.

13. The processing system of claim 8, wherein, The one or more processors are further configured to: Determine the loss value associated with the convolution output; and The loss value is used to update multiple weights associated with the convolutional layers of the machine learning model.

14. The processing system of claim 8, wherein the convolution operation includes a depth-separable convolution operation.

15. A non-transitory computer-readable medium comprising computer-executable instructions, which, when executed by one or more processors of a processing system, cause the processing system to perform a method, the method comprising: Perform a tabular operation on the input data to generate a tabular representation of the input data, wherein performing the tabular operation includes: populating the tabular representation of the input data according to an index rule, wherein the index rule defines multiple index values ​​based on the relationship between active elements of the input data and multiple elements in the input data adjacent to the active elements; Before performing the convolution operation, the active point vector components are removed from the tabular representation of the input data; The convolution operation is performed using the tabular representation of the input data to generate a convolution output; and The active point vector components are summed with the convolution output to generate the convolutional layer output. The input data includes sparse image sensor data or sparse light detection and ranging LiDAR sensor data.

16. The non-transient computer-readable medium of claim 15, wherein the method further comprises: The tabulation operation is performed based on the determination that the sparsity of the input data is greater than a threshold sparsity value.

17. The non-transient computer-readable medium of claim 15, wherein performing the convolution operation comprises: Perform matrix multiplication between the weight tensor and the tabular representation of the input data to generate the convolution output.

18. The non-transient computer-readable medium of claim 15, wherein, The sparsity of the convolution output is the same as the sparsity of the input data.

19. The non-transient computer-readable medium of claim 15, wherein, The method further includes: Multiple lists of one-dimensional hash pointers are generated based on the tabular representation. The convolution operation includes: Multiply the input value associated with each pointer in the first hash pointer list of the plurality of one-dimensional hash pointer lists by a scalar weight value based on the kernel index of the input value; and The weighted input values ​​associated with each pointer in the second list of the plurality of one-dimensional hash pointer lists are summed to generate the convolution output.

20. The non-transient computer-readable medium of claim 15, wherein, The method further includes: Determine the loss value associated with the convolution output; and The loss value is used to update multiple weights associated with the convolutional layers of the machine learning model.

Images