Computationally efficient neural convolutional network

Abstract

A computer-implemented method for the efficient computation of convolution operations, comprising the following steps: Receiving a tensor (X ∈ R) c in×k×w ) of input data to be processed and at least one filter (F j ∈ R c in×k×k , j ∈ {1.2, ..., c cut}) and initialize a locality-sensitive hash function. Then repeat the following steps for each patch in the tensor: Slice the current receptive field into a row (c in ) of matrices (k - b × k + b). Apply locality-sensitive hashing to each of the matrices to determine a hash representation for each matrix. Merge the matrices with substantially the same hash representation to form a new matrix. Create a reduced tensor (c in,reduced× k | b × k | b) by arranging the merged matrices in series; merging the filter coefficients in the same order in which the matrices were merged, and convolving the reduced subtensor with the merged filter.

Description

[0001] The invention relates to a method for the efficient calculation of convolutions by location-sensitive hashing, as well as a computer program, a machine-readable storage medium and a training system. State of the art

[0002] Reducing the computational costs of convolutional neural network (CNN) architectures for their application in an autonomous driving scenario as image classifiers or backbones for semantic segmentation is important for high computing power.

[0003] To reduce computational costs, the approach described in the literature, Dynamic Dual Gating Neural Networks by Fanrong Li et al. (available online at: https: / / openaccess.thecvf.com / content / ICCV2021 / papers / Li_Dynamic_Dual_Gating_Neural_Networks_ICCV_2021_paper.pdf), is one example. The authors employ a trainable method for detecting redundancies in the spatial and channel dimensions of a convolution module during the training of the underlying CNN. After the training procedure, their method can be used to dynamically reduce computational costs depending on the input image. This is achieved by generating spatial and channel masks from the trained additional parameters and effectively hiding redundant information from the convolution operation to reduce (theoretical) computational costs.

[0004] Other methods, such as Pruning Filters for Efficient ConvNets by Hao Li et al. (available online at: https: / / arxiv.org / pdf / 1608.08710.pdf), prune the filters of CNNs. This can be done after training by applying certain metrics (such as the L2 norm of filters) to determine whether a filter / channel can be pruned safely, i.e., without significantly impacting accuracy / model performance. However, these methods require a fine-tuning step after pruning to restore any lost accuracy.

[0005] The known approaches are either not parameter-free, requiring a specific training procedure (e.g., generating masks as seen in Dynamic Dual Gating Neural Networks), or not training-free (e.g., requiring a fine-tuning step after pruning). This means that existing trained models must either be completely retrained from scratch or at least fine-tuned for the pruning approaches to be usable (accuracy / FLOP trade-off). This can lead to several problems. First, the need for retraining or fine-tuning causes increased energy consumption and time expenditure. Second, the fine-tuning step can lead to catastrophic forgetting, where the model drastically loses performance on the task for which it was originally trained, especially when fine-tuned on data different from that used in training.Finally, the need for additional training creates a dependency on the availability of the dataset used to train the underlying CNN.

[0006] The present invention eliminates these limitations, as it requires neither a fine-tuning nor a retraining step and is completely parameter-free and data-free. Therefore, it can serve as a plug-and-play replacement for convolutional models in CNN architectures and directly reduce computational costs in the form of FLOPs (floating-point operations, which can be interpolated, for example, from the number of multiplication-accumulation operations, MACs). Advantages of the invention

[0007] This invention can be considered a structured pruning method, or more specifically, a channel pruning method. More precisely, the inventors propose a convolution module based on locality-sensitive hashing (LSH) that can serve as a plug-and-play replacement for any regular convolution module and instantly reduces FLOPs during inference. The invention thus eliminates restrictive requirements for reducing the computational complexity of CNNs, such as the need for retraining and / or fine-tuning, or pruned models and training data dependencies. It provides a plug-and-play pruning solution for instantly reducing FLOPs in CNN architectures, whereas the prior art requires time-consuming retraining and fine-tuning steps. Disclosure of the invention

[0008] In a first aspect, a method for the efficient computation of 2- or 3-dimensional convolution operations, e.g., in a 3-dimensional tensor, is proposed. Computational efficiency is achieved by dynamically reducing the number of channels in individual parts of the tensor. The method is preferably applied to a convolution layer of a neural network.

[0009] The process begins with a step of receiving a tensor (X∈R). cin×h×w ) of input data to be processed in particular by the convolution layer and at least one filter (F j ∈ R cin×h×h , j ∈ {1,2,..., c out}) in particular the convolution layer and initialization of a location-sensitive hash function. It should be noted that the order of the "receive" and "initialize" steps can be arbitrarily interchanged. The hash function can remain unchanged after initialization. The filter is preferably a 2- or 3-dimensional filter that includes a plurality (c in ) comprises 2-dimensional filter kernels (k × k) arranged in a row along a third dimension. The number of 2- or 3-dimensional kernels can be equal to the number of matrices that the tensor comprises along at least one of its dimensions; for example, the number of 2-dimensional kernels is c. in the number of arranged matrices c in of the tensor. It should be noted that the matrices merging described below is then performed along the dimension of the tensor, which corresponds to the number of kernels of the filter.

[0010] Then a loop follows, in which the following steps are performed for a plurality of 3-dimensional patches X. (p) ∈ R cin×(k+b)×(k+b)of the tensor. A 3-dimensional patch refers to a specified subtensor of the input tensor. Patches are chosen to have a size of (k + b × k + b), where, for example, b - 2, and maintain and overlap two pixels / units to the neighbor along a second and third dimension of the tensor, with patches along a first dimension having the same size as the tensor. The second and third dimensions can be referred to as spatial dimensions. Extracting a patch from the tensor is equivalent to tiling the input along the spatial dimension into regions of size (k × k), while maintaining the overlap necessary for the convolution operation with kernels of (k × k). In other words, the patches along the second and third dimensions of the tensor are cut out of the tensor. It is worth noting that the second and third dimensions of the tensor span matrices.The first dimension of the tensor can be viewed as the dimension in which the matrices are arranged, with the different matrices along the first dimension being referred to as channels or slices. Thus, the number of channels in the patch is identical to the number of channels in the tensor. It's worth noting that this concept can also be extended to a 3-dimensional filter, where the patches of the tensor then constitute a subtensor of the tensor.

[0011] The first step of the loop involves slicing the current patch along the first dimension into a row (c in) of matrices (k + b × k + b). This is followed by a step of applying locality-sensitive hashing to each of the matrices to determine a hash representation for each matrix. This is followed by a step of combining the matrices with essentially the same hash representation to form a new matrix. By 'essentially', it is understood that slightly different hash representations are assigned to the same hash bucket, but these are defined by a predefined number of hyperplanes. This is followed by a step of constructing a reduced tensor (c). in,reduced × k | b × k | b), by arranging the merged matrices in a row, in particular such that the two-dimensional matrices are opposite each other to obtain the three-dimensional reduced tensor. Then follows a step of merging along dimension c. inThe filter kernels are arranged in the same order in which the matrices were convolved. Generally, during the convolution process, each filter kernel, specifically each kernel (k × k), is placed at a given position in the filter along dimension c. in of the filter, on the matrix in the same position along dimension c in of the subtensor. If at least two matrices are combined at given positions, the corresponding filter kernels at the same positions in the filter are analogously combined, in particular summed. In other words, the corresponding kernels of the combined matrices are combined.

[0012] This is followed by a step of convolving the reduced subtensor with the merged filter. It should be noted that for this convolving step, the convolution is then performed by sliding the reduced filter over the enlarged receptive field and applying the convolution during the overthrust.

[0013] After the loop has finished, an optional step of outputting the result of all convolution operations for each patch can be performed.

[0014] It is proposed that the step of initializing the location-sensitive hashing be performed depending on an estimated number of hyperplanes for the location-sensitive hashing, where the number is estimated depending on a computational resource of the computing unit on which the convolutional layer is executed. Preferably, the number of hyperplanes is at least 20 and should not exceed 60, preferably 40.

[0015] In a further aspect of the present invention, a computer-implemented method for using the neural network with efficient convolution according to the first aspect of the invention as a classifier for classifying sensor signals is proposed. The classifier is applied using the method according to one of the preceding aspects of the invention, which comprises the following steps: receiving a sensor signal comprising data from the image sensor, determining an input signal that depends on the sensor signal, and inputting the input signal into the classifier to obtain an output signal that characterizes a classification of the input signal.

[0016] In a further aspect of the invention, a control system for operating the actuator is proposed. The control system comprises the classifier, which is used according to one of the preceding aspects of the invention, and is further configured to operate the actuator according to an output of the classifier.

[0017] Embodiments of the invention are discussed in more detail with reference to the following figures. The figures show: Fig. 1 a schematic flowchart of an embodiment of the invention; and Fig. 2 a schematic overview of a framework of an embodiment of the present invention.

[0018] The invention describes a dynamic / online pruning scheme for convolutional neural networks (CNNs), such as ResNets or VGGs. CNNs are a class of (deep) neural networks frequently used for computer vision or image processing tasks. They are characterized by the use of convolution filters that perform (discrete) convolutions on the input signal in a sliding-window fashion to generate an output signal. A convolution module can contain multiple convolution filters. Many such modules are then chained together, often with other modules, for example, batch normalization or activation functions (e.g., ReLU), to form a convolution layer. Finally, a typical CNN like ResNet often uses multiple convolution layers of different sizes to perform tasks such as image classification or, when used as a "backbone" (i.e.,Preprocessing step) / feature extractor, also to perform semantic segmentation or object recognition. In one aspect of the invention, a method for pruning channels (i.e., "slices") of the input signals that are fed into a convolution module is described in order to reduce the overall computational cost of processing the input while maintaining high performance.

[0019] The main aspect of the invention revolves around reducing unnecessary computations when performing convolution operations on high-dimensional inputs, since the input signal often contains redundancies in the form of similar channels or slices within the channels. In a regular convolution operation, however, all channels are treated as equally important, leading to high computational costs. The invention uses a scheme, referred to below as location-sensitive hashing, to detect these redundancies. This is an approximate nearest-neighbor search method that can cluster input slices based on their similarity. Given this clustering of channels, the redundant information in the input signal can be combined / minimized by performing cost-effective operations such as averaging or summing these channels. Convolution can then be applied to the reduced input.Although performing the hashing and merging steps incurs additional costs for the overall module, these operations, in combination with convolution on the reduced input, are still significantly cheaper than performing regular convolution on an input with redundancies. This behavior is primarily due to the fact that the convolution operation itself is expensive, as it performs many element-wise multiplications and additions. Furthermore, this convolution is repeated for each convolution filter in a convolution module. Therefore, filtering out redundancies and calculating their mean / sum upfront results in a small "one-time cost" that, in turn, drastically reduces the subsequent repeated costs of computing convolutions.

[0020] The invention has direct applications in autonomous driving, robotics, and other resource-constrained mobile applications of convolutional neural networks. Particularly for processing image sensor data, the size and computational complexity of AI models remain a problem. CNNs are often too large to fit on mobile hardware in terms of memory and latency requirements. Since model FLOPs directly measure how many computations the model must perform to produce an output from a given input, reducing FLOPs offers the potential for savings in runtime (latency) and energy consumption on mobile devices.

[0021] More specifically, one could envision applications such as system-on-a-chip video / image processing sensors, where available computing resources are limited. Furthermore, mobile applications often require the sharing of available resources among multiple computationally intensive modules. This can limit the maximum power consumption or latency that a model may exhibit. The present invention offers the possibility of reducing both of these limitations by lowering the computational costs. Moreover, other methods may be unsuitable for use due to data dependencies or retraining requirements. For example, models that are continuously trained on video data streams cannot simply be retrained from scratch or easily fine-tuned to achieve a reduction in computational costs through pruning.Other models may also require expensive training procedures that, in the worst case, would need to be repeated to successfully prune them. Our method avoids these limitations and offers a significant reduction in FLOPs.

[0022] Preferably, the invention according to the claims is used for analyzing data obtained from a sensor. The sensor can determine measurements of the environment in the form of sensor signals, which may be provided, for example, by digital images, e.g., video, radar, LiDAR, ultrasound, motion, thermal images.

[0023] Preferably, the invention according to the claims is used for classifying sensor data, detecting the presence of objects in the sensor data, or performing semantic segmentation of the sensor data, e.g., with regard to traffic signs, road surfaces, pedestrians, and / or vehicles. This is carried out based on low-level features (e.g., edges or pixel attributes for images).

[0024] The following section provides a detailed description of the present invention for channel pruning in convolutional neural networks. The approach is described for any convolutional module, as it can be applied to a convolutional module in a given network. Note the distinction between the use of convolution with filters of kernel size k >= 2 and filters of kernel size k = 1, also referred to as 1x1 convolutions or pointwise convolutions.

[0025] When initializing the proposed convolution module, we create hash functions by generating a set of L random hyperplanes through their normal vectors. The vectors are generated with a sparse distribution to save computational costs in the hashing process.

[0026] The location-sensitive hashing process is advantageous because it provides a computationally efficient way to perform an approximate nearest neighbor search. We use the "random projections" scheme, which is particularly well-suited for use in convolutional layers. However, alternatives such as MinHash or Winner Takes All Hash (WTA Hash) could also be viable choices. Case k >= 2:

[0027] When using convolution filters with kernel size k >= 2, our goal is to prune larger patches, rather than smaller context windows, observed by the convolution filter at each filter position p in the sliding window process. To achieve this, we hash a representation of each channel of the given patch in a locality-sensitive manner and cluster essentially similar channels in the hash buckets. We are then able to merge all approximately similar channels, reducing the channel dimension, or depth, of the input signal (patch). Similarly, we merge the corresponding channels of all convolution filters in this module on the fly, reducing their depth by the same amount. It is worth noting that the degree of depth reduction depends on the input signal, allowing for high compression ratios when the input is highly redundant.Furthermore, the compression ratio can also vary between filter positions, so that redundant image regions achieve a particularly high compression.

[0028] While the merging operation itself is essential to our procedure, the specific choice of merging operation can be changed. Instead of calculating the mean of the input channels, one can also attempt to calculate the element-wise maximum or the element-wise median.

[0029] Finally, we are able to perform a regular convolution operation between the reduced input and the reduced filters and produce an output of regular size that can be used immediately by the next module in the chain without any modifications. Case k = 1:

[0030] When using pointwise convolutions, the hashing process must be modified to function correctly. Therefore, the input is rasterized into non-overlapping 3×3 patches in the spatial dimension, padding them with zeros where necessary. Each patch can then be hashed similarly to the previous case k >= 2, resulting in a reduction of the input channel dimension and the filter dimensions. Within each reduced patch, we can perform 1×1 convolution with filters that are merged by combining corresponding channels to the same depth as the input patch. The output, in turn, retains the shape that the underlying convolution module would normally produce.

[0031] Location-sensitive hashing is a common approach for fast, approximate nearest-neighbor searches in high-dimensional spaces. A hash function h: R d → N is locality-sensitive if similar vectors exist in the input domain x, y ER dwith a high probability, the same hash codes h(x) - h(y) will be received. This contrasts with regular hash schemes, which attempt to minimize hash conflicts by widely distributing the input data across their hash buckets. More formally, we need a measure of similarity in the input space and a suitable hash function h. A particularly suitable measure for use in convolutional architectures is cosine similarity, since convolving the (approximately) normalized kernel with the normalized input is equivalent to calculating its cosine similarity.

[0032] A preferred family of hash functions that group input data by cosine similarity is given by random projections (RP). These functions partition the high-dimensional input space by L random hyperplanes such that each input vector can be assigned to exactly one section of this partitioning, called a hash bucket. Determining the position of an input x ∈ R d Relative to all L hyperplanes, this is done by taking the scalar product with their normal vectors v1 ∈ R. d , l ∈ {1,2, ..., L}, whose entries are from a standard normal distribution N(0,1) to be drawn. By defining: hl:Rd→{0,1},hl(x):={1, if vl⋅x>00, otherwise, The result is binary information representing which side of the l-th hyperplane our input x lies on. The hyperparameter L determines the power of discrimination of this procedure and divides the input space R. d in a total of 2 LDifferent regions or hash buckets. By concatenating all individual functions h1, we obtain the RP hash function: h:Rd→{0,1}L,h(x)−(h1(x),…,hL(x))

[0033] It should be mentioned that h(x) is an L-bit binary code, which uses as an identifier exactly one of the 2 L Hash buckets are used. Accordingly, we can convert this code into an integer and effectively range the hash buckets from 0 to 2. L - Mark 1: h:Rd→{0,1,…,2L−1},h(x)=2L−1hl(x)−⋯+20h1(x)

[0034] Although LSH already drastically reduces the computational complexity compared to exact nearest-neighbor search, binary code generation still requires L ∗ d multiplications and L ∗ (d - 1) additions per input. To further reduce the cost of this operation, the method described in Ping Li, Trevor Hastie, and Kenneth Church, "Very Sparse Random Projections," Vol. 2006, pp. 287–296, 08 2006. doi: 10, 1145 / 1150402, 1150436, can be used. Instead of using standard normally distributed vectors v l Alternatively, very sparsely populated vectors v can be used. l The normal vectors used are those containing only elements from the set {1,0, -1}. For a specific degree of sparsity s ∈ (0, 1), the normal vectors v l The hyperplane is randomly generated such that only an expected number of non-zero entries s occur. Suppose we choose s - 1 / 3, then our normal vectors v contain lExactly d / 3 are non-zero entries. Since the remaining entries are chosen to be either -1 or 1, the scalar product calculation reduces to L * (d / 3 - 1) additions and 0 multiplications. This allows us to exchange expensive multiplication operations for cost-effective additions.

[0035] After establishing LSH through sparse random projections as a computationally cost-effective way to find approximate nearest neighbors in high-dimensional spaces, our goal is now to use this method as a means of finding redundancies in the channel dimension of latent feature maps in convolution modules.

[0036] Formally, a convolution layer can be described as having several learned filters F j ∈ R cin×k×k , j ∈ {1, 2, ... c out} via the (filled) input feature map X ∈ R cn×h×wThe input is shifted and the discrete convolution is calculated at each point. Here, k is the kernel size, h and w denote the spatial dimensions of the input, and c in , c out describe the input and output channel dimensions.

[0037] For each patch p and the corresponding k + b, × k + b window of input X (p) ∈ R cin×k+h×k+h Many channels contain similar information. Nevertheless, all channels Xi(p)∈RR+2×R+2, i ∈ {1, 2, ..., c in} folded with their corresponding floating kernels to produce an output, ignoring existing redundancies.

[0038] We question this design choice and instead use redundant channels to save computations during the convolution operation. To retrieve these channels, we propose using the LSH scheme described above. More precisely, we flatten all Xi(p) on (k + 2) 2n-dimensional vectors are used to group similar channels. This set of vectors is then centered by the mean along the channel dimension and hashed by h, giving us a total of c in Hash codes received.

[0039] We then check which hash code occurs more than once, given that the LSH scheme determines that all elements in the same bucket are approximately similar. Consequently, by grouping the vector representations of Xi(p) through their hash code, sets of redundant feature map channels. These can then be combined to reduce the computational complexity of the convolution module.

[0040] It is particularly worth mentioning that our RP-LSH approach is invariant under the scaling of a given input vector. This means that k + b × k + b input channels of the same structure, but with different activation intensities, can still end up in the same hash bucket, effectively finding even more redundancies in the channel dimension.

[0041] After clustering redundant input channels Xi(p) In hash buckets, we can use this grouping to save FLOPs when performing the convolution operation.

[0042] To avoid repeated calculations for nearly identical channels, we reduce the size of each input context window by merging channels in the same hash bucket. The merging operation is performed by taking the average of all channels in a bucket. As a result, the number of remaining channels is reduced to c. inreduced < c in reduced.

[0043] This allows us to calculate a compression ratio: r:=1−cinreducedcin∈(0,1) to define the relative reduction of the feature map depth. It's worth noting that this ratio depends on the amount of redundancy in the input feature map X at filter position p. This allows us to dynamically prune channels, enabling different compression ratios across images and even in different regions of the same input.

[0044] Similar to reducing the depth of the input feature map, we introduce the corresponding channels of the convolution filters F. ltogether. However, instead of hashing the filter channels as well, we can simply merge these channels, which correspond to the folded input channels. This merging step is performed on the fly in each filter position p, maintaining the original weights for the next filter position. As a result, the one-to-one correspondence of channels remains intact, while simultaneously the filter depth remains at c. inreduced The number of computational steps required is reduced. It's worth noting that instead of taking the average of filter channels, we could simply add them together. In this way, a convolution between a reduced input window and a reduced filter maintains the same output intensity as the original convolution, while requiring fewer calculations.

[0045] The procedure can be defined by pseudocode, as follows:

[0046] Although hashing input channels and performing merge operations incur additional computational costs, the overall savings from calculating the convolution operation with reduced channel dimensions outweigh the additional effort.

[0047] The method presented above serves as a direct plug-and-play replacement for any regular convolution module in a CNN. It has only one hyperparameter, L, which determines the set of hyperplanes used in the LSH scheme. Its value directly determines a trade-off between compression ratio and preserved accuracy. When L is low, the random projections only roughly divide the input channels into groups. Although this leads to high compression ratios, the resulting output feature map can differ significantly from that of the full convolution operation, resulting in an overall reduction in model performance. Conversely, when L is large, we preserve almost full model performance at the cost of lower compression ratios. Therefore, the choice of L allows the generation of multiple model variants from an underlying base model, with the emphasis placed, for example, either on low FLOPs or on high performance.

[0048] The hyperparameter L, which determines the number of hyperplanes used in the hash scheme, directly acts as a control for the accuracy / FLOPs trade-off. It can be individually adjusted at any point in the network to find the desired balance between model and hardware performance.

[0049] Fig. Figure 1, in particular together with the preceding pseudocode, shows an algorithm of the approach described above, which includes a series of steps for processing input data of a convolution layer of a CNN by hashing and merging channels and merging filter kernels according to the merging of the channels before performing convolutions to generate the output tensor of, for example, the convolution layer.

[0050] The algorithm starts at step S11, 'Initializing a Hash Function', which initializes a hash function h. This function is used to hash representations of the data. Preferred choices of different hash functions have been discussed above.

[0051] In the next step S12, the algorithm enters a loop that iterates over each pixel in the input image or each data point of the feature map of the convolution layer. The loop traverses the patches p of the input image or feature.

[0052] In the next step, S13, within the loop, an empty list called HashCodes is created. This list is suitable for storing hash codes generated for each channel of the input data at the current pixel or data point.

[0053] In the next step S14 within the loop, there is another nested loop that iterates over each input channel i (from 1 to c).in ) iterates.

[0054] In the next step S15, a representation x is created for each input channel in the nested loop. i This is calculated by centering and flattening the channel values ​​at the current pixel p. This is done using the 'center' and 'flatten' operations. The result is a flattened vector.

[0055] In the next step S16, the algorithm then applies the hash function to the generated representation x. i It then generates a hash code. This hash code is appended to the HashCodes list.

[0056] In the next step, S17, after processing all channels at the current pixel, the algorithm combines these channels into a single bucket. This means that the algorithm combines the values ​​of all input channels at this pixel according to the calculated hash codes. The result is represented as Xreduced(p) saved.

[0057] In the next step S18, the algorithm combines the corresponding filter channels from the set of filters F based on the same hash codes. The result is denoted as F. reduced saved. Merging the filters is done in a similar way to merging the channels, except that instead of averaging the channels in a given bucket, the filters of the corresponding bucket are summed.

[0058] In the next step S19, the algorithm proceeds with the reduced input. Xreduced(p) and the reduced filters F reduced perform a convolution operation (e.g., a standard convolution) to obtain the output Y (p) to calculate the current pixel on the reduced input by convolution with the reduced filter.

[0059] Due to the two compressed data representations of the compressed filter and the compressed window, significantly fewer convolution operations need to be performed.

[0060] Finally, the algorithm returns the output tensor Y, which has the same dimensions as the input, but a reduced number of channels (c). cut ) exhibits.

[0061] Fig. Figure 2 shows an overview of the reduction of the context window and filters. Each patch of the input feature map is processed to find redundant channels. Detected redundancies are then merged, thereby reducing the depth of the sliding window and all convolution filters. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Dual Gating Neural Networks by Fanrong Li et al. (available online at: https: / / openaccess.thecvf.com / content / ICCV2021 / papers / Li_Dynamic_Dual_Gati ng_Neural_Networks_ICCV_2021_paper.pdf

[0003] Filters for Efficient ConvNets by Hao Li et al. (available online at: https: / / arxiv.org / pdf / 1608.08710.pdf

[0004] Ping Li, Trevor Hastie and Kenneth Church, Very Sparse Random Projections. Vol. 2006, ff. 287-296, 08 2006. doi: 10, 1145 / 1150402, 1150436

[0034]

Claims

[1] Computer-implemented method for the efficient calculation of convolutions, in particular a convolutional layer of a neural network, comprising the following steps: Receiving a tensor (X ∈ R cin×h×w ) of input data to be processed in particular by the convolutional layer and at least one filter (F j ∈ R cin×k×k , j ∈ {1,2, ... , c out}), in particular the convolutional layer, where the filter has several (c in ) filter kernels (k × k); Initialize a locality-sensitive hash function; repeat the following steps for a plurality of patches of the tensor (X): Slicing a current patch into a row (c in ) of matrices (k - b × k + b); applying locality-sensitive hashing to each of the matrices to determine a hash representation for each matrix; Merging the matrices with essentially the same hash representation into a new matrix. Creating a reduced tensor (c in,reduced × k | b × k | b) by arranging the merged matrices in series; Merging the filter kernels in the same order in which the matrices were merged; and Convolving the reduced subtensor with the merged filter. [2] The method of claim 1, wherein the step of initializing the locality-sensitive hashing is performed depending on an estimated number of hyperplanes for the locality-sensitive hashing, the number being estimated depending on a computing resource of the computing unit on which the convolution, in particular the convolution layer, is performed. [3] The method of claim 1 or 2, wherein the tensor is an image to be processed by the neural network or a feature map of a previous layer of the convolutional layer. [4] A method according to any preceding claim, wherein the neural network classifies its input based on the convolved reduced sub-sensor with the merged filter. [5] Method according to one of the preceding claims, wherein the convolution filter has a size of c in × k × k, where k ≥ 1 and k × k is the kernel size. [6] A method according to any one of the preceding claims, wherein the merging of the matrices is performed by averaging over the matrices having substantially the same hash representation or by taking a median or an element-wise maximum of the matrices having the same hash representation. [7] A method according to any one of the preceding claims, wherein the merging of the filter is performed by summing the filter kernels corresponding to the merged matrices having the same hash representations. [8] A method according to any one of the preceding claims, wherein the receptive field is artificially enlarged by at least one entry along each dimension (k - 1 × k + 1). [9] A computer program configured to cause a computer to perform the method according to any one of claims 1 to 8 with all its steps when the computer program is executed by a processor. [10] A machine-readable storage medium on which the computer program according to claim 9 is stored. [11] System configured to perform the method according to any one of claims 1 to 8.

Images