Convolutional-neural-network with high calculation efficiency

Abstract

To provide a computer-implemented method for effectively performing convolutional calculation, a program, a storage medium, and a training system.SOLUTION: The method receives a tensor of input data to be processed and at least one filter, and initializes a localized sensitive type hash function. In addition, the method repeats the steps of: forming a series of matrixes (cin)(k+b*k+b) by slicing a current patch in each patch in the tensor; determining a hash expression in each matrix by applying the localized sensitive type hash function to each matrix; merging the matrix having an essentially equivalent hash expression to a new matrix; creating a tensor (cin,reduced*k+b*k+b) contracted by the continuous arrangement of the merged matrix; and convoluting the matrix merged on a sub-tensor contracted by merging a filter kernel in an order similar to an order that the matrix is merged.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for efficiently computing convolutions via locality-sensitive hashing, as well as a computer program and a machine-readable storage medium and a training system. [Background technology]

[0002] When convolutional neural network architectures (CNNs) are applied to autonomous driving scenarios as image classifiers or semantic segmentation backbones, reducing the computational cost is crucial for enhancing the computational performance.

[0003] Regarding the reduction of computational costs, what is known from the literature is, for example, the method of "Dynamic Dual Gating Neural Networks" by Fanrong Li et al. (available online at: https: / / openaccess.thecvf.com / content / ICCV2021 / papers / Li_Dynamic_Dual_Gating_Neural_Networks_ICCV_2021_paper.pdf). The authors employ a trainable method to find redundancies in the spatial and channel dimensions of the convolutional modules during the training of the underlying CNN. After the training procedure, their method can be used to dynamically reduce the computational cost depending on the input image. This is done by generating spatial and channel masks from additional trained parameters, effectively hiding redundant information from the convolution operation to reduce the (theoretical) computational cost.

[0004] Other methods, such as "Pruning Filters for Efficient ConvNets" by Hao Li et al. (available online at: https: / / arxiv.org / pdf / 1608.08710.pdf), prune the filters of a CNN. This can happen after training using multiple metrics (such as the L2-norm of the filters) to determine whether the filters / channels can be pruned safely, i.e., without too much loss in accuracy / model performance. However, these methods require a fine-tuning step after pruning to recover the lost accuracy.

[0005] Known approaches are either not parameter-free (e.g., generation of masks as found in dynamic dual-gating neural networks) by requiring specialized training procedures, or are not training-free (e.g., requiring a fine-tuning step after the pruning step). This means that existing trained models need to be completely retrained from scratch, or at least fine-tuned so that the pruning approach is feasible (trade-off between accuracy and FLOPs). This creates various problems. First, the need for retraining or fine-tuning increases energy and time consumption. Second, the fine-tuning step can result in catastrophic forgetting, and the model loses a lot of performance on the task it was originally trained on, especially when fine-tuning on data different from the data used in training. Finally, the need for additional training creates a dependency on the availability of the dataset used to train the underlying CNN. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] “Dynamic Dual Gating Neural Networks” by Fanrong Li et al. (available online: https: / / openaccess.thecvf.com / content / ICCV2021 / papers / Li_Dynamic_Dual_Gating_Neural_Networks_ICCV_2021_paper.pdf) [Non-Patent Document 2] “Pruning Filters for Efficient ConvNets” by Hao Li et al. (available online: https: / / arxiv.org / pdf / 1608.08710.pdf) Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention eliminates these limitations because it does not require any fine-tuning or retraining steps and is completely parameter-free and data-free. It therefore serves as a plug-and-play replacement for convolutional modules in CNN architectures and can directly reduce computational costs in the form of FLOPs (e.g., floating-point operations that can be interpolated from the number of MACs, which are multiply-add operations). [Means for solving the problem]

[0008] Effect of the Invention The present invention can be seen as a method of structured pruning, or more specifically, as a method of channel pruning. More precisely, we propose a convolution module based on Locality-Sensitive Hashing (LSH), which can act as a plug-and-play replacement for any normal convolution module and instantly reduce FLOPs during inference. Thereby, the present invention lifts restrictive requirements for reducing the computational complexity of CNNs, such as the requirement of retraining and / or fine-tuning or pruned models and training data dependencies. While the prior art requires time-consuming retraining or fine-tuning steps, the present invention provides a plug-and-play pruning solution for instantly reducing FLOPs in CNN architectures.

[0009] In a first aspect, a method is proposed for efficiently computing 2D or 3D convolution operations, for example on a 3D tensor. Computational efficiency is achieved by dynamically reducing the number of channels of parts of said tensor individually. Preferably, the method is applied to a convolution layer of a neural network.

[0010] The method particularly focuses on the tensors (

number

number

[0011] This is followed by a loop in which multiple 3D patches of the tensor are

number

[0012] The first step of the loop slices the current patch along the first dimension to produce a set of matrices (cin )(k+b×k+b). This is followed by applying a locality sensitive hash to each of the matrices to determine a hashed representation of each matrix. This is followed by merging matrices with essentially equivalent hashed representations into a new matrix. By the term "essentially", it can be understood that slightly different hashed representations are assigned to the same hash bucket and are defined by a predefined number of hyperplanes. The merged matrices are then arranged consecutively to produce a reduced tensor (c in,reduced ×k+b×k+b), whereby, in particular, the two-dimensional matrices are placed behind each other, facing each other, to obtain a three-dimensional contracted tensor. Then, we merge the dimensions c of the filter kernel in the same order as the matrices were merged. in This is followed by a merging step along the filter dimension c in Each filter kernel at a given position in the filter along in , which is applied to matrices at similar positions along the filter. If at least two matrices at a given position are merged, corresponding filter kernels at similar positions in the filter are merged in an analogous manner, specifically summed together. In other words, corresponding kernels of the merged matrices are merged.

[0013] This is followed by a step of convolving the reduced subtensor with the merged filter. Note that in the convolution step, the convolution is then performed by sliding the reduced filter over the expanded receptive field and applying the convolution during the slide.

[0014] After the loop is completed, an optional step can be performed to output the results of all convolution operations for each patch.

[0015] It is proposed that the step of initializing the locality sensitive hashing is performed depending on an estimated number of hyperplanes for the locality sensitive hashing, which number is estimated depending on the computational resources of the computation unit on which the convolutional layer is executed. Preferably, the number of hyperplanes is at least 20 and should not exceed 60, preferably 40.

[0016] In a further aspect of the invention, a computer implemented method is proposed for using a neural network with efficient convolutions according to the first aspect of the invention as a classifier for classifying a sensor signal, said classifier being employed in a method according to any one of the previous aspects of the invention, comprising the steps of receiving a sensor signal comprising data from an imaging sensor, determining an input signal dependent on said sensor signal, and feeding said input signal to said classifier to obtain an output signal characterizing a classification of said input signal.

[0017] In a further aspect of the present invention, a control system for operating an actuator is proposed, said control system comprising a classifier employed according to any of the previous aspects of the present invention and configured to operate said actuator according to an output of said classifier.

[0018] Embodiments of the invention will now be described in more detail with reference to the following drawings. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic flow diagram of one embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic diagram of a framework for one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The present invention describes a dynamic / online pruning scheme for convolutional neural networks (CNNs) such as ResNet or VGG. CNNs are a type of (deep) neural network that are often used for computer vision or image processing tasks. They are characterized by the use of convolution filters that perform (discrete) convolutions in a sliding window manner on the input signal to generate an output signal. Multiple convolution filters may be included in one convolution module. Many such modules are then often concatenated together with other modules such as batch normalization or activation functions (e.g., ReLU) to form a convolution layer. Finally, typical CNNs such as ResNet often use multiple convolution layers of different sizes to perform tasks such as image classification, or even semantic segmentation or object detection if used as a "backbone" (i.e., pre-processing step) / feature extractor. In one aspect of the present invention, a method is described for pruning channels (i.e., "slices") of an input signal fed to a convolution module, thus reducing the overall computational cost of processing the input while maintaining high performance.

[0021] The main aspect of the invention is centered around reducing unnecessary computations when performing convolution operations on high-dimensional inputs, since the input signal often contains redundancies in the form of similar channels or slices within channels. However, in normal convolution operations, all channels are treated as being equally important, which makes them computationally expensive. To detect these redundancies, the invention uses the following scheme called locality-sensitive hashing. This is an approximate nearest neighbor search method that allows input slices to be clustered together based on their similarity. Given this clustering of channels, the redundant information in the input signal can be merged / reduced to a minimum by performing cheap operations such as taking the average or sum of these channels. The convolution can then be applied to the reduced input. Although performing the hashing and merging steps adds additional cost to the overall module, combining these operations with the convolution on the reduced input is much cheaper than performing a normal convolution on an input with redundancies. This behavior is mainly due to the fact that the convolution operation itself is expensive, performing many multiplications and additions per element. Moreover, this convolution is repeated for each convolution filter in one convolution module. Thus, by eliminating the redundancy and computing the average / sum up front, the "one-time cost" is small, and the repeated cost of computing subsequent convolutions is significantly reduced.

[0022] The present invention has direct application in the autonomous driving, robotics, and other resource-constrained mobile application domains of convolutional neural networks. In particular, for processing image sensor data, the size and computational complexity of AI models remains a challenge. CNNs are often too large in terms of memory and latency requirements to fit on mobile hardware. Since model FLOPs are a direct measure of the number of calculations that must be performed on a model to generate an output given an input, reducing FLOPs offers potential savings in terms of runtime (latency) and energy requirements of mobile devices.

[0023] More specifically, applications in terms of system-on-chip video / image processing sensors, where available computational resources are limited, can be envisaged. Moreover, mobile applications often need to share available resources between multiple computationally intensive modules. This may impose limitations on the maximum energy consumption or latency allowed for the model. The present invention offers the possibility to reduce both by reducing computational costs. Moreover, other methods may be practically unusable due to data dependency or retraining requirements. For example, models continuously trained on a stream of video data cannot simply be retrained from scratch, nor can they be easily fine-tuned to achieve computational cost reductions by pruning. Other models may also have expensive training procedures, which in the worst case require repeating them to successfully prune the training procedure. Our method circumvents these limitations and offers significant FLOP reductions.

[0024] Preferably, the claimed invention is used to analyze data obtained from sensors capable of determining measurements of the environment in the form of sensor signals that may be provided by digital images, for example video, radar, LiDAR, ultrasound, motion, thermal images, etc.

[0025] More preferably, the claimed invention is used to classify sensor data, to detect the presence of objects in the sensor data, or to perform semantic segmentation on the sensor data, e.g. with respect to traffic signs, road surfaces, pedestrians and / or vehicles, based on low-level features (e.g. image edges or pixel attributes).

[0026] In the following, a detailed description of the present invention for channel pruning in convolutional neural networks is given. The technique is applicable to any convolutional module in a given network, so it is described for any convolutional module. Note the difference between the case of using convolution with a filter of kernel size k>=2 and the case of using convolution with a filter of kernel size k=1, also called 1x1 convolution or point-wise convolution.

[0027] Once our proposed convolution module is initialized, we construct a hash function by generating a set of L random hyperplanes via their normal vectors. The vectors are generated sparsely to save computational cost in the hashing process.

[0028] Locality-sensitive hashing is advantageous because it serves as a computationally inexpensive way to perform approximate nearest neighbor searches. We use a "random projection" scheme that is particularly well suited for use in convolutional layers. However, alternatives such as MinHash or Winner Takes All Hash (WTA Hash) may also be viable choices.

[0029] Cases where k>=2: When using convolution filters with kernel size k>=2, the aim is to prune larger patches rather than the small context window observed by the convolution filters at all filter positions p of the sliding window process. To this end, we hash the representation of each channel of a given patch in a locality-sensitive manner, essentially clustering similar channels in hash buckets. All nearly similar channels can then be merged together such that the channel dimension or depth of the input signal (patch) is reduced. Similarly, corresponding channels of all convolution filters in this module are merged on the fly, and their depth is also reduced by a comparable amount. It should be noted that the degree of depth reduction depends on the input signal, allowing high compression ratios when the input is highly redundant. Furthermore, the compression ratio may vary between filter positions such that redundant image regions achieve particularly high compression.

[0030] The merging operation itself is essential to our method, but the particular choice of merging operation can be varied: instead of averaging the input channels, one could also try to compute the element-wise maximum or median.

[0031] Finally, a regular convolution operation can be performed between the reduced input and the reduced filter to produce a regular-sized output that can be immediately used by the next module in the chain without modification.

[0032] Example for k=1: When using point-wise convolutions, the hashing process needs to be modified to work correctly. For this, the input is rasterized into non-overlapping patches of size 3x3 in the spatial dimensions, zero-padded if necessary. Each patch can then be hashed as in the k>=2 case above, thereby reducing the input and filter channel dimensions. Inside each reduced patch, a 1x1 convolution can be performed with the merged filter to a depth equal to the input patch by merging corresponding channels. The output again preserves the shape that the underlying convolution module would normally produce.

[0033] Locality-sensitive hashing is a general method for approximate fast nearest neighbor search in high-dimensional spaces. For an input domain x,y∈R, d A hash function h:R if similar vectors in receive similar hash codes h(x) = h(y) with high probability. d →N is locality-sensitive. It contrasts with conventional hashing schemes, which reduce hash collisions to a minimum, by spreading the input data widely across its hash buckets. More formally, we need a similarity measure on the input space and a suitable hash function h. A particularly suitable measure for use in convolutional architectures is the cosine similarity, since convolving (approximately) normalized kernels with normalized inputs is equivalent to computing their cosine similarity.

[0034] One favored family of hash functions that groups input data by cosine similarity is given by random projection (RP). These functions partition the high-dimensional input space through L random hyperplanes, so that each input vector can be assigned to exactly one of the sections of this partition, called hash buckets. For an input x∈R for all L hyperplanes, d Determining the position of the normal vector v l ∈R d,l∈{1,2,...,L}, whose entries are drawn from a standard normal distribution N(0,1).

[0035] Define the following: (Formula 1)

number

[0036] We obtain binary information representing on which side of the l-th hyperplane the input x lies. The hyperparameter L controls the discriminatory power of the method and is the input space R d A total of 2 L For every individual function h l RP hash function is obtained by concatenating (Formula 2) h:R d →{0,1} L ,h(x)=(h 1 (x),...,h L (x)

[0037] h(x) is an L-bit binary code, L Note that each of these functions as an identifier for exactly one of the hash buckets. Similarly, we can convert this code to an integer and assign the hash buckets from 0 to 2. L It can be effectively labeled down to -1. (Formula 3) h:R d →{0,1,...,2 L -1},h(x)=2 L-1 h L (x)+···+2 0 h 1 (x)

[0038] Although LSH already reduces the computational complexity significantly compared to exact nearest neighbor search, binary code generation still requires L*d multiplications and L*(d-1) additions for each input. To further reduce the cost of this computation, we can use the method presented in "Very Sparse Random Projections. volume 2006, pp. 287-296, 08 2006. doi:10.1145 / 1150402.1150436" by Ping Li, Trevor Hastie, and Kenneth Church. Given a standard normally distributed vector v l Instead of using v, we use a very sparse vector v that contains only elements from the set {1,0,-1}. l Alternatively, we can use the following: Given a desired sparsity s ∈ (0,1), the hyperplane normal vector v l is randomly constructed so that only the expected number of s nonzero entries appear. Suppose we choose s=1 / 3, then a normal vector v l contains only d / 3 non-zero entries. If the remaining entries are chosen to be either -1 or 1, then the dot product computation reduces to L*(d / 3-1) additions and 0 multiplications. This allows us to exchange expensive multiplication operations for cheap additions.

[0039] Having established LSH via sparse random projection as a computationally cheap method for finding approximate nearest neighbors in high-dimensional spaces, we aim to leverage the method as a means to find redundancy in the channel dimension of latent feature maps in convolutional models.

[0040] Formally, a convolutional layer takes the (padded) input feature maps

number

number

[0041] Any patch p and input

number

number

[0042] We argue against this design choice and instead exploit redundant channels to save computations during the convolution operation. To extract these channels, we propose to use the LSH scheme described above. More specifically, to group similar channels, we use all

number

[0043] Next, we check which hash codes appear multiple times, since all elements that appear in the same bucket are determined to be nearly similar by the LSH scheme.

number

[0044] In particular, note that our RP LSH method is invariant to scaling of a given input vector, which means that k+b×k+b input channels of similar structure but different activation strengths will still fall into the same hash bucket, effectively finding further redundancy in the channel dimension.

[0045] Redundant Input Channels

number

[0046] To avoid repeated computations on nearly similar channels, we dynamically reduce the size of each input context window by merging channels in the same hash bucket. The merge operation is performed by taking the average of all channels in one bucket. As a result, the number of remaining input channels is reduced to

number

[0047] This results in a compression ratio of:

number

[0048] Similar to the input feature map depth reduction, the convolution filter F l However, instead of hashing the filter channels, we can simply merge channels that correspond to the folded input channels. This merging step is done on-the-fly at every filter position p, preserving the original weights for the next filter position. As a result, the filter depth is reduced while the one-to-one correspondence of channels remains intact.

number

[0049] The method can be defined by pseudocode as follows:

number

[0050] Although performing hashing and merging operations on the input channels incurs additional computational cost, the overall savings in reducing channel dimensions and computing convolution operations outweighs the additional overhead.

[0051] The method presented above serves as a direct plug-and-play alternative to any regular convolution module in CNNs. It features just one hyperparameter L, which controls the amount of hyperplanes used in the LSH scheme. Its value directly determines the trade-off between compression rate and preserved accuracy. When L is low, the random projections only roughly group the input channels. This results in high compression rates, but the resulting output feature maps may differ significantly from those of a full convolution operation, leading to an overall decrease in model performance. In contrast, when L is large, model performance is almost fully preserved at the expense of lower compression rates. The choice of L thus allows the generation of multiple model variants from one underlying base model, for example, focusing on either low FLOPs or high performance.

[0052] The hyperparameter L, which determines the number of hyperplanes used in the hashing scheme, acts directly as a controller of the accuracy / FLOPs tradeoff: it can be tuned individually at every point in the network to find the desired balance between model and hardware performance.

[0053] FIG. 1, in particular with the pseudocode above, illustrates the algorithm of the above technique, which includes a series of steps for processing input data of the convolutional layer of a CNN by hashing and merging channels, and by merging filter kernels according to the merging of channels before performing a convolution to generate, for example, an output tensor of the convolutional layer.

[0054] The algorithm starts with step S11 “Initialize hash function”, which initializes a hash function h. This function is used to hash the data representation. Preferred choices for various hash functions are described above.

[0055] In the following step S12, the algorithm enters a loop that iterates over each pixel of the input image or data point of the feature map of the convolutional layer. The loop operates on a patch p of the input image or feature.

[0056] In the following step S13, within the loop, an empty list called HashCode is created, suitable for storing the hash codes generated for each channel of the input data at the current pixel or data point.

[0057] In the following step S14, in the above loop, each input channel i (1 to c in There is another nested loop that iterates over

[0058] In the following step S15, in a nested loop for each input channel, the representation x is calculated by flattening the channel value at the current pixel p around the center. i is calculated. This is done using the "Center" and "Flatten" operations. The result is a flattened vector.

[0059] In the following step S16, the algorithm i Apply hash function f to to produce a hash code. This hash code is added to the HashCodes list.

[0060] In the next step S17, after processing all the input channels of the current pixel, the algorithm merges these channels into a single bucket, which means that the algorithm combines the values ​​of all the input channels at this pixel according to the calculated hash code. The result is

number

[0061] In the next step S18, the algorithm merges corresponding filter channels from the set of filters F based on identical hash codes. The result is F reduced Filter merging is performed similarly to channel merging, except that instead of averaging the channels in a given bucket, the filters of the corresponding buckets are summed.

[0062] In the next step S19, the reduced input

number

[0063] Finally, the algorithm returns an output tensor Y, which has the same dimensions as the input, but with additional channels (c out ) are few in number.

[0064] Figure 2 shows an overview of the context window and filter reduction. Each patch of the input feature map is processed to find redundant channels. Then, the detected redundancies are merged together and the sliding window and depth of all convolution filters are reduced.

Claims

1. A computer-implemented method for efficiently computing convolutions, particularly for a convolutional layer of a neural network, comprising: In particular, the tensor of input data to be processed by the convolutional layer ( [0010] ) and at least one filter ( [0025] ), wherein the filter receives a plurality of (c ｉｎ ) filter kernel (k×k); initializing a locality sensitive hash function; Repeating the following steps for multiple patches of the tensor (X): Slice the current patch into a series (c ｉｎ ) into a matrix (k+b×k+b); applying the locality sensitive hashing to each of the matrices to determine a hashed representation for each matrix; merging said matrices having essentially equivalent hashed representations into a new matrix; By arranging the merged matrices in succession, a contracted tensor (c ｉｎ，ｒｅｄｕｃｅｄ x k + b x k + b); merging the filter kernels in a similar order to the order in which the matrices were merged; convolving the reduced subtensor with the merged filter; Repeating the steps of: The method includes:

2. 2. The method of claim 1 , wherein the step of initializing the locality sensitive hashing is performed depending on an estimated number of hyperplanes of the locality sensitive hashing, said number being estimated depending on the computational resources of the computation unit on which the convolution, in particular the convolutional layer, is executed.

3. The method of claim 1 or 2, wherein the tensor is an image to be processed by the neural network or a feature map of a layer prior to the convolutional layer.

4. 4. The method of claim 1, wherein the neural network classifies its input based on the reduced subtensor convolved with the merged filter.

5. The convolution filter is ｉｎ 5. The method of claim 1, wherein the kernel has a size of k×k×k, where k≧1 and k×k represents the kernel size.

6. 6. The method of claim 1, wherein the merging of matrices is performed by averaging the matrices with essentially equal hashed representations, or by taking the median or element-wise maximum of the matrices with equal hashed representations.

7. The method of claim 1 , wherein the merging of filters is performed by summing the filter kernels corresponding to the merged matrix with the equivalent hashed representation.

8. 8. The method of claim 1, wherein the receptive field is synthetically expanded by at least one entry along each dimension (k+1 x k+1).

9. A computer program arranged to cause a computer to carry out the method according to any one of claims 1 to 8 with all its steps, when said computer program is executed by a processor.

10. 10. A machine-readable storage medium having stored thereon the computer program of claim 9.

11. A system configured to carry out the method of any one of claims 1 to 8.

Images