A method and system for realizing lossless acceleration of CNN inference based on predicted sparsity

Abstract

The application provides a method and system for realizing lossless acceleration of CNN inference based on predicted sparsity, comprising: obtaining an input vector and a convolution kernel weight vector, and determining a reference vector of the input vector; performing upper bound estimation of convolution according to the input vector, the reference vector and the convolution kernel weight vector, and obtaining an upper bound of a convolution operation output; and determining whether to perform the convolution operation according to the size of the upper bound. The application solves the defects of large operation amount and slow speed of the existing CNN convolutional neural network when performing video image calculation, so as to realize acceleration of the calculation process.

Description

Technical Field

[0001] This invention relates to the field of video and image processing technology, and in particular to a method and system for lossless acceleration of CNN inference based on prediction sparsity. Background Technology

[0002] Convolutional Neural Networks (CNNs) are widely used in computer vision tasks, including video surveillance, object detection, and image denoising. Due to their high accuracy with relatively small model sizes, CNNs are typically deployed on edge devices for computation.

[0003] Leveraging the sparsity of neural networks is a major technique for accelerating CNN inference. Due to the widespread use of the ReLU activation function in CNN models, the outputs of network layers can become extremely sparse. It has been found that CNN layer sparsity can reach over 90%. Sparsity-based acceleration techniques include model-level methods such as weight pruning and operator-level methods such as efficient sparse coding and sparse matrix approximation. Among these, operator-level lossless methods are preferable due to their performance guarantees and board applicability. A common Conv-ReLU structure (i.e., convolutional layers with ReLU activation) involves a dot product operation between the interacting convolutional filter and its receptive field, where optimizations can be applied. The key to achieving acceleration in the case of predictably sparse outputs is determining the negative output operation, which is resource-efficient and has a high success rate. However, existing CNN neural network computations require a large amount of computation, resulting in excessive resource consumption, low computational efficiency, and difficulty in meeting the ever-increasing computational demands. Summary of the Invention

[0004] This invention provides a method and system for lossless acceleration of CNN inference based on prediction sparsity, which solves the defects of existing CNN convolutional neural networks in terms of large computational load and slow speed when performing video image calculations, so as to accelerate the calculation process.

[0005] This invention provides a lossless acceleration method for CNN inference based on prediction sparsity, comprising:

[0006] Obtain the input vector and convolution kernel weight vector, and determine the reference vector of the input vector;

[0007] Based on the input vector, reference vector, and convolution kernel weight vector, perform convolution upper bound estimation to obtain the upper bound of the convolution operation output;

[0008] Whether to perform a convolution operation is determined based on the size of the upper bound.

[0009] According to the present invention, a lossless acceleration method for CNN inference based on prediction sparsity is provided, wherein obtaining the input vector and convolutional kernel weight vector, and determining the reference vector of the input vector specifically includes:

[0010] Obtain the position coordinates of the input vector;

[0011] The reference vector, which has already been calculated, is obtained based on the position coordinates of the input vector;

[0012] Based on the coordinate information of the input vector, for the inference task of the image, the spatially calculated adjacent vectors are obtained as reference vectors.

[0013] For inference tasks involving continuous video streams, the temporally computed neighboring vectors are used as reference vectors.

[0014] According to the present invention, a lossless acceleration method for CNN inference based on prediction sparsity is provided, which estimates the upper bound of the convolution operation output based on the input vector, reference vector, and convolution kernel weight vector, specifically including:

[0015] Obtain the upper bound of the dot product between the calculated reference vector and the convolution kernel weight vector;

[0016] Calculate the difference vector between the input vector and the reference vector;

[0017] Calculate the upper bound of the dot product between the difference vector and the convolution kernel weight vector;

[0018] The sum of the upper bound of the dot product of the reference vector and the weight vector and the upper bound of the dot product of the difference vector and the weight vector is assigned as the upper bound of the dot product of the current input vector and the weight vector.

[0019] According to the present invention, a lossless acceleration method for CNN inference based on prediction sparsity is provided, wherein calculating the upper bound of the dot product between the difference vector and the convolution kernel weight vector specifically includes:

[0020] Compare the signs of each element in the difference vector and the convolution kernel weight vector to generate the comparison result;

[0021] Based on the comparison results, a subset of the index sets of two vectors is obtained;

[0022] The subset of the subscript index set includes the subscript indices of elements with the same number and the subscript indices of elements with different numbers;

[0023] The upper bound of the dot product between the difference vector and the convolution kernel weight vector is generated based on the subscript indices of the elements with the same index and the subscript indices of the elements with different indexes.

[0024] According to the present invention, a lossless acceleration method for CNN inference based on prediction sparsity is provided, wherein comparing the signs of each element in the difference vector and the convolution kernel weight vector to generate a comparison result specifically includes:

[0025] Choose a set number of input vectors and convolution kernel weight vectors to compare their signs;

[0026] The set number is based on the first set number of elements with the largest absolute value in the convolution kernel weight vector.

[0027] According to the present invention, a lossless acceleration method for CNN inference based on prediction sparsity is provided, which determines whether to perform convolution operation based on the size of the upper bound, specifically including:

[0028] After obtaining the upper bound estimate, the magnitude of the upper bound estimate is judged.

[0029] If the estimated upper bound is not positive, the convolution operation is skipped.

[0030] If the estimated upper bound is positive, then convolution operation is performed.

[0031] This invention also provides a lossless acceleration system for CNN inference based on prediction sparsity, the system comprising:

[0032] The vector dot product module is used to obtain the input vector and the convolution kernel weight vector, and to determine the reference vector of the input vector.

[0033] The upper bound estimation module is used to estimate the upper bound of the convolution based on the input vector, reference vector and convolution kernel weights, and obtain the upper bound of the output of the convolution operation;

[0034] The dot product operation judgment module is used to determine whether to perform a convolution operation based on the size of the upper bound.

[0035] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lossless acceleration method for CNN inference based on prediction sparsity as described above.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lossless acceleration method for CNN inference based on prediction sparsity as described above.

[0037] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the lossless acceleration method for CNN inference based on prediction sparsity as described above.

[0038] This invention provides a lossless acceleration method and system for CNN inference based on prediction sparsity. By multiplying the input vector and a constant vector to estimate the vector dot product, an upper bound estimate is obtained. If the upper bound estimate is determined to be non-positive, the vector dot product operation is skipped, reducing the amount of computation and accelerating CNN operations without reducing the accuracy of the calculation. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is one of the flowcharts of a lossless acceleration method for CNN inference based on prediction sparsity provided by the present invention;

[0041] Figure 2 This is the second flowchart of a lossless acceleration method for CNN inference based on prediction sparsity provided by the present invention;

[0042] Figure 3 This is the third flowchart of a lossless acceleration method for CNN inference based on prediction sparsity provided by the present invention;

[0043] Figure 4 This is the fourth flowchart of a lossless acceleration method for CNN inference based on prediction sparsity provided by the present invention;

[0044] Figure 5 This is a schematic diagram of the module connection of a lossless acceleration system for CNN inference based on prediction sparsity provided by the present invention;

[0045] Figure 6 This is a schematic diagram comparing ordinary convolution and skipping long vector dot product operations in a lossless acceleration method for CNN inference based on prediction sparsity provided by the present invention.

[0046] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0047] Figure label:

[0048] 110: Vector dot product calculation module; 120: Upper bound estimation module; 130: Dot product operation judgment module; 710: Processor; 720: Communication interface; 730: Memory; 740: Communication bus. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] Unlike conventional neural networks, neurons in Convolutional Neural Networks (CNNs) are arranged in three dimensions: width, height, and depth. Width and height are easy to understand because convolution itself is a two-dimensional template. However, the depth in a CNN refers to the third dimension of the activation data volume, not the overall depth of the network, which refers to the number of layers. To illustrate width, height, and depth, suppose we use images from CIFAR-10 as input to a CNN. The dimensions of this input data volume are 32×32×3 (width, height, and depth). We will see that neurons in a layer will only connect to a small region in the previous layer, rather than using fully connected connections. For a CNN used to classify CIFAR-10 images, the final output layer has a dimension of 1×1×10 because the final part of the CNN structure compresses the full-size image into a vector containing the classification score, arranged along the depth direction.

[0051] Convolutional neural networks (CNNs) are mainly composed of these types of layers: input layer, convolutional layer, ReLU layer, pooling layer, and fully connected layer (the fully connected layer is the same as in a regular neural network). By stacking these layers, a complete CNN can be constructed. In practical applications, convolutional layers and ReLU layers are often collectively referred to as convolutional layers, so convolutional layers also undergo activation functions after convolution operations. Specifically, when convolutional layers and fully connected layers (CONV / FC) perform transformation operations on the input, they use not only activation functions but also many parameters, namely the weights w and biases b of the neurons; while ReLU layers and pooling layers perform a fixed function operation. The parameters in the convolutional and fully connected layers are trained using gradient descent, so that the classification scores calculated by the CNN match the labels of each image in the training set.

[0052] Convolutional layers are the core layers of convolutional neural networks, generating most of the network's computation. Pooling layers are periodically inserted between consecutive convolutional layers. Their function is to gradually reduce the spatial size of the data volume, thus reducing the number of parameters in the network, reducing computational resource consumption, and effectively controlling overfitting. Pooling layers use the MAX operation to operate independently on each depth slice of the input data volume, changing its spatial size. The most common form is that the pooling layer uses a 2x2 filter, downsampling each depth slice with a stride of 2, discarding 75% of the activation information. Each MAX operation takes the maximum value from four numbers (i.e., a 2x2 region within the depth slice), while keeping the depth unchanged. Fully connected layers, like in regular neural networks, have their activations first multiplied by matrix multiplication, then biased.

[0053] The following is combined with Figures 1-4 This invention describes a lossless acceleration method for CNN inference based on prediction sparsity, comprising:

[0054] S100. Obtain the input vector and convolution kernel weight vector, and determine the reference vector of the input vector;

[0055] S200. Based on the input vector, reference vector, and convolution kernel weight vector, perform convolution upper bound estimation to obtain the upper bound of the convolution operation output;

[0056] S300. Determine whether to perform convolution operation based on the size of the upper bound.

[0057] In image or video recognition processing, CNN neural networks are used to process images, accurately identifying image content and providing a data foundation for subsequent image processing. However, image processing involves large amounts of computation, consumes many resources, and is costly. In this invention, the upper bound of the vector dot product is used to determine whether to skip the dot product operation of long vectors, thereby reducing the amount of dot product computation and improving the processing speed.

[0058] References in this invention Figure 6 The dashed line at the bottom represents the original Conv-ReLU operator, which calculates the dot product of two long vectors and performs the ReLU operation.

[0059] Whether to skip the dot product of long vectors is determined by estimating an upper bound for the dot product. This estimation method relies on the pre-computed dot product of several reference vectors used in the convolution operation. For each reference, the input is x. ref The corresponding upper bound of the output is Therefore, it is possible to use x ref and x i，j To calculate y i，j，k The upper boundary.

[0060] The reference used refers to some input vector x i，j A vector that is close to the sum of a constant vector w. k The dot product is negative. Different inference tasks may use different references.

[0061] Specifically, for reasoning tasks involving a single image, x i，j The reference is its spatial neighbor {x} i，j-1 x i-1，j For inference tasks involving continuous video streams, x i，j References can also include temporal neighbors. x i，j Corresponding output y i，j，k Once the upper bound is calculated, it can also be used as a reference for other neighbors.

[0062] We calculate the vector dot product x i，j ·w k Previously, its upper bound was calculated.

[0063]

[0064] If the estimated upper bound Then we have x i，j ·w k ≤0, therefore y can be i，j，k = 0. Otherwise, a vector dot product operation x is required. i，j ·w k This process reduces computation by skipping the dot product operation of vectors.

[0065] Obtain the input vector and convolution kernel weight vector, and determine the reference vector for the input vector, specifically including:

[0066] S101. Obtain the position coordinates of the input vector;

[0067] S102. Obtain the reference vector that has already been calculated based on the position coordinates of the input vector;

[0068] S103. Based on the coordinate information of the input vector, for the inference task of the image, the adjacent vectors that have been calculated in space are obtained as reference vectors.

[0069] S104. For inference tasks involving continuous video streams, the temporally computed adjacent vectors are obtained as reference vectors.

[0070] Based on the input vector, reference vector, and convolution kernel weight vector, an upper bound estimation of the convolution operation is performed to obtain the upper bound of the convolution operation output, specifically including:

[0071] Obtain the upper bound of the dot product between the calculated reference vector and the convolution kernel weight vector;

[0072] Calculate the difference vector between the input vector and the reference vector;

[0073] Calculate the upper bound of the dot product between the difference vector and the convolution kernel weight vector;

[0074] The sum of the upper bound of the dot product of the reference vector and the weight vector and the upper bound of the dot product of the difference vector and the weight vector is assigned as the upper bound of the dot product of the current input vector and the weight vector.

[0075] Calculate the upper bound of the dot product between the difference vector and the convolution kernel weight vector, specifically including:

[0076] S201. Compare the signs of each element in the difference vector and the convolution kernel weight vector to generate the comparison result;

[0077] S202. Based on the comparison result, obtain a subset of the index sets of two vectors;

[0078] S203, The subset of the subscript index set includes the subscript indices of elements with the same number and the subscript indices of elements with different numbers;

[0079] S204. Generate the upper bound of the dot product between the difference vector and the convolution kernel weight vector based on the subscript indices of the elements with the same index and the subscript indices of the elements with different indexes.

[0080] The comparison of the signs of each element in the difference vector and the convolution kernel weight vector to generate the comparison result specifically includes:

[0081] Choose a set number of input vectors and convolution kernel weight vectors to compare their signs;

[0082] The set number is based on the first set number of elements with the largest absolute value in the convolution kernel weight vector.

[0083] The core of the upper bound estimation in this invention is the function φ. φ is used to estimate the vector x. i，j -x ref The upper bound of the dot product between x and y. For simplicity, let x... i，j -x ref Abbreviated as δ. The upper bound calculated by the proposed method can be written as:

[0084]

[0085] The I here diff-sign-sub This refers to the index subscript of the subvector. It is its complement. δ[I diff-sign-sub ], w k [I diff-sign-sub ]and It is δ and wk The subvector retrieved by index. Next, we will explain how to estimate δ·w. k The upper boundary.

[0086] First, by comparing δ and w k The sign of each element can be used to obtain two vector index sets I. all A subset of. I same-sign It is δ and w k The index of elements with the same sign in the middle, similarly, I diff-sign It is the index of the element with the opposite index.

[0087] I same-sign ={i|δ[i]×w k [i]>0}

[0088] I diff-sign ={i|δ[i]×w k [i]≤0}

[0089] δ and w k The dot product operation can be divided into two parts: one part results in a positive dot product (the dot product of subvectors with the same sign), and the other part results in a negative dot product (the dot product of subvectors with opposite signs). Therefore, we have

[0090] δ·w k =δ[I same-sign ]·w k [I same-sign ]+δ[I diff-sign ]·w k [I diff-sign ]≤||δ[I same-sign ]||×||w k [I same-sign ]||+δ[I diff-sign ]·w k [I diff-sign ]

[0091] Compare δ and w k Comparing the signs of all elements in the table is extremely time-consuming. In practice, only the signs of E elements (default setting E=6) are compared, which is w k Find the top E elements with the largest absolute values ​​in the I-value group. diff-sign-sub Let E represent the set of indices of the elements whose signs are opposite to those of the corresponding element δ, and let δ = ... make Let be the set of indexes of the elements of another subset. Then we have

[0092]

[0093] because It is a relatively large set, so we need to calculate the magnitude of the subvectors indexed by it (e.g., and It's still quite cumbersome. Different calculations can be performed offline before inference. Corresponding (Because E is very small, The combinations are relatively few. In addition, it can replace the equation in... It is ||δ||, that is

[0094]

[0095] The upper bound estimate used by Wakatsuki et al. is written as... Obviously, the φ(δ, w) proposed in this invention... k )Compare (δ,w k More compact because And δ[I diff-sign-sub ]·w k [I diff-sign-sub The result is negative.

[0096] The φ(δ, w) proposed in this invention k By comparing the signs of a small subset of elements (I) diff-sign-sub ), to obtain the dot product of the two short vectors (δ[I diff-sign-sub ]·w k [I diff-sign-sub The weights are multiplied by the magnitude of a vector (||δ||), and the magnitude of the weights is pre-calculated offline. final x i，j ·w k The upper bound is:

[0097]

[0098] Determining whether to perform a vector dot product operation based on the magnitude of the upper bound estimate specifically includes:

[0099] S301. After obtaining the upper bound estimate, determine the magnitude of the upper bound estimate.

[0100] S302. If the estimated upper bound value is not a positive number, skip the convolution operation;

[0101] S303. If the upper bound estimate is positive, then perform convolution operation.

[0102] After determining the upper bound estimate, it is possible to quickly determine whether to set the vector dot product estimate to 0 or perform a convolution operation, thereby reducing the amount of computation.

[0103] In this invention, the process of performing operations with the convolution kernel on the input at a specific position (i, j) is analyzed. The original convolution operator requires K vector dot products—input x i，j and all convolution kernels {w1, w2, ..., w K The dot product between}. Let L = C × R × S represent x. i，j and w k The length of the vector. Therefore, calculate the input x. i，j The total number of FLOPs required is KCRS.

[0104] if Is input x i，j The proportion of all convolutional kernels whose estimated upper bound is negative (that is...) When the output {y} after calculating the activation function is calculated. i，j，1 y i，j，2 , ..., y i，j，K The proposed method will then be skipped. This involves performing the vector dot product operation 10 times. The reduction in FLOPs is...

[0105] The introduced computational cost. The introduced computational cost is to compute each reference input x. ref vector magnitude ||x i，j -x ref ||, FLOPs is the FCRS, where F is the number of references (e.g., F=2 for single-image inference; F=3 for video stream inference). F upper bounds are obtained using F references, and let… It equals the minimum value among them. For each reference x ref In other words, it takes E sign comparisons to obtain I. diff-sign-su And E FLOPs to calculate Therefore, the total computational cost introduced is FCRS + 2FKE.

[0106] Therefore, when the amount of computation introduced is less than the amount of computation reduced, the algorithm can achieve an acceleration effect, that is...

[0107]

[0108] In conventional reasoning, since the chosen reference is quite good, it generally satisfies... Since FKE is generally very small and can be ignored, the proposed algorithm can easily achieve a speedup effect.

[0109] The overall speedup depends on the chosen reference and the nature of the input data. The reference used in this invention leverages temporal and spatial locality, thus providing excellent speedup for images with large areas of similar color or videos with minimal inter-frame variations. In the most ideal case, the found reference x...ref and x i，j Very close, i.e., ||x i，j -x ref If ≈0, then the proportion of vector dot products that can be skipped will be close to the output sparsity of the Conv-ReLU operator.

[0110] This invention provides a lossless acceleration method for CNN inference based on prediction sparsity. By multiplying the input vector and a constant vector to estimate the vector dot product, an upper bound estimate is obtained. If the upper bound estimate is not positive, the vector dot product operation is skipped, reducing the amount of computation and accelerating CNN operations without reducing the accuracy of the calculation.

[0111] refer to Figure 5 The present invention also discloses a lossless acceleration system for CNN inference based on prediction sparsity, the system comprising:

[0112] The vector dot product acquisition module 110 is used to obtain the input vector and the convolution kernel weight vector, and to determine the reference vector of the input vector;

[0113] The upper bound estimation module 120 is used to estimate the upper bound of the convolution based on the input vector, the reference vector and the convolution kernel weight vector, and obtain the upper bound of the output of the convolution operation.

[0114] The dot product operation judgment module 130 is used to determine whether to perform a convolution operation based on the size of the upper bound.

[0115] The vector dot product calculation module 110 obtains the position coordinates of the input vector;

[0116] The reference vector, which has already been calculated, is obtained based on the position coordinates of the input vector;

[0117] Based on the coordinate information of the input vector, for the inference task of the image, the spatially calculated adjacent vectors are obtained as reference vectors.

[0118] For inference tasks involving continuous video streams, the temporally computed neighboring vectors are used as reference vectors.

[0119] Upper bound estimation module 120 obtains the upper bound of the dot product between the calculated reference vector and the convolution kernel weight vector;

[0120] Calculate the difference vector between the input vector and the reference vector;

[0121] Calculate the upper bound of the dot product between the difference vector and the convolution kernel weight vector;

[0122] The sum of the upper bound of the dot product of the reference vector and the weight vector and the upper bound of the dot product of the difference vector and the weight vector is assigned as the upper bound of the dot product of the current input vector and the weight vector.

[0123] Calculate the upper bound of the dot product between the difference vector and the convolution kernel weight vector, specifically including:

[0124] Compare the signs of each element in the difference vector and the convolution kernel weight vector to generate the comparison result;

[0125] Based on the comparison results, a subset of the index sets of two vectors is obtained;

[0126] The subset of the subscript index set includes the subscript indices of elements with the same number and the subscript indices of elements with different numbers;

[0127] The upper bound of the dot product between the difference vector and the convolution kernel weight vector is generated based on the subscript indices of the elements with the same index and the subscript indices of the elements with different indexes.

[0128] The comparison results are generated by comparing the signs of each element in the difference vector and the convolution kernel weight vector, specifically including:

[0129] Choose a set number of input vectors and convolution kernel weight vectors to compare their signs;

[0130] The set number is based on the first set number of elements with the largest absolute value in the convolution kernel weight vector.

[0131] The core of the upper bound estimation in this invention is the function φ. φ is used to estimate the vector x. i，j -x ref and w k The upper bound of the dot product. For simplicity, let x... i，j -x ref Abbreviated as δ. The upper bound calculated by the proposed method can be written as:

[0132]

[0133] The I here diff-sign-sub This refers to the index subscript of the subvector. It is its complement. δ[I diff-sign-sub ], w k [I diff-sign-sub ]and It is δ and w k The subvector retrieved by index. Next, we will explain how to estimate δ·w. k The upper boundary.

[0134] First, by comparing δ and w k The sign of each element can be used to obtain two vector index sets I. all A subset of. Isame-sign It is δ and w k The index of elements with the same sign in the middle, similarly, I diff-sign It is the index of the element with the opposite index.

[0135] I same-sign ={i|δ[i]×w k [i]>0}

[0136] I diff-sign ={i|δ[i]×w k [i]≤0}

[0137] δ and w k The dot product operation can be divided into two parts: one part results in a positive dot product (the dot product of subvectors with the same sign), and the other part results in a negative dot product (the dot product of subvectors with opposite signs). Therefore, we have

[0138] δ·w k =δ[I same-sign ]·w k [I same-sign ]+δ[I diff-sign ]·w k [I diff-sign ]≤||δ[I same-sign ]||×||w k [I same-sign ]||+δ[I diff-sign ]·w k [I diff-sign ]

[0139] Compare δ and w k Comparing the signs of all elements in the table is extremely time-consuming. In practice, only the signs of E elements (default setting E=6) are compared, which is w k Find the top E elements with the largest absolute values ​​in the I-value group. diff-sign-sub Let E represent the set of indices of the elements whose signs are opposite to those of the corresponding element δ, and let δ = ... make Let be the set of indexes of the elements of another subset. Then we have

[0140]

[0141] because It is a relatively large set, so we need to calculate the magnitude of the subvectors indexed by it (e.g., and It's still quite cumbersome. Different calculations can be performed offline before inference. Corresponding (Because E is very small, The combinations are relatively few. In addition, the following can be substituted in equation (9). It is ||δ||, that is

[0142]

[0143] The upper bound estimate used by Wakatsuki et al. is written as... Obviously, the φ(δ, w) proposed in this invention... k )Compare More compact because And δ[I diff-sign-sub ]·w k [I diff-sign-sub The result is negative.

[0144] The φ(δ, w) proposed in this invention k By comparing the signs of a small subset of elements (I) diff-sign-sub ), to obtain the dot product of the two short vectors (δ[I diff-sign-sub ]·w k [I diff-sign-sub The weights are multiplied by the magnitude of a vector (||δ||), and the magnitude of the weights is pre-calculated offline. Ultimately, x i，j ·w k The upper bound is:

[0145]

[0146] The dot product operation judgment module 130, after obtaining the upper bound estimate, judges the magnitude of the upper bound estimate;

[0147] If the estimated upper bound is not positive, the convolution operation is skipped.

[0148] If the estimated upper bound is positive, then convolution operation is performed.

[0149] After determining the upper bound estimate, it is possible to quickly determine whether to set the vector dot product estimate to 0 or to perform the vector dot product operation, thereby reducing the amount of computation.

[0150] This invention provides a lossless acceleration system for CNN inference based on prediction sparsity. By multiplying the input vector and a constant vector to estimate the vector dot product, an upper bound estimate is obtained. If the upper bound estimate is not positive, the vector dot product operation is skipped, reducing the amount of computation and accelerating CNN operations without reducing the accuracy of the calculation.

[0151] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a lossless acceleration method for CNN inference based on prediction sparsity. This method includes: obtaining the input vector and the convolutional kernel weight vector, and determining the reference vector of the input vector;

[0152] Based on the input vector, reference vector, and convolution kernel weights, the upper bound of the convolution operation is estimated to obtain the upper bound of the output of the convolution operation.

[0153] Whether to perform a convolution operation is determined based on the size of the upper bound.

[0154] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute a lossless acceleration method for CNN inference based on prediction sparsity provided by the above methods, the method including: obtaining an input vector and a convolutional kernel weight vector, and determining a reference vector for the input vector;

[0156] Based on the input vector, reference vector, and convolution kernel weights, the upper bound of the convolution operation is estimated to obtain the upper bound of the output of the convolution operation.

[0157] Whether to perform a convolution operation is determined based on the size of the upper bound.

[0158] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a lossless acceleration method for CNN inference based on prediction sparsity provided by the above methods, the method comprising: obtaining an input vector and a convolutional kernel weight vector, and determining a reference vector for the input vector;

[0159] Based on the input vector, reference vector, and convolution kernel weights, the upper bound of the convolution operation is estimated to obtain the upper bound of the output of the convolution operation.

[0160] Whether to perform a convolution operation is determined based on the size of the upper bound.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for lossless acceleration of CNN inference based on predicted sparsity, comprising: The method comprises the following steps: obtaining an input vector and a convolution kernel weight vector, and determining a reference vector of the input vector; performing convolution upper bound estimation according to the input vector, the reference vector and the convolution kernel weight vector, and obtaining an upper bound of a convolution operation output; determining whether to perform the convolution operation according to the size of the upper bound; wherein the step of obtaining the input vector and the convolution kernel weight vector, and determining the reference vector of the input vector specifically comprises the following steps: obtaining the position coordinates of the input vector; obtaining a reference vector that has formed a calculation result before according to the position coordinates of the input vector; according to the coordinate information of the input vector, if the inference task is for a picture, then obtaining an adjacent vector that has been calculated in space as the reference vector; if the inference task is for a continuous video stream, then obtaining an adjacent vector that has been calculated in time as the reference vector; the step of performing convolution upper bound estimation according to the input vector, the reference vector and the convolution kernel weight vector, and obtaining the upper bound of the convolution operation output specifically comprises the following steps: obtaining the dot product upper bound of the reference vector and the convolution kernel weight vector; calculating a difference vector of the input vector and the reference vector; calculating the dot product upper bound of the difference vector and the convolution kernel weight vector; assigning the sum of the dot product upper bound of the reference vector and the weight vector and the dot product upper bound of the difference vector and the weight vector as the dot product upper bound of the current input vector and the weight vector; the step of determining whether to perform the convolution operation according to the size of the upper bound specifically comprises the following steps: after obtaining the upper bound estimation value, judging the size of the upper bound estimation value; if the upper bound estimation value is a non-positive number, then skipping the convolution operation; if the upper bound estimation value is a positive number, then performing the convolution operation.

2. The method of claim 1, wherein, the step of calculating the dot product upper bound of the difference vector and the convolution kernel weight vector specifically comprises the following steps: comparing the signs of each element in the difference vector and the convolution kernel weight vector to generate a comparison result; according to the comparison result, obtaining a subset of the index set of the two vectors; the subset of the index set comprises the index of the same sign element and the index of the different sign element; generating the upper bound of the dot product of the difference vector and the convolution kernel weight vector according to the index of the same sign element and the index of the different sign element.

3. The method of claim 2, wherein, the step of comparing the signs of each element in the difference vector and the convolution kernel weight vector to generate a comparison result specifically comprises the following steps: selecting a certain number of signs of the input vector and the convolution kernel weight vector for comparison; the certain number is determined according to the first certain number of elements with the largest absolute value in the convolution kernel weight vector.

4. A lossless acceleration system for CNN inference based on prediction sparsity, characterized in that, The system comprises: a vector dot product calculation module, configured to obtain an input vector and a convolution kernel weight vector, and determine a reference vector of the input vector; the vector dot product calculation module is specifically configured to: obtain the position coordinates of the input vector; obtain a reference vector that has formed a calculation result before according to the position coordinates of the input vector; according to the coordinate information of the input vector, if the inference task is for a picture, then obtain an adjacent vector that has been calculated in space as the reference vector; if the inference task is for a continuous video stream, then obtain an adjacent vector that has been calculated in time as the reference vector; An upper bound estimation module is configured to perform convolution upper bound estimation based on the input vector, the reference vector and the convolution kernel weight, and obtain an upper bound of the convolution operation output; the upper bound estimation module is specifically configured to: obtain a reference vector completed calculation and a convolution kernel weight vector dot product upper bound; calculate a difference vector between the input vector and the reference vector; calculate a dot product upper bound between the difference vector and the convolution kernel weight vector; and assign a sum of the dot product upper bound between the reference vector and the weight vector and the dot product upper bound between the difference vector and the weight vector to a dot product upper bound between the current input vector and the weight vector. A dot product operation judgment module is configured to determine whether to perform the convolution operation based on the size of the upper bound; the dot product operation judgment module is specifically configured to: after obtaining the upper bound estimation value, judge the size of the upper bound estimation value; if the upper bound estimation value is a non-positive number, skip the convolution operation; and if the upper bound estimation value is a positive number, perform the convolution operation.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the CNN inference lossless acceleration method based on the predicted sparsity according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the CNN inference lossless acceleration method based on the predicted sparsity according to any one of claims 1 to 3.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the CNN inference lossless acceleration method based on the predicted sparsity according to any one of claims 1 to 3.

Images