A method and system for realizing image classification based on full fusion and pruning technology of binary neural network

Abstract

The present invention relates to a method and system for realizing image classification based on full fusion and pruning technology of a binary neural network. Based on the data commonality between the convolution layer, pooling layer and the first fully connected layer of the binary neural network, the convolution layer, pooling layer and the first fully connected layer of the binary neural network are fused. At the same time, considering that there are a large number of redundant operations in the fusion process of the binary neural network, and these redundant operations do not affect the final classification accuracy, the number of operation operations is reduced by a pruning method during the fusion process. The present invention utilizes full fusion technology and pruning methods, can effectively reduce the operation time and the number of operations without reducing the image classification accuracy, and takes into account the advantages of high hardware acceleration efficiency of the binary neural network.

Description

Technical Field

[0001] The present invention relates to a method and system for realizing image classification based on full fusion and pruning technology of binarized neural network, belonging to the field of integrated circuit hardware design of artificial intelligence (AI). Background Art

[0002] Convolutional Neural Networks (CNN) is one of the classic AI structures and an important part of the field of deep learning. CNN generally uses floating-point calculations, which requires a large amount of storage space and computing power, so its hardware implementation requires a large amount of resource overhead. In recent years, deep convolutional neural networks have been widely used in computer vision, speech recognition, and natural language processing. However, in low-power and low-cost mobile application scenarios, the application of convolutional neural networks is subject to many restrictions: on the one hand, high-performance convolutional neural network models occupy a large amount of storage space and have high requirements on the hardware resources of mobile terminals. On the other hand, convolutional neural networks have high computational complexity. When running deeper convolutional neural network models, the power consumption of dot product operations is relatively high. In order to make CNN more suitable for edge computing, binarized neural networks (BNN) were proposed.

[0003] BNN binarizes the parameters in the network into 1 and -1 (or 1 and 0), which significantly reduces the demand for hardware resources of the neural network. BNN reduces the memory demand to one tenth of that of CNN. At the same time, BNN uses bitwise AND and XNOR operations to replace the multiplication and addition operations in CNN, which significantly reduces computing power consumption.

[0004] With the development of artificial intelligence, people have applied binary neural networks to mobile hardware acceleration optimization, which has improved the speed of model operation. However, the current hardware acceleration of binary neural networks is still based on software design ideas, which are carried out in a sequential, complete, and hierarchical manner. This hardware implementation method is inefficient, does not maximize the hardware acceleration advantages of binary neural networks, and cannot meet the low-power application requirements of actual embedded scenarios. In order to address the design problems of hardware implementation of such binary neural networks, a technology is needed to fuse and prune the layers of binary neural networks, so as to achieve efficient intelligent detection with a faster, lower power consumption, and less hardware resource-intensive hardware implementation method. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method for image classification based on full fusion and pruning technology of binary neural network.

[0006] The present invention fuses the convolution layer, pooling layer and the first fully connected layer of the binary neural network, and performs pruning based on the particularity of binary values ​​to effectively reduce the computing time and the number of operations, taking into account the hardware acceleration efficiency of the binary neural network. It is a binary neural network hardware acceleration method that can achieve both a shorter computing time and hardware acceleration efficiency.

[0007] The present invention also provides a picture classification system based on full fusion and pruning technology of binarized neural network.

[0008] Terminology explanation:

[0009] 1. CNN is the abbreviation of Convolutional Neural Networks, which is called convolutional neural network.

[0010] 2. BNN is the abbreviation of Binarized Neural Networks, which is called binarized neural network.

[0011] 3. Pruning refers to removing some parameters from the existing network to improve network efficiency while ensuring network accuracy.

[0012] The technical solution of the present invention is:

[0013] A method for image classification based on full fusion and pruning technology of binary neural network includes the following steps:

[0014] Step 1: Build a binary neural network architecture to implement the image binary classification task;

[0015] Step 2: Obtain the image data set and perform grayscale and 1,0 binary preprocessing on the original image data;

[0016] Step 3: Construct a sliding window with a size of R k *R k , a sliding window with a step size of m and a number of slides of c, where m is the product of the convolution step size and the pooling step size;

[0017] Step 4: Reconstruct the convolution kernel, that is, reconstruct the size and content of the 1,0 binary convolution kernel obtained by the binary neural network training. The reconstructed convolution kernel is named the reconstructed convolution kernel, and its size is R k *R k ;

[0018] Step 5: Prune the reconstructed convolution kernel, that is, remove the 0 part in the reconstructed convolution kernel;

[0019] Step 6: Select the weight of the first fully connected layer with a size of c*q, that is, select the weight of the first fully connected layer obtained by training the binary neural network, whose size is c*q, where c is the number of sliding window sliding times, and q is the output size of the first fully connected layer;

[0020] Step 7: Load the binary image preprocessed in step 2, the reconstructed convolution kernel pruned in step 5, and the weights of the first fully connected layer selected in step 6 into the hardware memory;

[0021] Step 8: Slide the sliding window from left to right and from top to bottom on the binary image preprocessed in step 2 with a step length of m;

[0022] Step 9: Perform convolution-pooling fusion pruning operation on the binary image data intercepted by the sliding window, that is, perform bit-by-bit AND operation on the binary image data intercepted by the sliding window and the pruned reconstructed convolution kernel, and remove the bit-by-bit AND operation that does not affect the classification accuracy during the bit-by-bit AND operation;

[0023] Step 10: Perform a fusion pruning operation on the convolution-pooling fusion pruning result of step 9 and part of the weights of the first fully connected layer selected in step 6, that is, the convolution-pooling fusion pruning result of the current data intercepted by the sliding window determines the output result of the first fully connected layer, and removes the bitwise XOR operation between the convolution-pooling fusion pruning result of the data intercepted by the sliding window and part of the weights of the first fully connected layer;

[0024] Step 11: Determine whether the sliding window has completed the traversal of the binary image data. If completed, proceed to step 12 to end the full fusion pruning operation of steps 9-10; otherwise, repeat steps 8-10;

[0025] Step 12: Accumulate and compare the fusion pruning operation results output from step 10 during the traversal process;

[0026] Step 13: Complete the operation of the second fully connected layer, that is, perform a bitwise XOR operation on the accumulation and comparison results obtained in step 12 and the weights of the second fully connected layer obtained by the binary neural network training;

[0027] Step 14: Take the category corresponding to the maximum value of the result in step 13 as the final classification label of the input image.

[0028] Furthermore, the network architecture of the binary neural network includes an input layer, a convolution layer, a pooling layer, two fully connected layers, and an output layer; the input layer preprocesses the original image into a binary image; the convolution layer extracts features of the input image data through a convolution kernel, and activates the convolution layer output with a Sign activation function; the pooling layer reduces network parameters by downsampling; the fully connected layer integrates the image features extracted by the convolution pooling layer, and activates the fully connected layer output with a Sign activation function; the output layer can be regarded as a classifier for predicting the category of the input image; the two fully connected layers include a first fully connected layer and a second fully connected layer.

[0029] Furthermore, in step 1, in the forward propagation process of the binarized neural network, the convolution kernel is binarized based on the binary function Binarized, and the outputs of the convolution layer and the fully connected layer are binarized based on the sign function Sign; wherein the Binarized function is shown in (I), and the Sign function is shown in (II):

[0030]

[0031]

[0032] In the process of back propagation, the Htanh function is used to calculate the gradient, where the Htanh function is shown in (III):

[0033] Htanh(x)=max(-1,min(1,x)) (Ⅲ)

[0034] Furthermore, in step 3, the calculation formula for the number of slides c is:

[0035]

[0036] In formula (IV), I is the size of the input binary image, R k is the unilateral size of the reconstructed convolution kernel, S is the convolution step size, and P is the unilateral size of the pooling layer.

[0037] Furthermore, in step 4, resizing means: changing the size of the convolution kernel obtained by training the binary neural network to R k *R k , where R k The calculation formula is shown in formula (V):

[0038] R k =K+(S*(P-1)) (V)

[0039] In formula (V), K represents the unilateral size of the convolution kernel of the convolution layer; S represents the convolution step size, (P-1) represents the unilateral size of the pooling layer minus 1; S*(P-1) represents the size of the remaining convolution kernels except the first convolution kernel.

[0040] Furthermore, in step 4, content reconstruction means: P*P identical convolution kernels are reconstructed according to the coordinates C with the convolution step size S as the minimum unit. F place;

[0041] The coordinate C F As shown in formula (VI):

[0042]

[0043] In formula (VI), P represents the single-side size of the pooling layer, and i represents the i-th convolution kernel;

[0044] Convolution is performed on P*P identical convolution kernels with a convolution step size of S as the minimum unit according to the coordinate C F Placement specifically means: the first convolution kernel is placed at (0,0), the second convolution kernel is placed at (S,0), the Pth convolution kernel is placed at ((P-1)*S,0), the P+1th convolution kernel is placed at (0,S), the P+2th convolution kernel is placed at (S,S), and the P*Pth convolution kernel is placed at ((P-1)*S,(P-1)*S)); wherein, the overlapping parts between the convolution kernels used for reconstruction are operated according to the OR logic, which means: if the overlapping part has a value of 1, then the value of the part after reconstruction is 1; if the overlapping part does not have a value of 1, then the value of the part after reconstruction is 0.

[0045] Furthermore, in step 5, the pruning operation on the reconstructed convolution kernel means: splitting the reconstructed convolution kernel containing 1, 0 into a reconstructed convolution kernel containing only 1, and pruning the 0 part; pruning the 0 part means setting the part corresponding to 0 in the reconstructed convolution kernel to empty, and not using it as part of the reconstructed convolution kernel.

[0046] Furthermore, in step 9, the convolution-pooling fusion and pruning operation is performed, which means that the input binary image data intercepted by the sliding window is subjected to a bit-by-bit AND operation with the reconstructed convolution kernel after pruning in step 5 to obtain the result of the convolution-pooling fusion; in the process of performing the bit-by-bit AND operation, when the result of a certain bit is calculated to be 1, the remaining bits are no longer subjected to the AND operation, and the output result is 1.

[0047] Furthermore, in the step 10, the convolution-pooling fusion pruning result of step 9 is fused and pruned with the first-layer fully connected layer weights selected in part of step 6, which means that after the convolution-pooling fusion and pruning operation of step 7 is completed on the binary image data intercepted by the sliding window for the jth sliding, the first-layer fully connected layer operation of the binary image data currently intercepted by the sliding window is performed; the first-layer fully connected layer weights selected in part of step 6 refer to the weights of the [(j-1)*q,j*q-1]th bit, where j represents the jth sliding of the sliding window; and q is the output size of the first fully connected layer.

[0048] Further, in step 10, the convolution-pooling fusion pruning result of the sliding window's j-th sliding intercepted data determines the result of the output of the first layer of the fully connected layer of the sliding intercepted data, which means that if the convolution-pooling fusion and pruning result of the sliding window's j-th sliding intercepted data is 1, then the output of the first layer of the fully connected layer of the intercepted data is the weight data selected this time by the first layer of the fully connected layer; if the convolution-pooling fusion and pruning result of the sliding window's j-th sliding intercepted data is 0, then the output of the first layer of the fully connected layer of the intercepted data is the bitwise inversion of the weight data selected this time by the first layer of the fully connected layer; the weights of the first layer of the fully connected layer and the bitwise inversion of the weights of the first layer of the fully connected layer are pre-stored in the memory, and the bitwise XOR operation of the convolution-pooling fusion pruning result and the weights of the first layer of the fully connected layer is removed.

[0049] Furthermore, in step 12, accumulating and comparing the fusion pruning operation results outputted in step 10 during the traversal process means: adding the convolution-pooling fusion pruning results outputted in step 10 during the traversal process bit by bit, and determining bit by bit whether the result after addition is greater than If it is greater, the output result is 1; otherwise, the output result is 0; where c is the number of sliding times of the sliding window.

[0050] A picture classification system based on full fusion and pruning technology of binary neural network, including:

[0051] The image data set acquisition and preprocessing module is configured to: acquire the image data set and preprocess the image data using a 1,0 binarization method;

[0052] The reconstructed convolution kernel and pruning module is configured to reconstruct the size and content of the convolution kernel obtained by binarization neural network training, and name the reconstructed convolution kernel as the reconstructed convolution kernel with a size of R. k *R k ; And perform pruning operation on the reconstructed convolution kernel, that is, remove the 0 part in the reconstructed convolution kernel;

[0053] The first fully connected layer weight selection module is configured to: select the first fully connected layer weight of size c*q, that is, select the first fully connected layer weight obtained by binarization neural network training;

[0054] The convolution-pooling fusion pruning operation module is configured to: perform a convolution-pooling fusion pruning operation on the binary image data intercepted by the sliding window, that is, perform a bitwise AND operation on the binary image data intercepted by the sliding window and the pruned reconstructed convolution kernel, and remove the bitwise AND operation that does not affect the classification accuracy during the bitwise AND operation;

[0055] The fully connected fusion pruning operation module is configured to: perform a fusion pruning operation on the convolution-pooling fusion pruning result and the selected first-layer fully connected layer weights, that is, the convolution-pooling fusion pruning result of the current data intercepted by the sliding window determines the output result of the first-layer fully connected layer, and removes the bitwise XOR operation between the convolution-pooling fusion pruning result of the data intercepted by the sliding window and part of the first-layer fully connected layer weights;

[0056] The sliding window sliding and traversal judgment module is configured to: slide the sliding window on the preprocessed binary image with a step length of m according to the rule of from left to right and from top to bottom; and judge whether the sliding window has completed the traversal of the binary image data;

[0057] The accumulation and comparison operation module is configured to: accumulate and compare all fusion and pruning operation results after the sliding window completes the traversal of the binary image data;

[0058] The subsequent fully connected layer operation module is configured to: complete the second layer fully connected layer operation, that is, perform a bitwise XOR operation on the accumulated and compared results and the second layer fully connected layer weights obtained by the binary neural network training;

[0059] The image classification result output module is configured to use the category corresponding to the maximum value of the result of the subsequent fully connected layer operation module as the final classification label of the input image.

[0060] A computer-readable storage medium, wherein a memory stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for implementing a method for image classification based on full fusion and pruning technology of a binarized neural network.

[0061] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded and executed by the processor to implement a method for image classification based on full fusion and pruning technology of a binarized neural network.

[0062] The beneficial effects of the present invention are:

[0063] 1. The present invention proposes a method for image classification based on full fusion and pruning technology of a binary neural network. The convolution layer, pooling layer and the first fully connected layer of the binary neural network are fused, and redundant operations that have no effect on the final accuracy during the fusion process are removed by a pruning method, thereby realizing hardware acceleration of the binary neural network in a more efficient, less computational delay and less hardware resource required manner.

[0064] 2. The present invention proposes a method for image classification based on full fusion and pruning technology of binary neural network, which combines the fusion operation between layers of binary neural network with the pruning method, reduces the hardware resources required for hardware acceleration, and ensures the accuracy of image classification.

[0065] 3. The present invention proposes a method for image classification based on full fusion and pruning technology of binary neural network. When the input binary image data is large, the hardware acceleration of the binary neural network can be achieved without performing a large number of operations, thereby saving a large amount of hardware resources and improving the hardware acceleration efficiency of the binary neural network.

[0066] 4. The present invention proposes a method for image classification based on full fusion and pruning technology of binary neural network, which supports hardware acceleration of binary neural networks with convolutional layers, pooling layers and the first fully connected layer of various sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of a method for realizing image classification based on full fusion and pruning technology of a binarized neural network according to the present invention;

[0068] Figure 2 It is a schematic diagram of reconstructing a convolution kernel in a method for realizing image classification based on full fusion and pruning technology of a binarized neural network of the present invention;

[0069] Figure 3 It is a binary neural network architecture diagram of a method for realizing image classification based on full fusion and pruning technology of a binary neural network according to the present invention;

[0070] Figure 4 It is a schematic diagram of a specific implementation process of a method for realizing image classification based on full fusion and pruning technology of a binary neural network of the present invention; DETAILED DESCRIPTION

[0071] The present invention will be further described below in conjunction with the accompanying drawings and embodiments, but is not limited thereto;

[0072] Example 1

[0073] A method for image classification based on full fusion and pruning technology of a binary neural network. Since there is data commonality among the convolutional layer, pooling layer and the first fully connected layer of the binary neural network, the present invention fuses the convolutional layer, pooling layer and the first fully connected layer of the binary neural network, and adopts a specific pruning method to reduce the number of operations in the fusion process. This method can effectively reduce the calculation time and the number of operations without reducing the accuracy of image classification. Since there are a large number of redundant operations in the fusion process of the convolutional layer, the pooling layer and the first fully connected layer, and these redundant operations do not cause a loss in the final image classification accuracy, the fusion process can be pruned. Figure 1 As shown, the steps include:

[0074] Step 1: Build a binary neural network architecture to implement the image binary classification task;

[0075] Step 2: Obtain the image data set and perform grayscale and 1,0 binary preprocessing on the original image data;

[0076] Step 3: Construct a sliding window with a size of R k *R k , a sliding window with a step size of m and a number of slides of c, where m is the product of the convolution step size and the pooling step size;

[0077] Step 4: Reconstruct the convolution kernel, that is, reconstruct the size and content of the 1,0 binary convolution kernel obtained by the binary neural network training. The reconstructed convolution kernel is named the reconstructed convolution kernel, and its size is R k *R k ;

[0078] Step 5: Prune the reconstructed convolution kernel, that is, remove the 0 part in the reconstructed convolution kernel;

[0079] Step 6: Select the weight of the first fully connected layer with a size of c*q, that is, select the weight of the first fully connected layer obtained by training the binary neural network, whose size is c*q, where c is the number of sliding window sliding times, and q is the output size of the first fully connected layer;

[0080] Step 7: Load the binary image preprocessed in step 2, the reconstructed convolution kernel pruned in step 5, and the weights of the first fully connected layer selected in step 6 into the hardware memory;

[0081] Step 8: Slide the sliding window from left to right and from top to bottom on the binary image preprocessed in step 2 with a step length of m;

[0082] Step 9: Perform convolution-pooling fusion pruning operation on the binary image data intercepted by the sliding window, that is, perform bit-by-bit AND operation on the binary image data intercepted by the sliding window and the pruned reconstructed convolution kernel, and remove the bit-by-bit AND operation that does not affect the classification accuracy during the bit-by-bit AND operation;

[0083] Step 10: Perform a fusion pruning operation on the convolution-pooling fusion pruning result of step 9 and part of the weights of the first fully connected layer selected in step 6, that is, the convolution-pooling fusion pruning result of the current data intercepted by the sliding window determines the output result of the first fully connected layer, and removes the bitwise XOR operation between the convolution-pooling fusion pruning result of the data intercepted by the sliding window and part of the weights of the first fully connected layer;

[0084] Step 11: Determine whether the sliding window has completed the traversal of the binary image data. If completed, proceed to step 12 to end the full fusion pruning operation of steps 9-10; otherwise, repeat steps 8-10;

[0085] Step 12: Accumulate and compare the fusion pruning operation results output from step 10 during the traversal process;

[0086] Step 13: Complete the operation of the second fully connected layer, that is, perform a bitwise XOR operation on the accumulation and comparison results obtained in step 12 and the weights of the second fully connected layer obtained by the binary neural network training;

[0087] Step 14: Take the category corresponding to the maximum value of the result in step 13 as the final classification label of the input image.

[0088] Example 2

[0089] The method for implementing image classification based on full fusion and pruning technology of binary neural network described in Example 1 is different in that:

[0090] like Figure 3 As shown in the figure, the network architecture of the binary neural network includes an input layer, a convolution layer, a pooling layer, two fully connected layers, and an output layer; the input layer preprocesses the original image into a binary image; the convolution layer extracts features of the input image data through the convolution kernel, and activates the convolution layer output with the Sign activation function; the pooling layer reduces network parameters by downsampling; the fully connected layer integrates the image features extracted by the convolution pooling layer, and activates the fully connected layer output with the Sign activation function; the output layer can be regarded as a classifier for predicting the category of the input image; the two fully connected layers include the first fully connected layer and the second fully connected layer.

[0091] In step 1, during the forward propagation of the binarized neural network, the convolution kernel is binarized based on the binary function Binarized, and the outputs of the convolution layer and the fully connected layer are binarized based on the sign function Sign; the Binarized function is shown in (Ⅰ), and the Sign function is shown in (Ⅱ):

[0092]

[0093]

[0094] In the process of back propagation, the Htanh function is used to calculate the gradient, where the Htanh function is shown in (III):

[0095] Htanh(x)=max(-1,min(1,x)) (Ⅲ)

[0096] In step 2, obtain an image dataset, that is, obtain an image dataset for training a binary neural network, and preprocess the original image data. In order to perform subsequent fusion and pruning operations between layers, perform 1,0 binarization on the input image data, process the input image into a grayscale image according to the grayscale processing principle, and construct the grayscale image into a binary image according to the selected binarization threshold as a binary image dataset for the binary neural network. The preprocessing consists of grayscale and 1,0 binarization operations. The grayscale operation refers to unifying the RGB values ​​of the three channels of each pixel point of the original image into a single channel value. The binarization operation refers to setting the image pixel values ​​greater than the selected threshold to 1 and the image pixel values ​​less than the selected threshold to 0.

[0097] In step 3, the calculation formula for the number of slides c is:

[0098]

[0099] In formula (IV), I is the size of the input binary image, R k is the unilateral size of the reconstructed convolution kernel, S is the convolution step size, and P is the unilateral size of the pooling layer.

[0100] In step 4, in order to save data storage between the convolution layer and the pooling layer and complete the operations of the convolution layer and the pooling layer in the same step, the size and content of the convolution kernel of size K*K obtained by the binary neural network training are reconstructed, and the reconstructed convolution kernel is named the reconstructed convolution kernel. According to the input binary data, binary convolution kernel, and convolution layer activation function, the size of R k *R k The reconstructed convolution kernel.

[0101] Resizing means changing the convolution kernel size obtained by training the binary neural network to R k*R k , where R k The calculation formula is shown in formula (V):

[0102] R k =K+(S*(P-1)) (V)

[0103] In formula (V), K represents the unilateral size of the convolution kernel of the convolution layer; S represents the convolution step size, (P-1) represents the unilateral size of the pooling layer minus 1; S*(P-1) represents the size of the remaining convolution kernels except the first convolution kernel.

[0104] In step 4, content reconstruction means: P*P identical convolution kernels are reconstructed with convolution step size S as the minimum unit according to coordinate C F place;

[0105] Coordinate C F As shown in formula (VI):

[0106]

[0107] In formula (VI), P represents the single-side size of the pooling layer, and i represents the i-th convolution kernel;

[0108] Convolution is performed on P*P identical convolution kernels with a convolution step size of S as the minimum unit according to the coordinate C F Placement specifically means: the first convolution kernel is placed at (0,0), the second convolution kernel is placed at (S,0), the Pth convolution kernel is placed at ((P-1)*S,0), the P+1th convolution kernel is placed at (0,S), the P+2th convolution kernel is placed at (S,S), and the P*Pth convolution kernel is placed at ((P-1)*S,(P-1)*S)); wherein, the overlapping parts between the convolution kernels used for reconstruction are operated according to the OR logic, which means: if the overlapping part has a value of 1, then the value of the part after reconstruction is 1; if the overlapping part does not have a value of 1, then the value of the part after reconstruction is 0.

[0109] In step 5, the pruning operation is performed on the reconstructed convolution kernel, which means: splitting the reconstructed convolution kernel containing 1, 0 into a reconstructed convolution kernel containing only 1, and pruning the 0 part; pruning the 0 part means setting the part corresponding to 0 in the reconstructed convolution kernel to empty and not treating it as part of the reconstructed convolution kernel.

[0110] In step 9, the convolution-pooling fusion and pruning operation is performed, which means that the input binary image data intercepted by the sliding window is subjected to a bit-by-bit AND operation with the reconstructed convolution kernel after pruning in step 5 to obtain the result of convolution-pooling fusion; in the process of performing the bit-by-bit AND operation, when the result of a certain bit is calculated to be 1, the remaining bits are no longer ANDed, and the output result is 1.

[0111] In step 10, the convolution-pooling fusion pruning result of step 9 is fused and pruned with the first fully connected layer weights selected in part of step 6, which means that after the convolution-pooling fusion and pruning operation of step 7 is completed on the binary image data intercepted by the sliding window for the jth sliding, the first fully connected layer operation of the binary image data currently intercepted by the sliding window is performed; the first fully connected layer weights selected in part of step 6 refer to the weights of the [(j-1)*q,j*q-1]th position, where j represents the jth sliding of the sliding window; q is the output size of the first fully connected layer.

[0112] In step 10, the convolution-pooling fusion pruning result of the j-th sliding intercepted data of the sliding window determines the result of the output of the first layer of the fully connected layer of the sliding intercepted data, which means that if the convolution-pooling fusion and pruning result of the j-th sliding intercepted data of the sliding window is 1, then the output of the first layer of the fully connected layer of the intercepted data is the weight data selected this time by the first layer of the fully connected layer; if the convolution-pooling fusion and pruning result of the j-th sliding intercepted data of the sliding window is 0, then the output of the first layer of the fully connected layer of the intercepted data is the bitwise inversion of the weight data selected this time by the first layer of the fully connected layer; the weights of the first layer of the fully connected layer and the bitwise inversion of the weights of the first layer of the fully connected layer are pre-stored in the memory, and the bitwise XOR operation of the convolution-pooling fusion pruning result and the weights of the first layer of the fully connected layer is removed. The bitwise XENO operation between the convolution-pooling fusion pruning result and the weight of the first fully connected layer is removed, that is, the output result of the first fully connected layer of the data intercepted by the sliding window for the jth time is obtained from the memory, and there is no need to complete the bitwise XENO operation between the convolution-pooling fusion pruning result and the selected weight of the first fully connected layer.

[0113] In step 12, accumulating and comparing the fusion pruning operation results outputted in step 10 during the traversal process means: adding the convolution-pooling fusion pruning results outputted in step 10 during the traversal process bit by bit, and determining bit by bit whether the result after addition is greater than If it is greater, the output result is 1; otherwise, the output result is 0; where c is the number of sliding times of the sliding window.

[0114] Example 3

[0115] According to the method for implementing image classification based on full fusion and pruning technology of binary neural network described in Example 2, as shown in FIG. Figure 4 As shown, the difference is:

[0116] The reconstructed convolution kernel is defined by the user. The schematic diagram of the reconstructed convolution kernel based on the full fusion technology and pruning method of the binary neural network is as follows: Figure 2 As shown, the convolution kernel size of the binary neural network is 3*3, the pooling layer size is 2*2, and the convolution step size is 1;

[0117] According to the size calculation formula of the reconstructed convolution kernel, the convolution kernel of size 3*3 obtained by the binary neural network training is resized to obtain a reconstructed convolution kernel of size 4*4.

[0118] Reconstruct the content of the convolution kernel: reconstruct the same convolution kernel with a number equal to the pooling layer size 2*2 and a convolution step size of 1 as the minimum unit according to the coordinate C F Place. That is, the first convolution kernel starts at (0, 0) at the bottom left, the second convolution kernel starts at (1, 0) at the bottom left, the third convolution kernel starts at (0, 1) at the bottom left, and the fourth convolution kernel starts at (1, 1) at the bottom left. The overlapping parts between the convolution kernels used for reconstruction are operated according to the OR logic. If the overlapping part has a value of 1, then the value of the part after reconstruction is 1. If the overlapping part does not have a value of 1, then the value of the part after reconstruction is 0. That is, the overlapping part A is operated according to the OR logic to obtain the result a: [1 1], the overlapping part B is obtained by b: [1 1], the overlapping part C is obtained by c: [1 0], the overlapping part D is obtained by d: [0 0], and the overlapping part E is obtained by e: result.

[0119] Example 4

[0120] The method for implementing image classification based on full fusion and pruning technology of binary neural network described in Example 2 is different in that:

[0121] The binary neural network convolution kernel size selected in this embodiment is 3*3, the pooling layer size is 2*2, the convolution step size is 1, and the pooling step size is 2.

[0122] Construct a sliding window. Construct a sliding window of size 4*4 and stride M, where M = convolution stride * pooling stride = 2;

[0123] The specific steps of reconstructing the content of the convolution kernel include: reconstructing the same convolution kernel with a number equal to the pooling layer size 2*2 with a convolution step size of 1 as the minimum unit according to the coordinate C F Placement. That is, the first convolution kernel is placed at (0, 0), the second convolution kernel is placed at (1, 0), the third convolution kernel is placed at (0, 1), and the fourth convolution kernel is placed at (1, 1). The overlapping parts between the convolution kernels used for reconstruction are operated according to the OR logic. If the overlapping part has a value of 1, then the value of this part after reconstruction is 1. If the overlapping part does not have a value of 1, then the value of this part after reconstruction is 0.

[0124] This embodiment uses the MIT-BIH dataset of the Massachusetts Institute of Technology, one of the internationally recognized standard ECG datasets, and compares the traditional hardware acceleration method of the binary neural network and the full fusion technology and pruning method based on the binary neural network based on the recognition of two ECG signals. Table 1 lists the voltage, clock frequency, network size, and network recognition accuracy used in this embodiment: the number of lookup tables, the number of registers, the number of cycles required for a single recognition, the overall delay, and the number of operation operations used by the full fusion technology and pruning method based on the binary neural network are compared with the number of lookup tables, the number of registers, the number of cycles required for a single recognition, the overall delay, and the number of operation operations used by the traditional hardware acceleration method of the binary neural network, and the percentage of reduction in the number of operation operations of the full fusion technology and pruning method based on the binary neural network compared to the traditional hardware acceleration method is listed.

[0125] Table 1

[0126]

[0127]

[0128] As can be seen from Table 1, the full fusion technology and pruning method based on the binary neural network are superior to the traditional hardware acceleration method of the binary neural network in terms of the number of hardware resources used, the number of cycles for a single recognition, the inference delay, and the number of operations. The number of lookup tables is reduced by about 40%, the number of registers is reduced by about 54%, the number of cycles required for a single recognition is reduced by about 92%, the delay time is reduced by about 92% (5.88us), and the overall number of operations is reduced by about 94.3%, greatly improving the efficiency.

[0129] Example 5

[0130] A picture classification system based on full fusion and pruning technology of binary neural network, including:

[0131] The image data set acquisition and preprocessing module is configured to: acquire the image data set and preprocess the image data using a 1,0 binarization method;

[0132] The reconstructed convolution kernel and pruning module is configured to reconstruct the size and content of the convolution kernel obtained by binarization neural network training, and name the reconstructed convolution kernel as the reconstructed convolution kernel with a size of R. k *R k ; And perform pruning operation on the reconstructed convolution kernel, that is, remove the 0 part in the reconstructed convolution kernel;

[0133] The first fully connected layer weight selection module is configured to: select the first fully connected layer weight of size c*q, that is, select the first fully connected layer weight obtained by binarization neural network training;

[0134] The convolution-pooling fusion pruning operation module is configured to: perform a convolution-pooling fusion pruning operation on the binary image data intercepted by the sliding window, that is, perform a bitwise AND operation on the binary image data intercepted by the sliding window and the pruned reconstructed convolution kernel, and remove the bitwise AND operation that does not affect the classification accuracy during the bitwise AND operation;

[0135] The fully connected fusion pruning operation module is configured to: perform a fusion pruning operation on the convolution-pooling fusion pruning result and the selected first-layer fully connected layer weights, that is, the convolution-pooling fusion pruning result of the current data intercepted by the sliding window determines the output result of the first-layer fully connected layer, and removes the bitwise XOR operation between the convolution-pooling fusion pruning result of the data intercepted by the sliding window and part of the first-layer fully connected layer weights;

[0136] The sliding window sliding and traversal judgment module is configured to: slide the sliding window on the preprocessed binary image with a step length of m according to the rule of from left to right and from top to bottom; and judge whether the sliding window has completed the traversal of the binary image data;

[0137] The accumulation and comparison operation module is configured to: accumulate and compare all fusion and pruning operation results after the sliding window completes the traversal of the binary image data;

[0138] The subsequent fully connected layer operation module is configured to: complete the second layer fully connected layer operation, that is, perform a bitwise XOR operation on the accumulated and compared results and the second layer fully connected layer weights obtained by the binary neural network training;

[0139] The image classification result output module is configured to use the category corresponding to the maximum value of the result of the subsequent fully connected layer operation module as the final classification label of the input image.

[0140] Example 6

[0141] A computer-readable storage medium, wherein a memory stores a plurality of instructions, the instructions being suitable for being loaded by a processor of a terminal device and executing a method for implementing image classification based on full fusion and pruning technology of a binary neural network as described in any one of Examples 1-4.

[0142] Example 7

[0143] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions and the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executed by any one of embodiments 1-4. A method for implementing image classification based on full fusion and pruning technology of a binary neural network.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0145] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the processes and / or boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0149] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for image classification based on full fusion and pruning technology of binary neural network, characterized in that: The steps include: Step 1: Build a binary neural network architecture to implement the image binary classification task; Step 2: Obtain the image data set and perform grayscale and 1,0 binary preprocessing on the original image data; Step 3: Construct a sliding window with a size of R k *R k , a sliding window with a step size of m and a number of slides of c, where m is the product of the convolution step size and the pooling step size; Step 4: Reconstruct the convolution kernel, that is, reconstruct the size and content of the 1,0 binary convolution kernel obtained by the binary neural network training, and name the reconstructed convolution kernel as the reconstructed convolution kernel, with a size of R k *R k ; Step 5: Prune the reconstructed convolution kernel, that is, remove the 0 part in the reconstructed convolution kernel; Step 6: Select the weight of the first fully connected layer with a size of c*q, that is, select the weight of the first fully connected layer obtained by training the binary neural network, whose size is c*q, where c is the number of sliding window sliding times, and q is the output size of the first fully connected layer; Step 7: Load the binary image preprocessed in step 2, the reconstructed convolution kernel pruned in step 5, and the weights of the first fully connected layer selected in step 6 into the hardware memory; Step 8: Slide the sliding window from left to right and from top to bottom on the binary image preprocessed in step 2 with a step length of m; Step 9: Perform convolution-pooling fusion pruning operation on the binary image data intercepted by the sliding window, that is, perform bit-by-bit AND operation on the binary image data intercepted by the sliding window and the pruned reconstructed convolution kernel, and remove the bit-by-bit AND operation that does not affect the classification accuracy during the bit-by-bit AND operation; Step 10: Perform a fusion pruning operation on the convolution-pooling fusion pruning result of step 9 and part of the weights of the first fully connected layer selected in step 6, that is, the convolution-pooling fusion pruning result of the current data intercepted by the sliding window determines the output result of the first fully connected layer, and removes the bitwise XOR operation between the convolution-pooling fusion pruning result of the data intercepted by the sliding window and part of the weights of the first fully connected layer; Step 11: Determine whether the sliding window has completed the traversal of the binary image data. If completed, proceed to step 12 to end the full fusion pruning operation of steps 9-10; otherwise, repeat steps 8-10; Step 12: Accumulate and compare the fusion pruning operation results output from step 10 during the traversal process; Step 13: Complete the operation of the second fully connected layer, that is, perform a bitwise XOR operation on the accumulation and comparison results obtained in step 12 and the weights of the second fully connected layer obtained by the binary neural network training; Step 14: Take the category corresponding to the maximum value of the result in step 13 as the final classification label of the input image.

2. According to claim 1, a method for image classification based on full fusion and pruning technology of binary neural network, characterized in that: The network architecture of the binary neural network includes an input layer, a convolution layer, a pooling layer, two fully connected layers, and an output layer; the input layer preprocesses the original image into a binary image; the convolution layer extracts features of the input image data through the convolution kernel, and activates the convolution layer output with the Sign activation function; the pooling layer reduces network parameters by downsampling; the fully connected layer integrates the image features extracted by the convolution and pooling layers, and activates the fully connected layer output with the Sign activation function; the output layer can be regarded as a classifier for predicting the category of the input image; the two fully connected layers include the first fully connected layer and the second fully connected layer.

3. The method for realizing image classification based on full fusion and pruning technology of binary neural network according to claim 1 is characterized in that: In the step 1, in the forward propagation process of the binarized neural network, the convolution kernel is binarized based on the binary function Binarized, and the outputs of the convolution layer and the fully connected layer are binarized based on the sign function Sign; The Binarized function is shown in (I) and the Sign function is shown in (II): In the back propagation process, the Htanh function is used to calculate the gradient, where the Htanh function is shown in (III): Htanh(x)=max(-1,min(1,x)) (III).

4. The method for realizing image classification based on full fusion and pruning technology of binary neural network according to claim 1 is characterized in that: In step 3, the calculation formula for the number of slides c is: In formula (VI), I is the size of the input binary image, R k is the unilateral size of the reconstructed convolution kernel, S is the convolution step size, and P is the unilateral size of the pooling layer; In step 4, the size reconstruction means: changing the convolution kernel size obtained by binarization neural network training to R k *R k , where R k The calculation formula is shown in formula (V): R k =K+(S*(P-1)) (V) In formula (V), K represents the unilateral size of the convolution kernel of the convolution layer; S represents the convolution step size, (P-1) represents the unilateral size of the pooling layer minus 1; S*(P-1) represents the size of the remaining convolution kernels except the first convolution kernel; Content reconstruction means: P*P identical convolution kernels are reconstructed with convolution step size S as the minimum unit according to coordinate C F place; The coordinate C F As shown in formula (VI): In formula (VI), P represents the single-side size of the pooling layer, and i represents the i-th convolution kernel; Convolution is performed on P*P identical convolution kernels with a convolution step size of S as the minimum unit according to the coordinate C F Placement specifically means: the first convolution kernel is placed at (0, 0), the second convolution kernel is placed at (S, 0), the Pth convolution kernel is placed at ((P-1)*S, 0), the P+1th convolution kernel is placed at (0, S), the P+2th convolution kernel is placed at (S, S), and the P*Pth convolution kernel is placed at ((P-1)*S, (P-1)*S)); wherein, the overlapping parts between the convolution kernels used for reconstruction are operated according to the OR logic, which means: if the overlapping part has a value of 1, then the value of the part after reconstruction is 1; if the overlapping part does not have a value of 1, then the value of the part after reconstruction is 0.

5. The method for realizing image classification based on full fusion and pruning technology of binary neural network according to claim 1 is characterized in that: In the step 5, the pruning operation on the reconstructed convolution kernel means: splitting the reconstructed convolution kernel containing 1 and 0 into a reconstructed convolution kernel containing only 1, and pruning the 0 part; pruning the 0 part means setting the part corresponding to 0 in the reconstructed convolution kernel to be empty and not being used as part of the reconstructed convolution kernel; In step 9, the convolution-pooling fusion and pruning operation is performed, which means that the input binary image data intercepted by the sliding window is subjected to a bit-by-bit AND operation with the reconstructed convolution kernel after pruning in step 5 to obtain the result of convolution-pooling fusion; in the process of performing the bit-by-bit AND operation, when a certain bit result is calculated to be 1, the remaining bits are no longer subjected to the AND operation, and the output result is 1.

6. The method for realizing image classification based on full fusion and pruning technology of binary neural network according to claim 1 is characterized in that: In the step 10, the convolution-pooling fusion pruning result of step 9 is fused and pruned with the first fully connected layer weight selected in part of step 6, which means that after the binary image data intercepted by the sliding window for the jth sliding is completed with the convolution-pooling fusion and pruning operation of step 7, the first fully connected layer operation of the binary image data currently intercepted by the sliding window is performed; the first fully connected layer weight selected in part of step 6 refers to the weight of the [(j-1)*q, j*q-1]th position, where j represents the jth sliding of the sliding window; q is the output size of the first fully connected layer; The convolution-pooling fusion and pruning result of the j-th sliding intercepted data of the sliding window determines the result of the output of the first layer of the fully connected layer of the sliding intercepted data, which means that if the convolution-pooling fusion and pruning result of the j-th sliding intercepted data of the sliding window is 1, then the output of the first layer of the fully connected layer of the intercepted data is the weight data selected this time by the first layer of the fully connected layer; if the convolution-pooling fusion and pruning result of the j-th sliding intercepted data of the sliding window is 0, then the output of the first layer of the fully connected layer of the intercepted data is the bitwise inversion of the weight data selected this time by the first layer of the fully connected layer; the weights of the first layer of the fully connected layer and the bitwise inversion of the weights of the first layer of the fully connected layer are pre-stored in the memory, and the bitwise XOR operation between the convolution-pooling fusion and pruning results and the weights of the first layer of the fully connected layer is removed.

7. A method for image classification based on full fusion and pruning technology of binary neural network according to any one of claims 1 to 6, characterized in that: In the step 12, accumulating and comparing the fusion pruning operation results outputted in step 10 during the traversal process means: adding the convolution-pooling fusion pruning results outputted in step 10 during the traversal process bit by bit, and determining bit by bit whether the result after addition is greater than If it is greater, the output result is 1; otherwise, the output result is 0; where c is the number of sliding times of the sliding window.

8. A picture classification system based on full fusion and pruning technology of binary neural network, characterized in that: include: The image data set acquisition and preprocessing module is configured to: acquire the image data set and preprocess the image data using a 1, 0 binarization method; The reconstructed convolution kernel and pruning module is configured to reconstruct the size and content of the convolution kernel obtained by binarization neural network training, and name the reconstructed convolution kernel as the reconstructed convolution kernel with a size of R. k *R k ; And perform pruning operation on the reconstructed convolution kernel, that is, remove the 0 part in the reconstructed convolution kernel; The first fully connected layer weight selection module is configured to: select the first fully connected layer weight of size c*q, that is, select the first fully connected layer weight obtained by binarization neural network training; The convolution-pooling fusion pruning operation module is configured to: perform a convolution-pooling fusion pruning operation on the binary image data intercepted by the sliding window, that is, perform a bitwise AND operation on the binary image data intercepted by the sliding window and the pruned reconstructed convolution kernel, and remove the bitwise AND operation that does not affect the classification accuracy during the bitwise AND operation; The fully connected fusion pruning operation module is configured to: perform a fusion pruning operation on the convolution-pooling fusion pruning result and the selected first-layer fully connected layer weights, that is, the convolution-pooling fusion pruning result of the current data intercepted by the sliding window determines the output result of the first-layer fully connected layer, and removes the bitwise XOR operation between the convolution-pooling fusion pruning result of the data intercepted by the sliding window and part of the first-layer fully connected layer weights; The sliding window sliding and traversal judgment module is configured to: slide the sliding window on the preprocessed binary image with a step length of m according to the rule of from left to right and from top to bottom; Determine whether the sliding window has completed the traversal of the binary image data; The accumulation and comparison operation module is configured to: accumulate and compare all fusion and pruning operation results after the sliding window completes the traversal of the binary image data; The subsequent fully connected layer operation module is configured to: complete the second layer fully connected layer operation, that is, perform a bitwise XOR operation on the accumulated and compared results and the second layer fully connected layer weights obtained by the binary neural network training; The image classification result output module is configured to use the category corresponding to the maximum value of the result of the subsequent fully connected layer operation module as the final classification label of the input image.

9. A computer-readable storage medium, wherein a memory stores a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing a method for implementing image classification based on full fusion and pruning technology of a binary neural network as described in any one of claims 1-7.

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing a method for image classification based on full fusion and pruning technology of a binary neural network as described in any one of claims 1-7.

Images