A neural network compression method and system based on random projection hashing

Abstract

This invention discloses a neural network compression method and system based on random projection hashing. The method includes the following steps: in the forward propagation, the input feature map and weight matrix of each neural network layer are compressed using a projection matrix to calculate the output feature map; in the backward propagation, the neural network's loss function is calculated based on the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated using the loss function; the weight matrix is ​​updated based on the gradient value of the weight matrix of the first layer. This method can customize the compression factor according to the accuracy and compression rate requirements, and has high flexibility.

Description

Technical Field

[0001] The present invention relates to a neural network compression method based on random projection hashing, and also relates to a neural network compression system for implementing the method. Background Art

[0002] In recent years, deep neural networks have demonstrated tremendous potential in many fields, including computer vision and speech recognition. Thanks to the rapid development of big data technology and GPU parallel computing, which provide powerful hardware support for computing complex networks and large neural networks, neural network models and deep learning methods are increasingly being applied to computer vision. They have significantly improved the accuracy and performance of other methods in areas such as object recognition, image classification, image retrieval, face verification, and video understanding. In computer vision, convolutional neural networks, a type of deep neural network, excels at mimicking the abstraction and iteration processes of the human brain, enabling them to accurately extract information from big data and achieve the highest standards in many applications. At the same time, we are seeing interesting advances in computer vision applications in virtual reality, augmented reality, and smart wearable devices. Overall, the need to integrate high-performance recognition systems into smart portable devices is urgent.

[0003] However, today's convolutional neural network-based recognition systems require large amounts of memory and high-performance computing power, typically running on expensive GPU clusters. While neural network training can be done on GPU clusters, real-time testing requires mobile devices. However, as datasets and features increase, deep convolutional neural networks also require larger models, larger parameter storage, and more computational complexity. This creates a high demand for computing power, making them nearly impossible to use on mobile or embedded devices, hindering their development.

[0004] As we all know, mobile devices, embedded devices, and regular mobile computers are limited by memory, computing power, and energy consumption. Most mobile devices have only 1GB of random access memory (RAM), but the parameters of the convolutional neural network trained to win the 2014 ImageNet competition reached 576MB, which consumes a large amount of RAM. Furthermore, loading over 500MB of parameters to test a single image is energy-consuming and unacceptable, not to mention the computational time. These high memory and computing power requirements are particularly evident on mobile devices. Therefore, these models significantly exceed the memory, power, and computing power requirements of mobile phones, embedded devices, and smart wearables.

[0005] However, more and more deep learning applications are moving towards mobile devices and embedded devices, such as image classification on smartphones and robots, and self-driving cars require real-time object recognition. Therefore, how to compress neural networks to reduce computing power and storage space has become an urgent need. Summary of the Invention

[0006] In view of the deficiencies of the existing technology, the primary technical problem to be solved by the present invention is to provide a neural network compression method based on random projection hashing.

[0007] Another technical problem to be solved by the present invention is to provide a neural network compression system based on random projection hashing.

[0008] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0009] According to a first aspect of an embodiment of the present invention, a neural network compression method based on random projection hashing is provided, comprising the following steps:

[0010] In the forward propagation, the input feature map and weight matrix of each neural network layer are compressed through the projection matrix to calculate the output feature map;

[0011] In the backward propagation, the loss function of the neural network is calculated based on the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated through the loss function;

[0012] Update the weight matrix according to the gradient value of the weight matrix of each layer.

[0013] Preferably, in the forward propagation, the input feature map and weight matrix of each neural network layer are compressed by the projection matrix, and the output feature map is calculated using the following steps:

[0014] Multiply the compressed input feature map and the weight matrix to obtain the output feature map.

[0015] Preferably, when compressing the input feature map and weight matrix of each neural network layer through the projection matrix, the input feature map and the weight matrix are compressed through the same projection matrix.

[0016] Preferably, the input feature map and weight matrix of each neural network layer are compressed through a projection matrix, including the following steps:

[0017] For the input feature map S k and the weight matrix W k , through the real-valued projection matrix Project the input feature map S k And the weight matrix is ​​converted into a low-dimensional input feature map and a low-dimensional weight matrix

[0018] The low-dimensional input feature map and a low-dimensional weight matrix Convert to binary hash code and Complete the compression of input feature maps and weight matrices;

[0019] Among them, k is the number of neural network layers and b is the compression factor.

[0020] Preferably, in the forward propagation, the input feature map and weight matrix of the convolutional layer are compressed by the projection matrix to calculate the output feature map; the steps include:

[0021] Reorganize the input feature map and weight matrix into a large matrix respectively;

[0022] Compress the reorganized input feature map and weight matrix through the projection matrix;

[0023] Multiply the compressed input feature map and the weight matrix to obtain the output feature map;

[0024] Perform col 2im operation on the output feature map to obtain the final output feature map.

[0025] Preferably, in the backward propagation, the gradient value of the input feature map and the weight matrix of each layer is calculated by the loss function; the non-differentiable y=sgn(x) function used in the forward process is approximated by y=x, so when sgn(x) included in the loss function is replaced by x, the gradient value of the input feature map is for:

[0026]

[0027] Gradient value of the weight matrix for:

[0028]

[0029] in, is the gradient value of the output feature map; is the weight matrix after hash encoding of real values; is a real-valued projection matrix; is the input feature map after hash encoding.

[0030] Preferably, in the backward propagation, the gradient value of the input feature map and the weight matrix of each layer is calculated by the loss function; the non-differentiable y=sgn(x) function used in the forward process is approximated by y=Htanh(x), so when sgn(x) contained in the loss function is replaced by Htanh(x), the gradient value of the input feature map is for:

[0031]

[0032] Gradient value of the weight matrix for:

[0033]

[0034] in, is the gradient value of the output feature map; is the weight matrix after hash encoding of real values; is a real-valued projection matrix; is the input feature map after hash encoding; Represents the gradient of the Hardtanh function.

[0035] The best way to update the weight matrix according to the gradient value of the weight matrix of each layer is to The matrix is ​​truncated using the clip function. Transform it into a new matrix where each element is in the range of [-1, 1], and assign the new matrix to the weight matrix W of the kth layer k ;

[0036] Where η is the learning rate; is the gradient value of the weight matrix; W k is the weight matrix.

[0037] Preferably, when a bias matrix exists when data is input into the neural network, the following steps are also included:

[0038] The gradient value of the bias matrix of each layer is calculated according to the loss function, and the bias matrix is ​​updated according to the gradient value of the bias matrix of each layer.

[0039] According to a second aspect of an embodiment of the present invention, a neural network compression system based on random projection hashing is provided, comprising a processor and a memory; the memory stores a computer program executable on the processor, and when the computer program is executed by the processor, the following steps are implemented:

[0040] In the forward propagation, the input feature map and weight matrix of each neural network layer are compressed through the projection matrix to calculate the output feature map;

[0041] In the backward propagation, the loss function of the neural network is calculated based on the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated through the loss function;

[0042] Update the weight matrix according to the gradient value of the weight matrix of each layer.

[0043] The neural network compression method based on random projection hashing provided by the present invention quantizes and approximates parameters in random projection through the random projection hashing algorithm, can be applied to the neural network faster and with less memory than similar methods, and can personalize the compression factor according to the needs of accuracy and compression rate, has high flexibility, and greatly reduces the storage requirements of parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of the neural network compression method based on random projection hashing provided by the present invention;

[0045] Figure 2 Schematic diagram of im2col operation in the embodiment provided by the present invention;

[0046] Figure 3 A schematic diagram of the Hardtanh function in the embodiment provided by the present invention;

[0047] Figure 4 This is a structural diagram of the neural network compression system based on random projection hashing provided by the present invention. DETAILED DESCRIPTION

[0048] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] Currently, many papers are researching compression acceleration for neural networks, particularly those that provide general formulas for binarization. However, current compression methods simply convert the weight matrix and input matrix into the same binary number. This results in a compression factor of 32, which is equivalent to converting a 4-byte real number into a 1-bit number. Computationally, this method significantly accelerates the speedup, as operations such as bitwise operations can be performed between bits. Furthermore, this method is simple, straightforward, easy to implement, and highly effective. However, the fixed compression factor of 32 is very restrictive. For example, on a small network, a 32-fold compression may result in too few parameters and a significant loss in accuracy. On a large network, a 32-fold compression ratio may be too low, and the model will still be very large after compression.

[0050] Therefore, in the neural network compression method based on random projection hashing provided by the present invention, improvements are made by combining the projection compression method. By adding a hyperparameter compression factor, compression can be performed on the input matrix, and the weight matrix will also have the same compression in one of its dimensions. Similarly, a binary input matrix and weight matrix will be obtained in the end. Therefore, the compression ratio that can be achieved can exceed the fixed compression ratio used in some existing research methods. At the same time, the compression ratio can also be customized according to the specific size of the network, which is more flexible. Moreover, because the compression ratio can be larger, when loading and testing on mobile devices, the required computing power and storage space are smaller, energy consumption is lower, and it is more adaptable to operations on low-performance devices.

[0051] like Figure 1 As shown, the neural network compression method based on random projection hashing provided by the present invention includes the following steps: first, in the forward propagation, the input feature map and weight matrix of each neural network layer are compressed using the projection matrix to calculate the output feature map; then, in the backward propagation, the loss function of the neural network is calculated based on the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated using the loss function; finally, the weight matrix is ​​updated based on the gradient value of the weight matrix of each layer. This process is described in detail below.

[0052] S1, in the forward propagation, the input feature map and weight matrix of each neural network layer are compressed through the projection matrix to calculate the output feature map.

[0053] Before describing the neural network compression method based on random projection hashing provided by the present invention, let's first introduce the hash algorithm. The main idea of ​​the hash algorithm is to map binary values ​​of arbitrary length to binary values ​​of fixed length using a designed hash function. These fixed-length binary values ​​are called hash values. This method compresses messages or data of varying sizes into a fixed data format, significantly reducing the amount of data. Hash random projection is used for retrieval and storage in the image field and is widely used for data encryption in the field of cryptography.

[0054] The three main elements of a hash algorithm are the input space, the hash function, and the output space. The output space is the space containing the hash value, so the most important thing for a hash algorithm is the hash function. Generally speaking, a hash algorithm requires several hash functions to work together to convert the original data into a hash code that maintains consistency with the input data. That is to say, if two sets of input data are similar in the input space, then their hash codes will also be similar in the output space, and vice versa. Among them, the locality sensitive hashing algorithm is a widely used hash algorithm. The general definition is as follows:

[0055] Given a set of hash functions H = {h:D→U}, for each function h∈H in H, for any two vectors p,q∈D, if the following conditions are met:

[0056] If d(p,q)≤d1, then Pr[h(q)=h(p)]≥P1;

[0057] If d(p,q)≥d2, then Pr[h(q)=h(p)]≤P2.

[0058] Where d(p,q) is the distance between p and q, Pr[h(q)=h(p)] represents the probability that h(q) and h(p) are equal, d1, d2, P1, P2 are thresholds, and generally d1<d2, P1>P2. Then we call the hash function cluster H position-sensitive, that is, (d1, d2, P1, P2)-sensitive.

[0059] In the image field, hash algorithms are widely used in image retrieval, image storage, video retrieval, and video storage. In traditional image retrieval, the image features extracted directly from images are high-dimensional and high-precision data. Using these features directly as a retrieval basis requires extremely complex calculations. Furthermore, image databases typically contain millions of images, making this computation time-consuming and difficult to achieve real-time requirements. Hash-based retrieval, on the other hand, directly utilizes hash codes derived from image feature transformations for similarity calculations and retrieval, significantly reducing computation time and storage space.

[0060] In the embodiments provided by the present invention, the random projection hashing algorithm is applied to the compression of the neural network. In the forward propagation, the input feature map and weight matrix of each neural network layer are compressed using the projection matrix to calculate the output feature map. The specific steps include:

[0061] S11, compress the input feature map and weight matrix of each neural network layer through the projection matrix.

[0062] When the input feature map and weight matrix of each neural network layer are compressed by the projection matrix, the input feature map and weight matrix are compressed by the same projection matrix; specifically, the steps include:

[0063] S111, for the input feature map S k and the weight matrix W k , through the real-valued projection matrix Project it and convert it into a low-dimensional input feature map and a low-dimensional weight matrix

[0064] S112, low-dimensional input feature map and a low-dimensional weight matrix Convert to binary hash code and Complete the compression of the input feature map and weight matrix. Where k is the number of neural network layers and b is the compression factor.

[0065] Then further action is performed on the hash code. In projection hashing, the hash matrix P may be data-insensitive, such as random projection.

[0066] S12, the compressed input feature map S k and the weight matrix W k Multiply them together to get the output feature map.

[0067] In the embodiment provided by the present invention, during the forward propagation process, the input feature map and weight matrix of each neural network layer are compressed by the same projection matrix, and the matrix obtained by multiplying the compressed matrices is used as the approximation of the original output feature map. That is, the original output feature map is converted into an approximation of the original output feature map by the real-valued input feature map S k , weight matrix W k Multiplication is calculated, and now the input matrix after binarization is obtained Weight Matrix Multiply and calculate, that is:

[0068]

[0069] Specifically, in each layer, assuming there are L layers of networks, the network operations for each layer, i.e., k=1 to L, are as follows:

[0070] Among them, for the fully connected layer, the fully connected layer performs a linear operation, and its calculation formula is as follows:

[0071]

[0072] Among them, R is the size of the compressed matrix, and m*n is the number of rows and columns of the matrix corresponding to the compressed input feature map. r are the number of rows and columns of the matrix corresponding to the compressed feature matrix, T k is the output feature map; C r *n are the number of rows and columns of the matrix corresponding to the compressed output feature map.

[0073] The convolutional layer performs a convolution operation, which is different from the linear operation in the fully connected layer. However, in the code of deep learning platforms such as Torch and Caffe, convolution is actually implemented by converting the high-dimensional matrix of the original input feature map into a corresponding form according to certain rules, and the weight matrix is ​​also converted into the corresponding form, and then a linear operation is performed. This rule is the im2col operation.

[0074] In the embodiment provided by the present invention, in the forward propagation, the input feature map and weight matrix of the convolutional layer are compressed by the projection matrix to calculate the output feature map; specifically, the following steps are included:

[0075] S01, reorganizes the input feature map and weight matrix into a larger matrix.

[0076] S02, compressing the reorganized input feature map and weight matrix through the projection matrix.

[0077] S03, multiplying the compressed input feature map and the weight matrix to obtain an output feature map.

[0078] S04: perform a col 2im operation on the output feature map to obtain the final output feature map.

[0079] Specifically, such as Figure 2 As shown, in the specific implementation, the input feature map and the weight matrix are reorganized into a larger matrix, wherein the process of reorganizing the input feature map and the weight matrix into a larger matrix adopts the conventional reorganization process in this field, which is not specifically limited here. Then, the reorganized input feature map and weight matrix are compressed through the projection matrix, and the compressed input feature map and weight matrix are multiplied to obtain the output feature map. Finally, the output feature map is subjected to the col 2im operation (col 2i operation is a conventional operation in this field and will not be repeated here) to implement convolution more efficiently. So in essence, there is not much difference between the compression implementation of the convolution layer and the fully connected layer, only a slight difference. So the calculation formula of the convolution layer is as follows:

[0080]

[0081] Among them, the binary hash code The size of the corresponding matrix R is b*(d t *d t ). m*C r and (d t *d t )*C r The meaning is the same as b*(d t *d t ) are similar and will not be described in detail here.

[0082] S2, in the back propagation, the loss function of the neural network is calculated according to the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated through the loss function; specifically, the loss function of the neural network is calculated according to the output feature map, and the gradient initial value of the final output layer can be obtained through the loss function Because the output of the k-1th layer corresponds to the input of the kth layer, that is, T k-1 =S k , calculate the gradient value of the input feature map and the weight matrix; for k = L to 2, recursively layer by layer, the gradient value of the input feature map and weight matrix of each layer can be obtained, and finally the weight matrix corresponding to the input feature map when the neural network initially inputs data is obtained, and the weight matrix is ​​updated by performing corresponding operations on the initial weight matrix.

[0083] In the embodiment provided by the present invention, the loss function of the neural network is calculated based on the output feature map, and the loss function of the neural network is calculated using the traditional cross entropy method, which will not be described in detail here.

[0084] Because some transformations are made to the input feature map and the output feature map during the forward propagation process, the gradients of the parameters will also change accordingly during the backward propagation. In addition, due to the existence of the projection matrix, the gradients of the input matrix and the weight matrix will change accordingly due to the operation of the projection matrix. It is necessary to further deduce their gradients through the chain rule.

[0085] For calculating the gradient parameters, the gradient of the final output layer of the convolutional neural network can be obtained directly, from which we can easily get the gradient of the error rate of each layer with respect to the output feature map. For k = 1 to L, According to the chain rule, we can get as follows:

[0086]

[0087] According to the forward propagation formula, we can get

[0088] In the above formula, Because it involves the derivative of the sgn function, which is not continuously differentiable, it cannot be directly given. Therefore, it is generally necessary to do some different treatments when derivatizing it. Most of the time, some approximate continuous functions are used for approximation. Therefore, different derivation treatments are performed according to different approximate functions.

[0089] If the approximation of y=sgn(x) is done with y=x, that is, directly approximating sgn, You can find:

[0090]

[0091] So the final gradient expression is:

[0092]

[0093] If y = sgn (x) is approximated by y = Htanh (x) = max (-1, min (1, x)), that is, approximated by the Hardtanh function, the Hardtanh function is as follows Figure 3 shown.

[0094] When x>=-1 and x<=1, otherwise Use 1 |x|≤1 Represents the gradient of the Hardtanh function, which can be obtained So the final gradient expression is:

[0095]

[0096] S3, updates the weight matrix according to the gradient value of the weight matrix of each layer.

[0097] Update the weight matrix according to the gradient value of the weight matrix of each layer using the following formula:

[0098]

[0099] Among them, η is the learning rate, which can be set according to the needs. When updating the weight matrix, The matrix is ​​truncated using the clip function. Transform it into a new matrix where each element is in the range of [-1,1] and assign it to the weight matrix W k .

[0100] In another embodiment provided by the present invention, when data is input into the neural network, there is a bias matrix b k In the forward propagation, the input feature map and weight matrix of each neural network layer are compressed by the projection matrix to calculate the output feature map; the output feature map at this time is:

[0101]

[0102] Then, in the backward propagation, the loss function of the neural network is calculated according to the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated through the loss function; finally, the weight matrix is ​​updated according to the gradient value of the weight matrix of each layer, and the gradient value of the bias matrix of each layer is calculated according to the loss function, and the bias matrix is ​​updated according to the gradient value of the bias matrix of each layer, using the following formula:

[0103]

[0104] When updating the bias matrix, The matrix is ​​truncated using the clip function. Transform it into a new matrix in which each element is in the range of [-1,1] and assign it to the bias matrix b of the first layer k .

[0105] The following is an analysis of the specific performance of the neural network compression method based on random projection hashing provided by the present invention in terms of compression and acceleration. The algorithm complexity is as follows:

[0106]

[0107]

[0108] Table 1 Algorithm complexity analysis table

[0109] In the fully connected layer, when actually compressing, we take the compression factor as n (n is greater than 1), then b is the input vector C s 1 / n, so we set b=C s Substituting / n into the table, we get the simplified algorithm complexity as follows:

[0110]

[0111] Table 2. Complexity analysis of the fully connected layer algorithm

[0112] From the ratio we can discuss three situations:

[0113] When C s ≈C t When , the computational acceleration is about n / 2, and the storage compression is about n.

[0114] When C s 》C t When , the calculation acceleration is The storage compression factor is approximately When C s When it is large enough, the acceleration and compression are both less than 1, and the effect is poor.

[0115] When C s C t When , the computational acceleration factor is n, and the storage compression factor is about 32n.

[0116] In actual use, the first case is more common, so in theory the compression is actually effective.

[0117] In the convolutional layer, when actually compressing, we take the compression factor as n (n>1), then b is C s d k 2 1 / n, so we set b=C s d k 2Substituting / n into the table, we get the simplified algorithm complexity as follows:

[0118]

[0119] Table 3 Convolutional layer algorithm complexity analysis

[0120] Among them, D k is the spatial scale of the weight w, D t is the spatial scale of the output feature map.

[0121] From the ratio we can discuss three situations:

[0122] When d k 2 C s ≈C t When , the computational acceleration is about n / 2, and the storage compression is about n.

[0123] when When , the calculation acceleration is The storage compression factor is approximately When C s When it is large enough, the acceleration and compression are both less than 1, and the effect is poor.

[0124] when When , the computational acceleration factor is n, and the storage compression factor is about 32n.

[0125] Similarly, in actual use, the first case is more common, so in theory the compression is actually effective.

[0126] The present invention also provides a neural network compression system based on random projection hashing. Figure 4 As shown, the system includes a processor 42 and a memory 41 storing instructions executable by the processor 42;

[0127] The processor 42 may be a general-purpose processor, such as a central processing unit (CPU), or a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0128] Memory 41 is used to store program code and transmit the program code to the CPU. Memory 41 may include volatile memory, such as random access memory (RAM); non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; or a combination of these types of memory.

[0129] Specifically, an embodiment of the present invention provides a neural network compression system based on random projection hashing, including a processor 42 and a memory 41; the memory 41 stores a computer program that can be run on the processor 42. When the computer program is executed by the processor 42, the following steps are implemented:

[0130] In the forward propagation, the input feature map and weight matrix of each neural network layer are compressed through the projection matrix to calculate the output feature map;

[0131] In the backward propagation, the loss function of the neural network is calculated based on the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated through the loss function;

[0132] Update the weight matrix according to the gradient value of the weight matrix of each layer.

[0133] Embodiments of the present invention also provide a computer-readable storage medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one location to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. Alternatively, the ASIC can be located in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0134] The above describes in detail the neural network compression method and system based on random projection hashing provided by the present invention. For those skilled in the art, any obvious modification made to the present invention without departing from the spirit of the present invention will constitute an infringement of the present invention's patent rights and will result in corresponding legal liability.

Claims

1. A neural network compression method based on random projection hashing for image retrieval, which extracts image features from the image to obtain a feature map, characterized by The steps include: In the forward propagation, the input feature map and weight matrix of each neural network layer are compressed by the projection matrix, and the compressed input feature map and weight matrix are multiplied to calculate the output feature map. For the input feature map S k and the weight matrix W k , through the real-valued projection matrix Project the input feature map S k And the weight matrix is ​​converted into a low-dimensional input feature map and a low-dimensional weight matrix The low-dimensional input feature map and a low-dimensional weight matrix Convert to binary hash code and Complete the compression of the input feature map and weight matrix; k is the number of neural network layers, and b is the compression factor; In the backward propagation, the loss function of the neural network is calculated based on the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated through the loss function; Update the weight matrix according to the gradient value of the weight matrix of each layer.

2. The neural network compression method according to claim 1, wherein In the forward propagation, the input feature map and weight matrix of the convolutional layer are compressed through the projection matrix to calculate the output feature map; the steps include: Reorganize the input feature map and weight matrix into a large matrix respectively; Compress the reorganized input feature map and weight matrix through the projection matrix; Multiply the compressed input feature map and the weight matrix to obtain the output feature map; Perform col2im operation on the output feature map to obtain the final output feature map.

3. The neural network compression method according to claim 2, wherein: In the back propagation, the gradient value of the input feature map and weight matrix of each layer is calculated by the loss function; when sgn(x) contained in the loss function is replaced by x, the gradient value of the input feature map is for: Gradient value of the weight matrix for: in, is the gradient value of the output feature map; is the weight matrix after hash encoding of real values; is a real-valued projection matrix; is the input feature map after hash encoding.

4. The neural network compression method according to claim 1, wherein In the backward propagation, the gradient value of the input feature map and weight matrix of each layer is calculated by the loss function; when the sgn(x) contained in the loss function is replaced by Htanh(x), the gradient value of the input feature map is for: Gradient value of the weight matrix for: in, is the gradient value of the output feature map; is the weight matrix after hash encoding of real values; is a real-valued projection matrix; is the input feature map after hash encoding; Represents the gradient of the Hardtanh function.

5. The neural network compression method according to claim 1, wherein: Update the weight matrix according to the gradient value of the weight matrix of each layer, The matrix is ​​truncated using the clip function. Transform it into a new matrix where each element is in the range of [-1, 1], and assign the new matrix to the weight matrix W of the kth layer k ; Where η is the learning rate; is the gradient value of the weight matrix; W k is the weight matrix.

6. The neural network compression method according to claim 1, wherein When a bias matrix exists when data is input into the neural network, the following steps are also included: The gradient value of the bias matrix of each layer is calculated according to the loss function, and the bias matrix is ​​updated according to the gradient value of the bias matrix of each layer.

7. A neural network compression system based on random projection hashing for image retrieval, which obtains a feature map from the image features extracted from the image, characterized in that The system comprises a processor and a memory; the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the following steps are implemented: In the forward propagation, the input feature map and weight matrix of each neural network layer are compressed by the projection matrix, and the compressed input feature map and weight matrix are multiplied to calculate the output feature map. For the input feature map S k and the weight matrix W k , through the real-valued projection matrix Project the input feature map S k And the weight matrix is ​​converted into a low-dimensional input feature map and a low-dimensional weight matrix The low-dimensional input feature map and a low-dimensional weight matrix Convert to binary hash code and Complete the compression of the input feature map and weight matrix; k is the number of neural network layers, and b is the compression factor; In the backward propagation, the loss function of the neural network is calculated based on the output feature map, and the gradient value of the input feature map and weight matrix of each layer is calculated through the loss function; Update the weight matrix according to the gradient value of the weight matrix of the first layer.

Images