Neural network compression and image classification method and system based on random initialization

Through the neural network compression method based on random initialization, the problem of high computing resources and slow storage in image classification tasks is solved, and efficient image classification is achieved, suitable for devices with resource-constrained.

CN119048799APending Publication Date: 2024-11-29SHANGHAI JIAOTONG UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411009929.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing neural networks have high computing resource requirements and slow storage and deployment inference speed in image classification tasks, especially on resource-constrained devices that affect the efficiency and convenience of applications.

Method used

Using a neural network compression method based on random initialization, the network to be compressed, its parameter initialization method and random number seeds are obtained, and the parameter difference value is obtained by searching for the scaling factor. Perform compression processing to realize compression and decompression of neural networks.

Benefits of technology

It improves the compression rate of neural networks, reduces the computing resource requirements, accelerates processing speed, adapts to edge devices and mobile devices with insufficient storage capabilities, and improves image classification effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048799A_ABST
    Figure CN119048799A_ABST
Patent Text Reader

Abstract

The invention provides a neural network compression and image classification method and system based on random initialization. The method comprises the steps of obtaining a to-be-compressed network, a parameter initialization method of the to-be-compressed network and an initialized random number seed; generating an initialized network parameter by using the random number seed according to the initialization method; searching a scaling factor for each channel of each layer, multiplying the initialized network parameter by the scaling factor to obtain a scaled network parameter, and enabling the quadratic sum of an error between the scaled network parameter and the network parameter to be compressed to be minimum; subtracting the scaled network parameter from the network parameter to be compressed to obtain a parameter difference value; and performing compression processing on the parameter difference value to obtain a compressed parameter difference value. According to the invention, a high compression rate is realized, the storage and transmission overhead of the neural network is reduced, and resources are saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to image processing technology in the field of neural networks and artificial intelligence, and in particular to a neural network compression and image classification method and device based on random initialization. Background Art

[0002] In the past decade, neural networks have achieved good performance in image processing tasks such as image classification, but the increasingly complex neural networks and diverse image classification requirements have made the design of neural networks a cumbersome, inefficient, and computationally resource-intensive task. In actual industrial applications, image classification tasks require a more efficient and convenient solution, such as efficient application to different types of image data (such as natural images, medical images, etc.), and efficient configuration on devices with different computing resources (such as central servers, edge devices, etc.).

[0003] Compared with traditional models, deep neural networks have huge parameters and have advantages such as high representation power, good generalization ability and fast convergence. Despite these benefits, over-parameterization brings challenges such as high computing resource requirements, slow neural network storage and deployment inference speed, especially on resource-constrained devices such as mobile phones or IoT devices.

[0004] Network quantization can alleviate the requirements for storing deep neural networks and the limitations of transmitting network parameters between different devices during the computation process. It maps high-precision floating-point weights to low-bitwidth integers without changing the network architecture. Specifically, post-training quantization methods can be directly applied to trained deep neural networks, selecting appropriate quantization parameters to minimize the difference between network outputs. Existing quantization methods focus on directly quantizing weights using learning steps or mixed precision. However, these methods ignore the redundancy between initialization parameters and optimization parameters during deep neural network training. There are still challenges in applications such as high computing resource requirements and slow neural network storage and deployment inference speed, which in turn affects the efficiency and convenience of image processing tasks such as image classification in applications. Summary of the invention

[0005] In view of the deficiencies in the prior art, the object of the present invention is to provide a neural network compression method and device based on a random number generator.

[0006] According to one aspect of the present invention, there is provided a neural network compression based on random initialization, comprising:

[0007] Obtaining a network to be compressed and a parameter initialization method and a random number seed for initialization, wherein the network to be compressed is composed of multiple network layers connected in parallel or in series, and the network layers contain trained network parameters;

[0008] Use the random number seed to generate initialized network parameters according to the initialization method;

[0009] Searching for a scaling factor for each channel of each layer of the network to be compressed, and multiplying the initialized network parameters by the scaling factor to obtain scaled initialized network parameters, so that the sum of squares of errors between the scaled initialized network parameters and the trained network parameters is minimized;

[0010] Subtract the scaled initialization network parameters from the trained network parameters to obtain parameter differences;

[0011] The parameter difference is compressed to obtain compressed network parameters.

[0012] Preferably, searching for a scaling factor for each channel of each layer of the network to be compressed, multiplying the initialized network parameters by the scaling factor to obtain scaled initialized network parameters, so that the sum of squares of errors between the scaled initialized network parameters and the trained network parameters is minimized, comprises:

[0013] Set the scaling factor to The set q is a positive real number less than 1, Returns the largest integer less than 1 / q;

[0014] For each channel in each layer of the network to be compressed and all possible values ​​of the scaling factor, calculate the scaling factor multiplied by the initialized network parameters to obtain the scaled initialized network parameters;

[0015] For the scaled initialization network parameters, the sum of squares of errors between the scaled initialization network parameters and the trained network parameters, i.e., the squared error, is calculated;

[0016] The value of the minimum square error is obtained from all possible values ​​of the scaling factor as the final scaling factor, and the scaled network parameters corresponding to the final scaling factor are used as the final scaled network parameters.

[0017] Optionally, compressing the parameter difference to obtain the compressed network parameter is achieved by one or both of quantization processing and pruning processing:

[0018] The quantization process refers to mapping the parameter difference represented by a 32-bit floating point number into a 2 n discrete values, where n represents the bit width of the quantized data, and an n-bit integer is used to index the discrete value;

[0019] The pruning process refers to setting a portion of elements with small absolute values ​​in the parameter difference to zero, and only storing non-zero elements in the parameter difference.

[0020] According to a second aspect of the present invention, a neural network decompression method based on random initialization is provided, comprising:

[0021] Get the network initialization method and random number seed to be decompressed, the scaling factor of each channel in each layer, and the parameter difference after compression;

[0022] Use the random number seed to generate initialized network parameters according to the initialization method;

[0023] Decompressing the compressed parameter difference to obtain a full-precision parameter difference;

[0024] Multiplying the initialized network parameters by the scaling factor of the corresponding channel to obtain scaled initial network parameters;

[0025] The full-precision parameter difference is added to the scaled initial network parameter to obtain the decompressed trained network parameter.

[0026] According to a third aspect of the present invention, there is provided a method for an efficient image classification task, comprising:

[0027] Get the image to be classified;

[0028] Providing a trained neural network model for image classification;

[0029] Compressing the neural network model to obtain a scaling factor and a compressed parameter difference;

[0030] storing or transmitting the scaling factor and the parameter difference to a terminal for image classification;

[0031] Obtaining a scaling factor and a parameter difference from the terminal of the image classification, and decompressing them to obtain trained network parameters;

[0032] Based on the neural network model, the image to be classified is classified using the trained network parameters obtained by decompression to obtain a classification result;

[0033] Among them, the neural network model is compressed using the above-mentioned neural network compression method based on random initialization; the scaling factor and parameter difference are obtained from the image classification terminal, and the trained network parameters are decompressed, using the above-mentioned neural network decompression method based on random initialization.

[0034] According to a fourth aspect of the present invention, there is provided an efficient image classification system using a compression network, comprising:

[0035] Image acquisition module: obtain the image to be classified;

[0036] Classification model providing module: providing a trained neural network model for image classification;

[0037] Compression module: compressing the neural network model to obtain a scaling factor and a compressed parameter difference; storing or transmitting the scaling factor and the parameter difference to a terminal for image classification;

[0038] Decompression module: obtains the scaling factor and parameter difference from the terminal of the image classification, and decompresses to obtain the trained network parameters;

[0039] Classification module: Based on the neural network model, the trained network parameters obtained by decompression are used to classify the image to be classified to obtain a classification result;

[0040] Among them, the compression module adopts the above-mentioned neural network compression method based on random initialization; the decompression module adopts the above-mentioned neural network decompression method based on random initialization.

[0041] According to a fifth aspect of the present invention, there is provided a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above methods when executing the program.

[0042] According to a sixth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which implements any of the above methods when executed by a processor.

[0043] According to a seventh aspect of the present invention, there is provided a chip system, comprising a processor, the processor being coupled to a memory, the memory storing program instructions, and any one of the above methods being implemented when the program instructions stored in the memory are executed by the processor.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The neural network compression and decompression method based on random initialization provided by the present invention is a plug-and-play technology. It only needs to obtain a random seed for training to improve the compression rate, and is adaptable to most compression technologies at this stage.

[0046] The neural network compression and decompression method based on random initialization provided by the present invention can solve the problems of high computing resource requirements and slow neural network storage and deployment inference speed in existing neural networks by compressing the neural network, and improve the efficiency and convenience in image processing task applications such as image classification.

[0047] The image classification method and system provided by the present invention are based on randomly initialized neural network compression and decompression, which reduces the requirements for computing resources and accelerates the processing speed. Compared with the existing image classification methods, while ensuring the classification accuracy, the parameter scale of the image classifier is reduced, which is helpful to deploy a neural network-based image classifier on edge devices and mobile devices with insufficient storage capacity, thereby improving the image classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1 This is a flow chart of a neural network compression method based on random initialization according to an embodiment of the present invention;

[0050] Figure 2 This is a flowchart of a neural network decompression method based on random initialization according to an embodiment of the present invention; DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0052] In actual industrial applications, image classification tasks are expected to be efficient and convenient. In existing image classification schemes using neural networks, neural network quantization can be used to alleviate the requirements of storing deep neural networks and the limitations of transmitting network parameters between different devices during the calculation process. Neural network quantization in image classification maps high-precision floating-point weights to low-bit width integers without changing the network architecture. Specifically, post-training quantization methods can be directly applied to trained deep neural networks, and appropriate quantization parameters are selected to minimize the difference between network outputs. Existing quantization methods focus on directly quantizing weights using learning steps or mixed precision. However, it has been found that they ignore the redundancy between initialization parameters and optimization parameters during training. Specifically, the network parameters are obtained by gradient descent based on the initialization parameters, which can be easily recovered using a 32-bit random number seed. Deep neural networks usually have multiple equivalent local minima, and the initialization parameters determine which minimum the neural network converges to. Therefore, there is a correlation between the initialization parameters and the trained network parameters, which is beneficial for network compression, but is not currently applied in existing neural network compression techniques.

[0053] Based on the above research, the present invention provides a neural network compression and image classification method and device based on random initialization, which compresses the neural network by utilizing the correlation between the initialization parameters and the trained network parameters, thereby improving the compression rate. Furthermore, the present invention realizes image classification based on the compressed neural network, thereby reducing the computing resources of image classification in industrial applications and improving the processing speed.

[0054] Reference Figure 1 As shown, a neural network compression method based on a random number generator provided by an embodiment of the present invention includes the following steps S101-105:

[0055] S101, obtaining a network to be compressed and a parameter initialization method and an initialized random number seed;

[0056] In this step, the network to be compressed is composed of multiple network layers connected in parallel or in series, the network layer contains trained network parameters, and the initialization method and random number seed are used when training the network to be compressed.

[0057] Specifically, the network to be compressed is a neural network that is composed of multiple network layers containing parameters connected in parallel or in series to complete a specific function mapping relationship. In practical applications, the output of the network layer containing parameters has multiple channels, and each channel is determined by a subset of the network layer parameters. For example, in some embodiments, W i [j,:] represents the parameter subset of the jth output channel in the i-th network layer. The initialization method refers to sampling a sequence from a random distribution and using the random sequence as a network parameter, which may include but is not limited to initialization based on Gaussian distribution and initialization based on uniform distribution.

[0058] S102, using a random number seed to generate initialized network parameters according to an initialization method;

[0059] In this step, a pseudo-random number generation algorithm may be used to generate the initialized network parameters according to the initialization method. In a specific embodiment, the Philox algorithm may be used to generate all network parameters using the random number seed. Of course, in other embodiments, other pseudo-random number generation algorithms may also be used.

[0060] S103, using a search method to select a scaling factor so that the initialization network parameter multiplied by the scaling factor is closest to the network parameter to be compressed;

[0061] In this step, a scaling factor is searched for each channel of each layer of the compressed network, and the initialized network parameters are multiplied by the scaling factor to obtain scaled initialized network parameters, so that the sum of squares of errors between the scaled initialized network parameters and the trained network parameters is minimized.

[0062] In some embodiments, the following operations may be performed:

[0063] S1031, set the value range of the scaling factor to The set q is a positive real number less than 1, Returns the largest integer less than 1 / q;

[0064] S1032, for each channel in each layer of the network to be compressed and all possible values ​​of the scaling factor, calculate the scaling factor multiplied by the initialized network parameter to obtain the scaled initialized network parameter;

[0065] S1033, for the scaled initialization network parameters, calculating the sum of squares of errors between the scaled initialization network parameters and the trained network parameters, that is, the square error;

[0066] S1034, obtaining a minimum square error value from all possible values ​​of the scaling factor as a final scaling factor, and taking the scaled network parameters corresponding to the final scaling factor as the final scaled network parameters.

[0067] Exemplarily, the operations when executing S103 may include:

[0068] S1031, set the set S of scaling factors to be a sequence of numbers with intervals q between the intervals [0, 1]. For example, in a specific embodiment, q may be 0.015625.

[0069] S1032, for the parameter W of the j-th output channel in the i-th network layer to be compressed i [j,:], traverse each element of the set of scaling factors Calculate the scaling factor and multiply it by the corresponding output dimension parameter in the initialized network Get the set of scaled initialization parameters;

[0070] S1033, for each element in the scaling factor set, calculate the square error between the corresponding output dimension parameter of the network layer to be compressed Select the scaling factor with the smallest square error as the scaling factor of the current channel;

[0071] S1034, iterating the above process, selecting scaling factors for all output dimensions of all network layers, and calculating the scaling factor multiplied by the initial parameter as the scaled initialization parameter.

[0072] Among them, the scaling factor s selected for the j-th output channel in the i-th network layer is k (i, j) satisfies the following relationship:

[0073]

[0074] In the above formula, k is the number of elements in the set of scaling factors.

[0075] S104, subtracting the scaled initialization network parameter from the network parameter to be compressed to obtain a parameter difference;

[0076] In this step, after obtaining the network parameters to be compressed and the scaled initialization parameters, the parameter difference is calculated. For example, the parameter difference is

[0077] S105, compressing the parameter difference to obtain a compressed parameter difference.

[0078] In this step, the parameter difference Δ can be quantized using post-training quantization techniques. i,j to compress.

[0079] Specifically, the difference in parameters after compression Among them, c is the minimum quantization interval, z is the quantization zero point value, and the high-bitwidth floating-point number is converted into a low-bitwidth integer number.

[0080] Of course, in other embodiments, other compression technologies may be used to perform compression processing to obtain compressed parameter differences.

[0081] In another embodiment of the present invention, a neural network decompression method based on a random number generator is provided, which is used to decompress the neural network compressed in the above steps S101-105.

[0082] Specifically, Figure 2 As shown, the neural network decompression method based on the random number generator in this embodiment includes steps S201-S205:

[0083] S201, obtaining an initializer and a random number seed, a scaling factor, and a compressed parameter difference of a network parameter to be decompressed;

[0084] S202, using the random number seed to generate initialized network parameters according to the initialization method;

[0085] S203, decompressing the compressed difference to obtain a full-precision parameter difference;

[0086] S204, multiplying the initialized network parameter by the scaling factor to obtain the scaled initial network parameter;

[0087] S205: Add the full-precision parameter difference to the scaled initial network parameter to obtain a decompressed network parameter.

[0088] In the above S201 and S202, the initialization method and random number seed are the initialization method and random number seed of the neural network compression method in the previous embodiment, and the initialization network parameters are obtained using the same random number generation algorithm as in the neural network compression method in the previous embodiment. No further details will be given here.

[0089] In the above S203, the decompression method corresponds to the inverse operation of the compression process used by the neural network compression method in the previous embodiment. In a specific embodiment, the decompression process is inverse quantization, that is, Among them, c is the minimum quantization interval, z is the quantization zero point value, and both are data used in the quantization process.

[0090] Based on the above-mentioned neural network compression method and decompression method based on random initialization, in another embodiment of the present invention, a method for efficient image classification is provided, comprising:

[0091] M101, obtain the image to be classified;

[0092] M102, providing a trained neural network model for image classification;

[0093] M103, compressing the neural network model to obtain a scaling factor and a compressed parameter difference; wherein the compression is performed using Figure 1 The neural network compression method based on random initialization in steps S101-104 shown;

[0094] M104, storing or transmitting the scaling factor and parameter difference to a terminal for image classification;

[0095] M105, obtaining the scaling factor and parameter difference from the terminal of the image classification, and decompressing them to obtain the trained network parameters; wherein the decompression adopts the above Figure 2 The neural network decompression method based on random initialization described in steps S201-S205 is shown.

[0096] M106, based on the neural network model, using the trained network parameters obtained by decompression, classify the image to be classified to obtain a classification result.

[0097] The above-mentioned neural network model may be an existing trained open source neural network model that can be used for image classification.

[0098] The image classification method of the present invention reduces the requirements for computing resources, speeds up processing, and can be applied to different types of image data (such as natural images, medical images, etc.), and can be efficiently configured in devices with different computing resources (such as central servers, edge devices, etc.). The above-mentioned terminal can be any high-end or non-high-end intelligent terminal such as a computer, a smart phone, a PDA, etc.

[0099] Based on the same technical concept, in another embodiment of the present invention, an efficient image classification system using a compression network is also provided, comprising:

[0100] Image acquisition module: obtain the image to be classified;

[0101] Classification model providing module: providing a trained neural network model for image classification;

[0102] Compression module: compress the neural network model to obtain a scaling factor and a compressed parameter difference; store or transmit the scaling factor and the parameter difference to a terminal for image classification; wherein the compression module adopts Figure 1 The neural network compression method based on random initialization in steps S101-104 shown;

[0103] Decompression module: obtains the scaling factor and parameter difference from the image classification terminal, and decompresses to obtain the trained network parameters; the decompression module adopts the above Figure 2 The neural network decompression method based on random initialization described in steps S201-S205 shown;

[0104] Classification module: Based on the neural network model, the trained network parameters obtained by decompression are used to classify the image to be classified to obtain a classification result.

[0105] Exemplarily, the trained parameters of four mainstream neural network models VGG11, ResNet34, MobilenetV2 and InceptionV3 are obtained from the open source image classification model library Pytorch-cifar, and the compressed parameter difference and scaling factor are obtained according to the neural network compression method based on the random number generator in the above embodiment; the compressed parameter difference and scaling factor are transmitted to a computer running an image classification task; the trained network parameters are obtained from the compressed parameter difference and scaling factor according to the neural network decompression method based on the random number generator in the above embodiment; according to the calculation method of the neural network models VGG11, ResNet34, MobilenetV2 and InceptionV3, image classification is performed on the test images on the CIFAR-10 data set to obtain the classification accuracy.

[0106] Specifically, in some embodiments, an image classification technique is used to classify images mainly containing animals. The CIFAR-10 dataset is a labeled 10-class color image dataset with a picture size of 32×32, including animals such as birds, cats, deer, dogs, frogs, horses, and other common objects such as airplanes, ships, and cars. The training set consists of 50,000 color pictures, and the test set consists of 10,000 color pictures. Table 1 describes the classification accuracy of the proposed compression method at low bit width.

[0107] Table 1

[0108]

[0109] The above experiments show that, on the CIFAR-10 dataset proposed in the embodiment of the present invention, at the same compression rate, both the 10.7 times compression rate (INT3) and the 16 times compression rate (INT2) achieve higher accuracy than the existing neural network compression technology, proving the effectiveness of the present invention in the field of neural network compression.

[0110] Based on the above, an embodiment for image classification tasks on the ImageNet dataset is provided, using the trained neural network image classifiers provided in the pytorch.vision library, namely ResNet18, ResNet50 and MobileNetV2. The ImageNet dataset classifies animal categories in more detail, for example, birds are subdivided into 15 species and dogs are subdivided into 125 species. The compression module obtains the scaling factor and compressed parameter difference of the image classifier, and transmits the scaling factor and compressed parameter difference to the device that performs image classification; the decompression module recovers the trained neural network image classifier parameters from the scaling factor and compressed parameter difference; the classification module classifies 50,000 images in the test set to obtain classification results. Table 2 compares the accuracy results of the proposed method and the prior art. Under almost all network structures and three compressed bit widths, the proposed method is superior to the existing image classification technology.

[0111] Table 2

[0112]

[0113] Based on the same concept as the above embodiment, in another embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored, and when the program is executed by a processor, any of the following methods is implemented: the neural network compression method based on random initialization in the above embodiment, or, the neural network decompression method based on random initialization, or, an efficient image classification method using a compressed network.

[0114] Based on the same concept as the above embodiment, in another embodiment of the present invention, a chip system is also provided, including a processor, a coupling between the processor and a memory, the memory storing program instructions, and when the program instructions stored in the memory are executed by the processor, any one of the following methods is implemented: the neural network compression method based on random initialization in the above embodiment, or, the neural network decompression method based on random initialization, or, an efficient image classification method using a compressed network.

[0115] Based on the same concept as the above embodiment, in another embodiment of the present invention, a chip system is also provided, including a processor, a coupling between the processor and a memory, the memory storing program instructions, and when the program instructions stored in the memory are executed by the processor, any one of the following methods is implemented: the neural network compression method based on random initialization in the above embodiment, or, the neural network decompression method based on random initialization, or, an efficient image classification method using a compressed network.

[0116] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory 62 is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0117] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0118] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0119] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0120] Computer-readable media includes both computer storage media and communication media, including any media that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a general or special purpose computer.

[0121] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A neural network compression method based on random initialization, characterized in that: include: Obtaining a network to be compressed and a parameter initialization method and a random number seed for initialization, wherein the network to be compressed is composed of a plurality of network layers connected in parallel or in series, and the network layers contain trained network parameters; Use the random number seed to generate initialized network parameters according to the initialization method; Searching for a scaling factor for each channel of each layer of the network to be compressed, and multiplying the initialized network parameters by the scaling factor to obtain scaled initialized network parameters, so that the sum of squares of errors between the scaled initialized network parameters and the trained network parameters is minimized; Subtract the scaled initialization network parameters from the trained network parameters to obtain parameter differences; The parameter difference is compressed to obtain compressed network parameters.

2. The neural network compression method based on random initialization according to claim 1, characterized in that: Searching for a scaling factor for each channel of each layer of the network to be compressed, multiplying the initialized network parameters by the scaling factor to obtain scaled initialized network parameters, so that the sum of squares of errors between the scaled initialized network parameters and the trained network parameters is minimized, comprising: Set the scaling factor to The set q is a positive real number less than 1, Returns the largest integer less than 1 / q; For each channel in each layer of the network to be compressed and all possible values ​​of the scaling factor, calculate the scaling factor multiplied by the initialized network parameters to obtain the scaled initialized network parameters; For the scaled initialization network parameters, the sum of squares of errors between the scaled initialization network parameters and the trained network parameters, i.e., the squared error, is calculated; The value of the minimum square error is obtained from all possible values ​​of the scaling factor as the final scaling factor, and the scaled network parameters corresponding to the final scaling factor are used as the final scaled network parameters.

3. The neural network compression method based on random initialization according to claim 1, characterized in that: The compressing the parameter difference to obtain the compressed network parameter is achieved by one or both of quantization processing and pruning processing, wherein: The quantization process refers to mapping the parameter difference represented by a 32-bit floating point number into a 2 n discrete values, where n represents the bit width of the quantized data, and an n-bit integer is used to index the discrete value; The pruning process refers to setting a portion of elements with small absolute values ​​in the parameter difference to zero, and only storing non-zero elements in the parameter difference.

4. The neural network compression method based on random initialization according to claim 1, characterized in that: The random number seed is a 32-bit integer used to generate any number of pseudo-random numbers.

5. A neural network decompression method based on random initialization, characterized in that: include: Get the network initialization method and random number seed to be decompressed, the scaling factor of each channel in each layer, and the parameter difference after compression; Use the random number seed to generate initialized network parameters according to the initialization method; Decompressing the compressed parameter difference to obtain a full-precision parameter difference; Multiplying the initialized network parameters by the scaling factor of the corresponding channel to obtain scaled initial network parameters; The full-precision parameter difference is added to the scaled initial network parameter to obtain the decompressed trained network parameter.

6. An efficient image classification method using a compression network, characterized in that: include: Get the image to be classified; Providing a trained neural network model for image classification; Compressing the neural network model to obtain a scaling factor and a compressed parameter difference; storing or transmitting the scaling factor and the parameter difference to a terminal for image classification; Obtaining a scaling factor and a parameter difference from the terminal of the image classification, and decompressing them to obtain trained network parameters; Based on the neural network model, the image to be classified is classified using the trained network parameters obtained by decompression to obtain a classification result; Wherein, the neural network model is compressed using the neural network compression method based on random initialization described in any one of claims 1 to 4; The scaling factor and parameter difference are obtained from the terminal of the image classification, and the trained network parameters are obtained by decompressing them, which is achieved by the neural network decompression method based on random initialization described in claim 5.

7. An efficient image classification system using a compression network, characterized in that include: Image acquisition module: obtain the image to be classified; Classification model providing module: providing a trained neural network model for image classification; Compression module: compressing the neural network model to obtain a scaling factor and a compressed parameter difference; storing or transmitting the scaling factor and the parameter difference to a terminal for image classification; Decompression module: obtains the scaling factor and parameter difference from the terminal of the image classification, and decompresses to obtain the trained network parameters; Classification module: Based on the neural network model, the trained network parameters obtained by decompression are used to classify the image to be classified to obtain a classification result; Wherein, the compression module adopts the neural network compression method based on random initialization according to any one of claims 1 to 4; The decompression module adopts the neural network decompression method based on random initialization as described in claim 5.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A chip system, comprising a processor, the processor coupled to a memory, the memory storing program instructions, characterized in that: When the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Device, apparatus and method for compressing / decompressing neural network model

    CN108229644A

  • Method and system for determining representative input datasets for post-training quantization of artificial neural networks

    CN116894455A

  • Image quantization processing method, electronic equipment and computer readable storage medium

    CN118014866A

  • Image data compression

    GB202218417D0

  • Neural network compression

    US20190251445A1