Neural network compression method, device, equipment and computer-readable storage medium

Abstract

The present invention discloses a neural network compression method, apparatus, device, and computer-readable storage medium. The method comprises: performing a pooling operation on convolution kernels in a neural network to be compressed to obtain pooling values corresponding to the convolution kernels; performing an integer quantization operation on the pooling values to obtain integer values; and pruning the convolution kernels whose corresponding integer values are target values to compress the neural network to be compressed. The present invention reduces the burden on devices running neural networks, thereby enabling edge devices with limited computing power to run neural networks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a neural network compression method, device, equipment and computer-readable storage medium. Background Art

[0002] With the development of neural network technology, neural networks have been applied to various fields to solve many practical problems that are difficult for modern computers to solve. For example, neural networks have been widely used in pattern recognition, intelligent robotics, automatic control, biology, medicine, and economics. However, as the practical problems they solve become more complex, the model structure of neural networks also becomes increasingly complex. A deep neural network may contain millions or even tens of millions of parameters and dozens or even dozens of layers. However, due to the large number of neural network parameters, the computational cost and storage space required are often very high, making it difficult to run neural networks on edge devices with limited computing power, which limits their application. Summary of the Invention

[0003] The main purpose of the present invention is to provide a neural network compression method, device, equipment and computer-readable storage medium, aiming to solve the problem that existing neural networks have many parameters, making it difficult to run neural networks in edge devices with limited computing power.

[0004] To achieve the above object, the present invention provides a neural network compression method, which comprises the following steps:

[0005] Performing a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel;

[0006] Performing an integer quantization operation on the pooled value to obtain an integer value;

[0007] The convolution kernels whose corresponding integer values are target values are cropped to compress the neural network to be compressed.

[0008] Optionally, the step of performing a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel includes:

[0009] An average pooling operation or a maximum pooling operation is performed on the convolution kernel in the compressed neural network according to the size of the convolution kernel to obtain a pooling value corresponding to the convolution kernel.

[0010] Optionally, the target value is a preset first value, and the step of performing an integer quantization operation on the pooled value to obtain an integer value includes:

[0011] Detecting whether the pooling value is greater than zero;

[0012] If the pooling value is less than or equal to zero, the preset first value is used as the integer value corresponding to the pooling value;

[0013] If the pooling value is greater than zero, the preset second value is used as the integer value corresponding to the pooling value.

[0014] Optionally, after the step of trimming the convolution kernels whose integer values are target values in the convolution kernels to compress the neural network to be compressed, the method further includes:

[0015] Obtaining output data of a fully connected layer in the neural network to be compressed, wherein the output data is obtained by inputting a preset data set into the neural network to be compressed and passing through the fully connected layer;

[0016] Performing integer quantization operations on each element of the output data to obtain an integer value;

[0017] The element whose corresponding integer value in each element of the output data is the target value is used as the target element, and the neurons in the fully connected layer associated with the target element are pruned to compress the neural network to be compressed.

[0018] Optionally, before the step of performing a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel, the step further includes:

[0019] Obtaining a baseline neural network for this round of model update, and using a copy of the baseline neural network as the neural network to be compressed;

[0020] After the step of clipping the convolution kernels whose corresponding integer values are the target values in the convolution kernels to compress the neural network to be compressed, the method further includes:

[0021] The compressed neural network to be compressed is used as a compression network, and a preset data set is used to update the model parameters of the compression network to obtain an updated compression network;

[0022] The model parameters in the updated compression network are used to update the model parameters in the reference neural network to complete this round of model updating.

[0023] Optionally, the method is applied to a participant in federated learning, and before the step of performing a pooling operation on a convolution kernel in a compressed neural network to obtain a pooling value corresponding to the convolution kernel, further includes:

[0024] Obtaining a baseline neural network for this round of federated learning model updates, where the baseline neural network includes a private network and a public network;

[0025] Copying the reference neural network to obtain a network copy, and using the private network in the network copy as the neural network to be compressed;

[0026] After the step of clipping the convolution kernels whose corresponding integer values are the target values in the convolution kernels to compress the neural network to be compressed, the method further includes:

[0027] The network copy containing the compressed neural network to be compressed is used as the compressed network, and the compressed network is used to participate in the current round of federated learning model update to obtain an updated compressed network;

[0028] The model parameters in the updated compression network are used to update the model parameters in the benchmark neural network to complete this round of federated learning model update.

[0029] Optionally, the step of adopting the compression network to participate in this round of federated learning model update to obtain an updated compression network includes:

[0030] Using a preset data set to update the model parameters of the compression network, obtaining a first gradient corresponding to the private network and a second gradient corresponding to the public network in the compression network;

[0031] Sending the second gradient to the coordinator participating in the federated learning, so that the coordinator can obtain a global gradient after globally fusing the second gradients sent by each participant and return it;

[0032] The public network in the compressed network is updated by using the global gradient received from the coordinator, and the private network in the compressed network is updated by using the first gradient, so as to update the compressed network.

[0033] Optionally, the preset data set is an image data set, and after the step of using the model parameters in the updated compression network to update the model parameters in the benchmark neural network to complete the current round of federated learning model update, the method further includes:

[0034] When it is detected that the updated baseline neural network meets the preset model conditions, the updated baseline neural network is used as the target neural network;

[0035] The image data to be classified is input into the target neural network for classification processing to obtain a classification result of the image data to be classified.

[0036] To achieve the above objectives, the present invention provides a neural network compression device, comprising:

[0037] A pooling module is used to perform a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel;

[0038] a quantization module, configured to perform an integer quantization operation on the pooled value to obtain an integer value;

[0039] A cropping module is used to crop the convolution kernels whose corresponding integer values are target values in the convolution kernels to compress the neural network to be compressed.

[0040] To achieve the above-mentioned objectives, the present invention also provides a neural network compression device, which includes: a memory, a processor, and a neural network compression program stored in the memory and executable on the processor. When the neural network compression program is executed by the processor, the steps of the neural network compression method described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer-readable storage medium, on which a neural network compression program is stored. When the neural network compression program is executed by a processor, the steps of the neural network compression method described above are implemented.

[0042] In the present invention, a pooling operation is performed on the convolution kernel in the neural network to be compressed to obtain a pooling value corresponding to the convolution kernel; an integer quantization operation is performed on the pooling value to obtain an integer value corresponding to the convolution kernel; and the convolution kernel whose corresponding integer value is the target value is clipped to compress the neural network to be compressed. It can be understood that in the present invention, the compressed neural network reduces some convolution kernels compared to the original neural network to be compressed, that is, reduces some model parameters. Then, during the operation of the neural network, the parameters required to load and calculate the compressed neural network are greatly reduced compared to the original neural network, thereby reducing the demand for device resources such as CPU computing power, battery and memory of the device, reducing the burden on the device when running the neural network, so that edge devices with limited computing power can also run neural networks, expanding the application scope of neural networks in specific fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the hardware operating environment involved in an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of the first embodiment of the neural network compression method of the present invention;

[0045] Figure 3 This is a functional schematic module diagram of a preferred embodiment of the neural network compression device of the present invention.

[0046] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0049] It should be noted that the neural network compression device in the embodiment of the present invention can be a smart phone, a personal computer, a server and other devices, and no specific restrictions are made here.

[0050] like Figure 1 As shown, the neural network compression device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0051] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the neural network compression device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0052] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a neural network compression program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the neural network compression program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used to communicate data with the client; the network interface 1004 is mainly used to establish a communication connection with the server; the processor 1001 can be used to call the neural network compression program stored in the memory 1005 and perform the following operations:

[0053] Performing a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel;

[0054] Performing an integer quantization operation on the pooled value to obtain an integer value;

[0055] The convolution kernels whose corresponding integer values are target values are cropped to compress the neural network to be compressed.

[0056] Furthermore, the step of performing a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel includes:

[0057] An average pooling operation or a maximum pooling operation is performed on the convolution kernel in the compressed neural network according to the size of the convolution kernel to obtain a pooling value corresponding to the convolution kernel.

[0058] Furthermore, the target value is a preset first value, and the step of performing an integer quantization operation on the pooled value to obtain an integer value includes:

[0059] Detecting whether the pooling value is greater than zero;

[0060] If the pooling value is less than or equal to zero, the preset first value is used as the integer value corresponding to the pooling value;

[0061] If the pooling value is greater than zero, the preset second value is used as the integer value corresponding to the pooling value.

[0062] Furthermore, after the step of clipping the convolution kernels whose integer values are the target values in the convolution kernels to compress the neural network to be compressed, the processor 1001 may also be used to call the neural network compression program stored in the memory 1005 and perform the following operations:

[0063] Obtaining output data of a fully connected layer in the neural network to be compressed, wherein the output data is obtained by inputting a preset data set into the neural network to be compressed and passing through the fully connected layer;

[0064] Performing integer quantization operations on each element of the output data to obtain an integer value;

[0065] The element whose corresponding integer value in each element of the output data is the target value is used as the target element, and the neurons in the fully connected layer associated with the target element are pruned to compress the neural network to be compressed.

[0066] Furthermore, before the step of performing a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel, the processor 1001 may also be used to call the neural network compression program stored in the memory 1005 and perform the following operations:

[0067] Obtaining a baseline neural network for this round of model update, and using a copy of the baseline neural network as the neural network to be compressed;

[0068] After the step of clipping the convolution kernels whose corresponding integer values are the target values in the convolution kernels to compress the neural network to be compressed, the method further includes:

[0069] The compressed neural network to be compressed is used as a compression network, and a preset data set is used to update the model parameters of the compression network to obtain an updated compression network;

[0070] The model parameters in the updated compression network are used to update the model parameters in the reference neural network to complete this round of model updating.

[0071] Furthermore, the method is applied to a participant in federated learning, and before the step of performing a pooling operation on a convolution kernel in a compressed neural network to obtain a pooling value corresponding to the convolution kernel, the method further includes:

[0072] Obtaining a baseline neural network for this round of federated learning model updates, where the baseline neural network includes a private network and a public network;

[0073] Copying the reference neural network to obtain a network copy, and using the private network in the network copy as the neural network to be compressed;

[0074] After the step of clipping the convolution kernels whose corresponding integer values are the target values in the convolution kernels to compress the neural network to be compressed, the processor 1001 may also be used to call the neural network compression program stored in the memory 1005 to perform the following operations:

[0075] The network copy containing the compressed neural network to be compressed is used as the compressed network, and the compressed network is used to participate in the current round of federated learning model update to obtain an updated compressed network;

[0076] The model parameters in the updated compression network are used to update the model parameters in the benchmark neural network to complete this round of federated learning model update.

[0077] Furthermore, the step of using the compression network to participate in the current round of federated learning model update to obtain an updated compression network includes:

[0078] Using a preset data set to update the model parameters of the compression network, obtaining a first gradient corresponding to the private network and a second gradient corresponding to the public network in the compression network;

[0079] Sending the second gradient to the coordinator participating in the federated learning, so that the coordinator can obtain a global gradient after globally fusing the second gradients sent by each participant and return it;

[0080] The public network in the compressed network is updated by using the global gradient received from the coordinator, and the private network in the compressed network is updated by using the first gradient, so as to update the compressed network.

[0081] Furthermore, the preset data set is an image data set. After the step of using the model parameters in the updated compression network to update the model parameters in the baseline neural network to complete the current round of federated learning model update, the processor 1001 may also be used to call the neural network compression program stored in the memory 1005 to perform the following operations:

[0082] When it is detected that the updated baseline neural network meets the preset model conditions, the updated baseline neural network is used as the target neural network;

[0083] The image data to be classified is input into the target neural network for classification processing to obtain a classification result of the image data to be classified.

[0084] Based on the above structure, various embodiments of the neural network compression method are proposed.

[0085] Reference Figure 2 , Figure 2 This is a flowchart of the first embodiment of the neural network compression method of the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than here. The execution subjects of each embodiment of the neural network compression method of the present invention can be devices such as smart phones, personal computers and servers. For ease of description, the execution subjects are omitted in the following embodiments. In this embodiment, the neural network compression method includes:

[0086] Step S10, performing a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel;

[0087] In this embodiment, a neural network that needs to be compressed can be used as a neural network to be compressed. For example, during the training process of a neural network, multiple rounds of model parameter updates are required to continuously optimize and update the model parameters in the neural network. Before each round of model parameter updates, the neural network can be used as a neural network to be compressed.

[0088] The neural network to be compressed includes multiple network layers, some of which are convolutional layers, each of which contains at least one convolution kernel. The sizes of the convolution kernels are different, for example, there are 3*3 convolution kernels and 5*5 convolution kernels. That is, the neural network to be compressed includes multiple convolution kernels. In this embodiment, the neural network to be compressed is compressed by cropping some of the convolution kernels.

[0089] Specifically, a pooling operation can be performed on each convolution kernel in the compressed neural network. Among them, the pooling layer (also called the downsampling layer) is a commonly used network layer in the convolutional neural network. The pooling operation is performed on the feature map output by the previous network layer through the pooling layer to compress the feature map; unlike the existing pooling layer that performs a pooling operation on the feature map, in this embodiment, the convolution kernel of the neural network is pooled using the principle of the existing pooling operation. The object of the pooling operation is not the feature map output by the intermediate network layer of the neural network, but the model parameters of the neural network itself (that is, the parameters in the convolution kernel). The specific pooling operation can adopt any existing pooling operation and is not limited here.

[0090] After performing pooling operations on each convolution kernel, the pooling values corresponding to each convolution kernel are obtained.

[0091] Furthermore, the step S10 includes:

[0092] In step S101, an average pooling operation or a maximum pooling operation is performed on the convolution kernel in the compressed neural network according to the size of the convolution kernel to obtain a pooling value corresponding to the convolution kernel.

[0093] In one embodiment, pooling operations are performed on each convolution kernel in the compressed neural network. Specifically, taking a convolution kernel as an example, an average pooling operation or a maximum pooling operation can be performed according to the size of the convolution kernel to obtain a pooling value for the convolution kernel. Among them, performing a pooling operation according to the size of the convolution kernel means that the size of the convolution kernel, that is, the size of the pooling area corresponding to the pooling operation, for example, if the convolution kernel is 3*3 in size, then the size of the pooling area is also 3*3, that is, a pooling operation is performed on the 9 parameters in the convolution kernel to obtain a pooling value. The maximum pooling operation is to take the maximum value in the pooling area as the pooling value, and the average pooling operation is to take the average value of each value in the pooling area as the pooling value.

[0094] Step S20, performing an integer quantization operation on the pooled value to obtain an integer value;

[0095] After obtaining the pooling values corresponding to each convolution kernel, the pooling values of each convolution kernel are respectively subjected to integer quantization operations to obtain the integer values corresponding to each convolution kernel. Specifically, the integer quantization operation can be performed by pre-setting a quantization cutoff point, and the pooling value greater than the quantization cutoff point is quantized to an integer value, and the pooling value less than or equal to the quantization cutoff point is quantized to another integer value. Alternatively, other integer quantization methods can be used to quantize the pooling value, which can be set specifically as needed and is not limited here.

[0096] Furthermore, the step S20 includes:

[0097] Step S201, detecting whether the pooling value is greater than zero;

[0098] Step S202: If the pooling value is less than or equal to zero, the preset first value is used as the integer value corresponding to the pooling value;

[0099] Step S203: If the pooling value is greater than zero, a preset second value is used as an integer value corresponding to the pooling value.

[0100] In one embodiment, the pooling value may be integer quantized into one of a preset first value and a preset second value, that is, binary quantization is performed, and the preset first value is used as the target value. The preset first value and the preset second value can be set according to specific needs, for example, the first value is set to 0 and the second value is set to 1. The specific method of integer quantizing the pooling value may be: detecting whether the pooling value is greater than zero, that is, setting zero as the quantization dividing point; if the pooling value is greater than zero, then using the second value as the integer value corresponding to the pooling value, that is, using the second value as the integer value of the corresponding convolution kernel; if the pooling value is less than or equal to zero, then using the first value as the integer value corresponding to the pooling value, that is, using the first value as the integer value of the corresponding convolution kernel.

[0101] When the first value is 0 and the second value is 1, the above-mentioned integer quantization process can be implemented by the following process: input the pooling value into the signal loss function max(a, 0), and detect whether the pooling value is greater than zero through the signal loss function, that is, if the pooling value is greater than zero, the pooling value is retained, and if the pooling value is less than 0, the pooling value is changed to 0; the output value of the signal loss function is integerized to 0 or 1, specifically, if the output value is 0, it is integerized to 0, and if the output value is non-zero, it is integerized to 1, so as to realize integer quantization of each pooling value to 0 or 1.

[0102] Step S30: Crop the convolution kernels whose corresponding integer values are the target values in the convolution kernels to compress the neural network to be compressed.

[0103] After integer quantization of each pooling value to obtain the integer value corresponding to each convolution kernel, the integer value corresponding to each convolution kernel may or may not be the target value. The convolution kernels whose corresponding integer values are the target value are pruned to compress the neural network to be compressed. That is, the network portion remaining after pruned is the compressed neural network. Pruning the convolution kernel directly removes the convolution kernel from the neural network to be compressed.

[0104] In this embodiment, a pooling operation is performed on the convolution kernels in the neural network to be compressed to obtain the pooling value corresponding to the convolution kernel; the pooling value is integer quantized to obtain the integer value corresponding to the convolution kernel; and the convolution kernels whose corresponding integer values are target values are clipped to compress the neural network to be compressed. It can be understood that in this embodiment, the compressed neural network reduces some convolution kernels compared to the original neural network to be compressed, that is, reduces some model parameters. Then, during the operation of the neural network, the parameters required to load and calculate the compressed neural network are greatly reduced compared to the original neural network, thereby reducing the demand for device resources such as CPU computing power, battery and memory of the device, reducing the burden on the device when running the neural network, so that edge devices with limited computing power can also run neural networks, expanding the application scope of neural networks in specific fields.

[0105] Furthermore, based on the above-mentioned first embodiment, a second embodiment of the neural network compression method of the present invention is proposed. In this embodiment, the method further includes:

[0106] Step S40, obtaining output data of a fully connected layer in the neural network to be compressed, wherein the output data is obtained by inputting the data set into the neural network to be compressed and passing through the fully connected layer;

[0107] In this embodiment, if the neural network to be compressed includes a fully connected layer, some neurons in the fully connected layer of the neural network to be compressed may also be pruned to further compress the neural network to be compressed. The pruning operation on the fully connected layer may be performed before or after the pruning operation on the convolution kernel, or may be performed simultaneously with the pruning operation on the convolution kernel. That is, in this embodiment, the order of steps S10-S30 and S40-S60 is not limited.

[0108] Specifically, the output data of the fully connected layer in the neural network to be compressed can be obtained. The output data can be obtained by inputting a preset dataset into the neural network to be compressed and passing it through the fully connected layer. The preset dataset can be a pre-collected dataset used to train the neural network to be compressed. For example, when the neural network to be compressed is a model for image classification, the preset dataset can be a dataset consisting of multiple image data. When the neural network before a round of parameter updates is used as the neural network to be compressed, the output data of the fully connected layer obtained by inputting the preset dataset into the neural network during the previous round of parameter updates can be obtained.

[0109] Step S50, performing an integer quantization operation on each element of the output data to obtain an integer value;

[0110] The output data of the fully connected layer is generally in the form of a vector or matrix, including multiple elements. Integer quantization operations can be performed on each element of the output data to obtain an integer value corresponding to each element. The specific process of performing integer quantization operations on each element can be referred to the specific process of performing integer quantization operations on pooled values in the first embodiment, and will not be described in detail here.

[0111] Step S60: taking the element whose corresponding integer value in each element of the output data is the target value as the target element, and pruning the neurons in the fully connected layer associated with the target element to compress the neural network to be compressed.

[0112] The integer values corresponding to each element of the output data may or may not be the target value. The element whose corresponding integer value is the target value can be used as the target element, and the neurons in the fully connected layer associated with the target element can be pruned to compress the neural network to be compressed. When pushing back from the target element to the input layer of the neural network, the neurons in the fully connected layer that it passes through are the neurons associated with the target element. Pruning a neuron directly removes it from the neural network to be compressed.

[0113] In this embodiment, by obtaining the output data of the fully connected layer in the neural network to be compressed, each element of the output data is integer-quantized to obtain the integer value corresponding to each element; the element with an integer value of the target value in each element of the output data is used as the target element, and the neurons in the fully connected layer associated with the target element are pruned to further compress the neural network to be compressed, so that the model parameters of the compressed neural network are further simplified compared to the original neural network to be compressed, and the parameters required to be loaded and calculated during the operation of the neural network are further reduced, which further reduces the demand for device resources such as CPU computing power, battery and memory of the device, and alleviates the burden on the device when running the neural network.

[0114] Furthermore, based on the first and / or second embodiments above, a third embodiment of the neural network compression method of the present invention is proposed. In this embodiment, before step S10, the method further includes:

[0115] Step A10, obtaining a reference neural network for this round of model update, and using a copy of the reference neural network as the neural network to be compressed;

[0116] The neural network model structure can be set according to the specific model training task, and training data corresponding to the model training task can be collected to initialize the model parameters in the neural network. The training data can be used to perform multiple rounds of model updates on the neural network to continuously optimize and update the model parameters in the neural network to obtain a neural network model that meets the prediction accuracy requirements. During each round of model update, the neural network model can be compressed to reduce the burden on the device's computing power, battery, memory, and other resources during the model update process. For example, an image classification task can be set in advance, and images of different categories can be collected as training data. The initialized neural network can be trained using this training data to obtain an image classification model that meets the required image classification accuracy.

[0117] Specifically, taking a round of model updates as an example, we can first obtain the baseline neural network for this round of model updates. In the first round of model updates, the baseline neural network is the initialized neural network. In subsequent model updates, the baseline neural network is the neural network obtained after the previous round of model updates.

[0118] The obtained reference neural network is copied to obtain a copy of the reference neural network, and the copy is used as the neural network to be compressed. The neural network to be compressed is compressed using the neural network compression method described in the above embodiment. The number of parameters of the compressed neural network to be compressed is greatly reduced compared to the reference neural network.

[0119] After step S30, the method further includes:

[0120] Step A20, using the compressed neural network to be compressed as a compressed network, and updating the model parameters of the compressed network using a preset data set to obtain an updated compressed network;

[0121] The compressed neural network to be compressed is referred to as a compressed network, also referred to as a compressed network. A preset data set is used to update the model parameters of the compressed network to obtain an updated compressed network. Specifically, the preset data set is a pre-collected data set used to train a neural network, such as an image data set collected for training an image classification model. The method for updating the model parameters of the compressed network using the preset data set can refer to the supervised training method in existing machine learning models and will not be described in detail here. After the model parameter update, the various model parameters in the compressed network are updated.

[0122] Step A30: Update the model parameters in the reference neural network using the model parameters in the updated compression network to complete this round of model updating.

[0123] The model parameters in the baseline neural network are updated using the model parameters in the updated compressed network to complete this round of model updates. Specifically, if the number of model parameters in the compressed network is smaller than the number of model parameters in the baseline neural network, the model parameters in the corresponding positions in the baseline neural network are updated using the model parameters in the updated compressed network, while the model parameters in other positions in the baseline neural network can remain unchanged. In the next round of model updates, the updated baseline neural network can be used as the baseline neural network for the next round.

[0124] Since the parameters of the compressed neural network are not necessarily the same each time, after multiple rounds of model updates, each model parameter in the initialized neural network is updated in different rounds of model updates, thus ensuring that each model parameter of the neural network is optimized and updated, and guaranteeing the prediction accuracy of the final trained model. Furthermore, each round of model update updates the model parameters of the compressed network, rather than the baseline neural network. This significantly reduces the number of model parameters that the device needs to load and calculate during each round of model update, reducing the burden on the device's computing power, battery, memory, and other resources, allowing edge devices with limited computing power to run neural networks.

[0125] Furthermore, based on the first and / or second embodiments described above, a fourth embodiment of the neural network compression method of the present invention is proposed. In this embodiment, the method is applied to participants in federated learning, and before step S10, the method further includes:

[0126] Step A40: Obtain a baseline neural network for updating the federated learning model in this round, wherein the baseline neural network includes a private network and a public network;

[0127] In this embodiment, each participant in federated learning can locally configure its neural network model structure based on the specific model training task. Each participant's neural network may consist of a private network and a public network. The private network structures of each participant may be identical or different, while the public network model structure of each participant is the same. Model parameters for both the private and public networks can be randomly initialized. Each participant communicates with the coordinator of the federated learning process and completes the federated learning process under the coordination of the coordinator. During the federated learning process, each participant performs multiple rounds of federated learning model updates on its neural network, ultimately obtaining a model that meets the prediction accuracy requirements.

[0128] During a round of model updates, participants can obtain the baseline neural network for this round of federated learning model updates. In the first round of model updates, the baseline neural network is the initialized neural network. In subsequent model updates, the baseline neural network is the neural network obtained after the previous round of model updates.

[0129] Step A50, duplicating the reference neural network to obtain a network copy, and using the private network in the network copy as the neural network to be compressed;

[0130] The participants copy the baseline neural network to obtain a network replica and use the private network in the network replica as the neural network to be compressed. The participants can compress the neural network to be compressed using the same compression method as described in the above-mentioned embodiments. That is, the participants only compress the private network portion of the network replica, while the public network portion remains uncompressed. After the private network in the network replica is compressed, the number of parameters in the network replica is significantly reduced compared to the parameters of the baseline network.

[0131] After step S30, the method further includes:

[0132] Step A60: Using the network copy containing the compressed neural network to be compressed as the compressed network, and using the compressed network to participate in the current round of federated learning model update to obtain an updated compressed network;

[0133] Each participant uses the network copy containing the compressed neural network to be compressed as the compressed network. That is, the network copy with the private network portion compressed is used as the compressed network. The compressed network is then used to update the federated learning model in this round, resulting in an updated compressed network. Specifically, the federated learning model update process can use the existing federated learning model update process.

[0134] Furthermore, the step of using the compressed network to participate in the current round of federated learning model update to obtain an updated compressed network in step A60 includes:

[0135] Step a, using a preset data set to update the model parameters of the compression network to obtain a first gradient corresponding to the private network and a second gradient corresponding to the public network in the compression network;

[0136] The model parameters of the compression network are updated using a preset dataset to obtain a first gradient corresponding to the private network and a second gradient corresponding to the public network in the compression network. Specifically, the preset dataset is a pre-collected dataset used to train a neural network, such as an image dataset collected for training an image classification model. The method for updating the model parameters of the compression network using the preset dataset to obtain the corresponding gradients can be referenced by supervised training methods in existing machine learning models and will not be described in detail here.

[0137] Step b: sending the second gradient to the coordinator participating in the federated learning, so that the coordinator can obtain a global gradient after globally fusing the second gradients sent by each participant and return it;

[0138] Each participant sends the second gradient to the coordinator. The coordinator receives the second gradients sent by each participant and globally integrates them to obtain a global gradient. Specifically, the global integration method can be a weighted average of the second gradients. The coordinator returns the global gradient to each participant.

[0139] Step c: using the global gradient received from the coordinator to update the public network in the compressed network, and using the first gradient to update the private network in the compressed network, so as to update the compressed network;

[0140] Each participant uses the global gradient received from the coordinator to update the public network in the compressed network, thereby updating the model parameters in the public network. Each participant also uses the first gradient to update the private network in the compressed network, thereby updating the model parameters in the private network. After updating both the private and public networks, the compressed network is updated.

[0141] Step A70: Update the model parameters in the baseline neural network using the model parameters in the updated compression network to complete this round of federated learning model update.

[0142] Participants use the model parameters in the updated compressed network to update the model parameters in the baseline neural network to complete this round of federated learning model updates. Specifically, if the number of model parameters in the compressed network is smaller than that in the baseline neural network, the model parameters in the corresponding positions in the baseline neural network are updated using the model parameters in the updated compressed network, while the model parameters in other positions in the baseline neural network can remain unchanged. In the next round of federated learning model updates, the updated baseline neural network can be used as the baseline neural network for the next round.

[0143] In existing horizontal federated learning frameworks, the local model structures of each participant are identical, and each party jointly updates its local model. If each participant compresses its local model, the coordinator will be unable to perform global fusion operations, and thus, federated learning will be impossible, because the compressed local models of each party are different. Therefore, in existing horizontal federated learning, each participant cannot compress the model according to the conditions of their respective devices, resulting in some devices with limited computing power being unable to participate in federated learning. Compared to existing horizontal federated learning, in this embodiment, each participant's local neural network includes a private network and a public network. The private network portion can be different for each participant, and the private network can be compressed according to the conditions of each device, thereby reducing the burden on the device's computing power, battery, memory, and other resources during the federated learning process. Moreover, through the global fusion of the public network, the advantages of federated learning are utilized, thereby improving the prediction accuracy of the final trained model compared to participants training their local neural networks individually.

[0144] Furthermore, in one embodiment, the preset data set is an image data set, and after step A70, the method further includes:

[0145] Step A80, when it is detected that the updated reference neural network meets the preset model conditions, the updated reference neural network is used as the target neural network;

[0146] The model training task may be to train an image classification model, and the preset data set may be pre-collected image data of various categories. After a round of model update, it may be detected whether the updated baseline neural network meets the preset model conditions. The preset model conditions may be set according to specific needs. For example, the classification accuracy of the updated baseline neural network may be less than a certain accuracy, or the loss function of the updated baseline neural network may converge, or the number of training rounds may be greater than a certain number of training rounds, or the training duration may be greater than a certain training duration, etc. When it is detected that the updated baseline neural network meets the preset model conditions, the updated baseline neural network may be used as the target neural network, that is, as the trained neural network model. If it is detected that the updated baseline neural network does not meet the preset model conditions, the next round of model update may be performed.

[0147] Step A90: input the image data to be classified into the target neural network for classification processing to obtain the classification result of the image to be classified.

[0148] After obtaining the target neural network, it can be used to perform image classification and prediction. For example, the target neural network can be deployed in a shopping app, and a function for searching for similar products based on images can be set up. When an uploaded image of an item is detected, the image is input into the target neural network as the image to be classified for classification. The classification result of the image to be classified is obtained, that is, the type of product it belongs to. Based on this classification result, the target neural network can be searched from the product database for identical or similar products to be recommended to the user. In this scenario, the target neural network can be obtained by federated learning with various users' mobile terminal devices. Since users' mobile terminals generally have limited resources such as computing power, battery, and memory, the federated learning method of this embodiment can be used for joint training. This allows mobile terminals with limited resources to participate in the federated learning training, thereby expanding the number of participants in the federated learning and the amount and richness of the training data, making the target neural network more robust and achieving higher prediction accuracy.

[0149] In addition, the embodiment of the present invention also proposes a neural network compression device, referring to Figure 3 , the device comprises:

[0150] A pooling module 10 is used to perform a pooling operation on the convolution kernel in the compressed neural network to obtain a pooling value corresponding to the convolution kernel;

[0151] a quantization module 20, configured to perform an integer quantization operation on the pooled value to obtain an integer value;

[0152] The cropping module 30 is used to crop the convolution kernels whose corresponding integer values are target values in the convolution kernels to compress the neural network to be compressed.

[0153] Furthermore, the pooling module 10 is also used to perform an average pooling operation or a maximum pooling operation on the convolution kernel in the compressed neural network according to the size of the convolution kernel to obtain a pooling value corresponding to the convolution kernel.

[0154] Furthermore, the target value is a preset first value, and the quantization module 20 includes:

[0155] a detection unit, configured to detect whether the pooling value is greater than zero;

[0156] A determination unit, configured to use the preset first value as the integer value corresponding to the pooling value if the pooling value is less than or equal to zero; and to use the preset second value as the integer value corresponding to the pooling value if the pooling value is greater than zero.

[0157] Furthermore, the device further comprises:

[0158] A first acquisition module is configured to acquire output data of a fully connected layer in the neural network to be compressed, wherein the output data is obtained by inputting a preset data set into the neural network to be compressed and then passing through the fully connected layer;

[0159] The quantization module 20 is further configured to perform integer quantization operations on each element of the output data to obtain an integer value;

[0160] The clipping module 30 is further configured to take the element whose corresponding integer value in each element of the output data is the target value as the target element, and clip the neurons in the fully connected layer associated with the target element to compress the neural network to be compressed.

[0161] Furthermore, the device further comprises:

[0162] A second acquisition module is used to obtain a reference neural network for this round of model update, and use a copy of the reference neural network as the neural network to be compressed;

[0163] The first updating module is used to use the compressed neural network to be compressed as the compression network, and use a preset data set to update the model parameters of the compression network to obtain an updated compression network; and use the model parameters in the updated compression network to update the model parameters in the baseline neural network to complete this round of model update.

[0164] Furthermore, the device is deployed on a participant participating in federated learning, and the device further includes:

[0165] A third acquisition module is used to obtain a baseline neural network for updating the federated learning model in this round, wherein the baseline neural network includes a private network and a public network;

[0166] a copy module, configured to copy the reference neural network to obtain a network copy, and use the private network in the network copy as the neural network to be compressed;

[0167] The device further comprises:

[0168] The second update module is used to use the network copy containing the compressed neural network to be compressed as the compressed network, and use the compressed network to participate in the current round of federated learning model update to obtain an updated compressed network; use the model parameters in the updated compressed network to update the model parameters in the baseline neural network to complete this round of federated learning model update.

[0169] Furthermore, the second update module includes:

[0170] A first updating unit is configured to update the model parameters of the compression network using a preset data set to obtain a first gradient corresponding to the private network and a second gradient corresponding to the public network in the compression network;

[0171] a sending unit, configured to send the second gradient to a coordinator participating in federated learning, so that the coordinator globally integrates the second gradients sent by each participant to obtain a global gradient and returns the global gradient;

[0172] a second updating unit, configured to update a public network in the compressed network using the global gradient received from the coordinator, and update a private network in the compressed network using the first gradient, so as to update the compressed network;

[0173] Furthermore, the preset data set is an image data set, and the device further includes:

[0174] a determination module, configured to use the updated baseline neural network as a target neural network when it is detected that the updated baseline neural network meets a preset model condition;

[0175] The classification module is used to input the image data to be classified into the target neural network for classification processing to obtain the classification result of the image data to be classified.

[0176] The expanded content of the specific implementation of the neural network compression device of the present invention is basically the same as the various embodiments of the above-mentioned neural network compression method, and will not be repeated here.

[0177] In addition, an embodiment of the present invention also proposes a computer-readable storage medium, on which a neural network compression program is stored. When the neural network compression program is executed by a processor, the steps of the neural network compression method described below are implemented.

[0178] The various embodiments of the neural network compression device and computer-readable storage medium of the present invention can refer to the various embodiments of the neural network compression method of the present invention, and will not be repeated here.

[0179] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0180] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0181] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0182] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A neural network compression method, characterized in that: The method is applied to a participant in federated learning, and the method includes the following steps: Obtaining a baseline neural network for this round of federated learning model updates, where the baseline neural network includes a private network and a public network; Copying the reference neural network to obtain a network copy, and using the private network in the network copy as the neural network to be compressed; Performing a pooling operation on the model parameters in the convolution kernel in the neural network to be compressed to obtain a pooling value corresponding to the convolution kernel; Performing an integer quantization operation on the pooled value to obtain an integer value; Clipping the convolution kernels whose corresponding integer values are the target values in the convolution kernels to compress the neural network to be compressed; The network copy containing the compressed neural network to be compressed is used as the compressed network, and the compressed network is used to participate in the current round of federated learning model update to obtain an updated compressed network; Using the model parameters in the updated compression network to update the model parameters in the baseline neural network to complete this round of federated learning model update; When it is detected that the updated baseline neural network meets the preset model conditions, the updated baseline neural network is used as the target neural network; Inputting the image data to be classified into the target neural network for classification processing to obtain a classification result of the image data to be classified; The target value is a preset first value, and the step of performing an integer quantization operation on the pooled value to obtain an integer value includes: Detecting whether the pooling value is greater than zero; If the pooling value is less than or equal to zero, the preset first value is used as the integer value corresponding to the pooling value; If the pooling value is greater than zero, the preset second value is used as the integer value corresponding to the pooling value.

2. The neural network compression method according to claim 1, wherein: After the step of clipping the convolution kernels whose integer values are the target values in the convolution kernels to compress the neural network to be compressed, the method further includes: Obtaining output data of a fully connected layer in the neural network to be compressed, wherein the output data is obtained by inputting a preset data set into the neural network to be compressed and passing through the fully connected layer; Performing integer quantization operations on each element of the output data to obtain an integer value; The element whose corresponding integer value in each element of the output data is the target value is used as the target element, and the neurons in the fully connected layer associated with the target element are pruned to compress the neural network to be compressed.

3. The neural network compression method according to any one of claims 1 to 2, characterized in that: Before the step of performing a pooling operation on the model parameters in the convolution kernel in the neural network to be compressed to obtain the pooling value corresponding to the convolution kernel, the method further includes: Obtaining a baseline neural network for this round of model update, and using a copy of the baseline neural network as the neural network to be compressed; After the step of clipping the convolution kernels whose corresponding integer values are the target values in the convolution kernels to compress the neural network to be compressed, the method further includes: The compressed neural network to be compressed is used as a compression network, and a preset data set is used to update the model parameters of the compression network to obtain an updated compression network; The model parameters in the updated compression network are used to update the model parameters in the reference neural network to complete this round of model updating.

4. The neural network compression method according to claim 1, wherein: The step of using the compression network to participate in the current round of federated learning model update to obtain an updated compression network includes: Using a preset data set to update the model parameters of the compression network, obtaining a first gradient corresponding to the private network and a second gradient corresponding to the public network in the compression network; Sending the second gradient to the coordinator participating in the federated learning, so that the coordinator can obtain a global gradient after globally fusing the second gradients sent by each participant and return it; The public network in the compressed network is updated by using the global gradient received from the coordinator, and the private network in the compressed network is updated by using the first gradient, so as to update the compressed network.

5. A neural network compression device, characterized in that: The device is deployed on a participant participating in federated learning, and includes: A third acquisition module is used to obtain a baseline neural network for updating the federated learning model in this round, wherein the baseline neural network includes a private network and a public network; a copy module, configured to copy the reference neural network to obtain a network copy, and use the private network in the network copy as the neural network to be compressed; A pooling module, configured to perform a pooling operation on the model parameters in the convolution kernel in the neural network to be compressed to obtain a pooling value corresponding to the convolution kernel; a quantization module, configured to perform an integer quantization operation on the pooled value to obtain an integer value; A clipping module, configured to clip the convolution kernels whose corresponding integer values are target values in the convolution kernels to compress the neural network to be compressed; A second updating module is configured to use the network copy containing the compressed neural network to be compressed as the compressed network, and adopt the compressed network to participate in the current round of federated learning model update to obtain an updated compressed network; and adopt the model parameters in the updated compressed network to update the model parameters in the baseline neural network to complete the current round of federated learning model update; a determination module, configured to use the updated baseline neural network as a target neural network when it is detected that the updated baseline neural network meets a preset model condition; A classification module is used to input the image data to be classified into the target neural network for classification processing to obtain a classification result of the image data to be classified; The target value is a preset first value, and the quantization module is further used to detect whether the pooling value is greater than zero; if the pooling value is less than or equal to zero, the preset first value is used as the integer value corresponding to the pooling value; if the pooling value is greater than zero, the preset second value is used as the integer value corresponding to the pooling value.

6. A neural network compression device, characterized in that The neural network compression device includes: a memory, a processor, and a neural network compression program stored in the memory and executable on the processor. When the neural network compression program is executed by the processor, the steps of the neural network compression method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a neural network compression program, which, when executed by a processor, implements the steps of the neural network compression method according to any one of claims 1 to 4.

Images