An image processor

Abstract

The present invention relates to neural network technology, and in particular to an iterative retraining method based on layer pruning sensitivity and an image processor. The method comprises the following steps: initializing an initial threshold and a threshold increment; calculating a scaling factor according to the sum of the absolute values of a weight matrix and the depth of the weight matrix; updating the initial threshold and the threshold increment using the scaling factor, and obtaining the input and output of the current weight matrix before retraining as training data; calculating a pruning threshold according to the initial threshold, the threshold increment, and a precision factor; pruning the weight matrix according to the pruning threshold, and retraining the pruned weight matrix using the training data; judging whether an error of the pruned weight matrix is within a set range, and if so, incrementing the pruning precision factor by 1; judging whether other matrices need to be pruned if the precision does not meet the standard, and if so, switching to the pruned weight matrix; and the present invention is faster than existing methods in calculating the sensitivity of weight matrices of each layer.

Description

Technical Field

[0001] The present invention relates to neural network technology, and in particular to an image processor. Background Art

[0002] In pursuit of exceptional performance, the number of parameters in deep learning models is exploding. In 2018, Google proposed the BERT model, which contained 110 million parameters. In 2019, OpenAI released the GPT-2 model, which contained 1.5 billion parameters. Building on this, they developed the GPT-3 model, which now has 175 billion parameters. The globally popular chat app, ChatGPT, is based on the GPT series of models. In 2021, Microsoft and NVIDIA launched the MT-NLG deep learning model, which contains 530 billion parameters. While these large language models are highly effective, they consume an incredible amount of resources. Pruning methods can reduce the number of model parameters, thereby reducing this overhead.

[0003] Pruning algorithms are divided into structural pruning and non-structural pruning. Structural pruning reduces the number of parameters by reducing the number of neurons. Non-structural pruning reduces the number of parameters by reducing the weights in the weight matrix.

[0004] In 2016, Han et al. used the absolute value of weights as the basis for non-structural pruning of the model and restored accuracy through an iterative approach. Although this method can achieve a high pruning rate, iterative retraining to restore accuracy consumes a large amount of time. This is because a unified pruning threshold and threshold increment are used, and the sensitivity of different weight matrices to pruning is not considered. Chinese patent application CN114358257A discloses a method for obtaining the pruning sensitivity of each layer in the target model through pre-training, and then obtaining the pruning rate of each layer of the model. This method obtains sensitivity through an iterative approach and is very time-consuming. This paper proposes an iterative retraining method based on layer pruning sensitivity. This method calculates the pruning sensitivity of a weight matrix by taking the sum of the absolute values of the weight matrix and the depth of the model in which the weight matrix is located. The obtained pruning sensitivity is then used to optimize iterative retraining, thereby reducing the time of iterative retraining. Summary of the Invention

[0005] In order to solve the time-consuming problems of iterative retraining methods and sensitivity calculation, the present invention proposes an iterative retraining method based on layer pruning sensitivity and an image processor. The iterative retraining process of the retraining method specifically includes the following steps:

[0006] 101. Start retraining and initialize the initial threshold and threshold increment;

[0007] 102. Calculate the scaling factor of the current weight matrix according to the sum of the absolute values of the weight matrix to be updated and the depth of the weight matrix in the network, and initialize the pruning precision factor to 0;

[0008] 103. Update the initial threshold and the threshold increment using the scaling factor, and obtain the input and output of the current weight matrix before retraining as training data;

[0009] 104. Calculate the pruning threshold based on the initial threshold and threshold increment obtained in step 103, as well as the precision factor.

[0010] 105. Pruning the weight matrix according to the pruning threshold, and retraining the pruned weight matrix using the training data;

[0011] 106. Determine whether the error of the current weight matrix after pruning is within the set range. If so, increment the pruning precision factor by 1, and then return to step 104.

[0012] 107. If the accuracy does not meet the requirements, determine whether there are other matrices that need to be pruned. If so, switch to the pruned weight matrix and return to step 102.

[0013] Furthermore, the calculation of the pruning threshold includes:

[0014] Pruning threshold = initial threshold + threshold increment × j;

[0015] Where j represents the pruning precision factor.

[0016] Furthermore, the process of obtaining the scaling factor of the weight matrix includes the following steps:

[0017] Calculate the sum of the absolute values of each element in the current weight matrix;

[0018] Get the depth value of the current weight matrix, where the closer the current weight matrix is to the input layer, the smaller the depth value, and the closer it is to the output layer, the larger the depth value;

[0019] Calculate the pruning sensitivity based on the depth value and the sum of the absolute values of each element in the matrix;

[0020] The ratio of the sensitivity of the current weight matrix to the minimum value of the sum of the sensitivity values of all weight matrices of the neural network is used as the value of the scaling factor.

[0021] Furthermore, the sum of the absolute values of each matrix is expressed as:

[0022]

[0023] Among them, sum represents the sum of the absolute values of the matrix; n and m represent the dimensions of the weight matrix.

[0024] Furthermore, the pruning sensitivity is calculated based on the sum of the depth value and the absolute value of the matrix, including:

[0025]

[0026] Among them, st represents the pruning sensitivity of the matrix; sum represents the sum of the absolute values of the matrix; β is the balance factor, 0<β<1; x is the depth of the current weight matrix.

[0027] Furthermore, the calculation of the scaling factor includes:

[0028]

[0029] Where sf is the scaling factor; st i represents the sensitivity of the i-th weight matrix; n represents the number of weight matrices in the neural network.

[0030] Furthermore, the process of updating the initial threshold and the threshold increment using the scaling factor in step 103 includes:

[0031] td=ti×sf

[0032] ad=adi×sf

[0033] Among them, ti is the initial threshold, td is the initial threshold applicable to the current weight matrix after scaling, sf is the scaling factor; adi is the initial threshold increment, and ad is the threshold increment applicable to the current weight matrix after scaling.

[0034] The present invention further provides an image processor, which is obtained by training according to an iterative retraining method based on layer pruning sensitivity and is applied to image processing scenarios, comprising:

[0035] An image processing unit, used for inputting the image to be processed into the pruned neural network;

[0036] The weight matrices of each layer in the retrained neural network based on the iterative retraining method of layer pruning sensitivity perform image processing on the image to be processed to obtain the corresponding processing results. The retrained neural network based on the iterative retraining method of layer pruning sensitivity includes a pruning threshold calculation unit, a pruning unit, and a retraining unit, wherein:

[0037] A pruning threshold calculation unit is used to calculate the pruning sensitivity based on the depth of the current weight matrix to be pruned in the neural network and the absolute value of the weight matrix, and to update the initial threshold and threshold increment for training the current weight matrix using the pruning sensitivity. After the update, the pruning threshold is calculated using the updated threshold;

[0038] Pruning unit, which performs pruning operation on the current layer using the pruning threshold;

[0039] The retraining unit uses the training data to train the pruned neural network.

[0040] Compared with the existing pruning method, the present invention is faster in calculating the sensitivity of the weight matrix of each layer; compared with the traditional iterative retraining method, this method can reduce the retraining time after optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is an overall flow chart of an iterative retraining method based on layer pruning sensitivity of the present invention;

[0042] Figure 2 A schematic diagram of depth values of a weight matrix in an iterative retraining method based on layer pruning sensitivity of the present invention;

[0043] Figure 3 Flowchart of the traditional iterative retraining process;

[0044] Figure 4 Flowchart of the iterative retraining process after optimizing the scaling factor for the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] The present invention proposes an iterative retraining method based on layer pruning sensitivity and an image processor. The iterative retraining process of the retraining method specifically includes the following steps:

[0047] 101. Start retraining and initialize the initial threshold and threshold increment;

[0048] 102. Calculate the scaling factor of the current weight matrix according to the sum of the absolute values of the weight matrix to be updated and the depth of the weight matrix in the network, and initialize the pruning precision factor to 0;

[0049] 103. Update the initial threshold and the threshold increment using the scaling factor, and obtain the input and output of the current weight matrix before retraining as training data;

[0050] 104. Calculate the pruning threshold based on the initial threshold and threshold increment obtained in step 103, as well as the precision factor.

[0051] 105. Pruning the weight matrix according to the pruning threshold, and retraining the pruned weight matrix using the training data;

[0052] 106. Determine whether the error of the current weight matrix after pruning is within the set range. If so, increment the pruning precision factor by 1, and then return to step 104.

[0053] 107. If the accuracy does not meet the requirements, determine whether there are other matrices that need to be pruned. If so, switch to the pruned weight matrix and return to step 102.

[0054] Neural networks are widely used in fields such as computer vision, machine translation, and speech recognition. However, with the development of neural network technology, the design of network structures has become increasingly complex. The weight matrix corresponding to the convolution kernel also contains an increasing number of weights, which increases the computational workload and parameter count of the neural network, making it difficult to deploy neural networks on hardware resources with limited computing power and storage space. Therefore, neural network pruning is necessary.

[0055] However, the neural network pruning method provided by the prior art uses a fixed pruning rate group manually configured based on human experience. However, using a fixed pruning rate group to prune the neural network can only reduce the number of parameters of the pruned neural network, but cannot increase the running speed of the pruned neural network.

[0056] In view of this, this embodiment proposes an iterative retraining method based on layer pruning sensitivity. In this embodiment, a specific embodiment is given by taking a neural network that processes image data as an example. The method of the present invention can also process any other arbitrary data that can be input into a neural network model for processing, such as voice data, text data, etc.

[0057] In this embodiment, a process of an iterative retraining method based on layer pruning sensitivity is provided as follows: Figure 1 As shown, it includes the following five steps:

[0058] Step 1: Calculate the sum of the absolute values of the weight matrix

[0059] In this embodiment, the weight matrix refers to the weight matrix in the neural network. This embodiment takes the n×m dimension weight as an example. The sum of the absolute values of the weight matrix is expressed as:

[0060]

[0061] Among them, sum represents the sum of the absolute values of the matrix; n and m represent the dimensions of the weight matrix.

[0062] Step 2: Get the depth of the weight matrix

[0063] The depth of the weight matrix refers to the position of the weight matrix in the neural network, such as Figure 2 , if there are two layers of weight matrices, namely the weight matrix between the input layer and the hidden layer, and the weight matrix between the hidden layer and the output layer, the depth value of the weight matrix closer to the input layer is smaller, and the depth value of the weight matrix closer to the output layer is larger. For example, Figure 2 The depth value of the weight matrix between the input layer and the hidden layer is set to 1, and the depth value of the weight matrix between the hidden layer and the output layer is set to 2. Those skilled in the art can flexibly set the depth value according to the principle that the depth value of the weight matrix closer to the input layer is smaller and the depth value of the weight matrix closer to the output layer is larger.

[0064] Preferably, on the basis of satisfying the principle that the closer to the input layer, the smaller the depth value of the weight matrix, and the closer to the output layer, the larger the depth value of the weight matrix, the maximum depth value and the minimum depth value are determined, and the depth of the weight matrix is grouped, each group is divided into a dense part and a sparse part, and the difference in depth values of the weight matrices of two adjacent depths in the sparse part is greater than the difference in depth values of the weight matrices of two adjacent depths in the dense part.

[0065] For example, there are M layers of weight matrices, whose depth values are {1, 2, 3, ..., M}. Divide these M (M ≥ 3) layers of weight matrices into several groups (number of groups ≥ 1), each group includes S (S ≥ 3) layers of weight matrices. Taking the group of layers 1 to S as an example, the smallest depth value is 1 and the largest depth value is S. The depth difference between two adjacent layers in the layer is compressed, and the depth difference between the remaining layers is expanded. The compression ratio X is set (the compression ratio is best controlled within 10% to 2%). The depth value after compression and expansion is {1, 1.8, 3}, that is, if the depth difference between two adjacent layers is Y, then after compression and expansion, The depth values of the layer are evenly distributed in Layer depth values are evenly distributed in

[0066] Step 3: Calculate weight matrix pruning sensitivity

[0067] Calculating the pruning sensitivity based on the sum of the depth value and the absolute value of the matrix includes:

[0068]

[0069] Among them, st represents the pruning sensitivity of the matrix; sum represents the sum of the absolute values of the matrix; β is the balance factor, 0<β<1; x is the depth of the current weight matrix.

[0070] Step 4: Calculate the scaling factor

[0071] The scaling factor for iterative retraining is calculated using the pruning sensitivity of the weight matrix of each layer obtained in the third step. The calculation formula is as follows:

[0072]

[0073] Where sf is the scaling factor; st i represents the sensitivity of the i-th weight matrix; n represents the number of weight matrices in the neural network.

[0074] Step 5: Optimize iterative retraining

[0075] The flowchart of traditional iterative retraining is as follows Figure 3 As shown in Figure 3, the initial threshold and threshold increment are fixed for all weight matrices without considering the pruning sensitivity of different weight matrices, resulting in too many iterations and time-consuming retraining.

[0076] The flowchart of the iterative retraining method optimized by scaling factor in this embodiment is as follows: Figure 4 As shown, it is Figure 3 A layer is added on top of the , which gets the initial threshold and threshold increment of the current layer according to the scaling factor. The calculation formula is as follows:

[0077] td=ti×sf

[0078] ad=adi×sf

[0079] Among them, ti is the initial threshold, td is the initial threshold applicable to the current weight matrix after scaling, sf is the scaling factor; adi is the initial threshold increment, and ad is the threshold increment applicable to the current weight matrix after scaling.

[0080] In this embodiment, the scaling factor sf is multiplied by the initial threshold and the threshold increment as the pruning initial threshold and the threshold increment of the current weight matrix, thereby reducing the number of iterations and thus reducing the time of the entire iterative retraining method.

[0081] This embodiment further provides an image processor, which is trained according to an iterative retraining method based on layer pruning sensitivity and is applied to image processing scenarios. The processor includes:

[0082] An image processing unit, used for inputting the image to be processed into the pruned neural network;

[0083] The weight matrices of each layer in the retrained neural network based on the iterative retraining method of layer pruning sensitivity perform image processing on the image to be processed to obtain the corresponding processing results. The retrained neural network based on the iterative retraining method of layer pruning sensitivity includes a pruning threshold calculation unit, a pruning unit, and a retraining unit, wherein:

[0084] A pruning threshold calculation unit is used to calculate the pruning sensitivity based on the depth of the current weight matrix to be pruned in the neural network and the absolute value of the weight matrix, and to update the initial threshold and threshold increment for training the current weight matrix using the pruning sensitivity. After the update, the pruning threshold is calculated using the updated threshold;

[0085] Pruning unit, which performs pruning operation on the current layer using the pruning threshold;

[0086] The retraining unit uses the training data to train the pruned neural network.

[0087] The present invention provides an iterative retraining method based on layer pruning sensitivity, which can be applied not only to image processors but also to neural network models that process other data. By performing retraining after pruning, the model parameter calculation time and retraining time can be reduced.

[0088] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An image processor, characterized in that: include: An image processing unit, used for inputting the image to be processed into the pruned neural network; The weight matrices of each layer in the retrained neural network according to the iterative retraining method based on layer pruning sensitivity perform image processing on the image to be processed to obtain corresponding processing results. The neural network according to the iterative retraining method based on layer pruning sensitivity includes a pruning threshold calculation unit, a pruning unit, and a retraining unit, wherein: the pruning threshold calculation unit is used to calculate the pruning sensitivity based on the depth of the current weight matrix to be pruned in the neural network and the absolute value of the weight matrix, and the pruning sensitivity is used to update the initial threshold and threshold increment for training the current weight matrix, and after the update, the pruning threshold is used to calculate the pruning threshold for each pruning; the pruning unit uses the pruning threshold to perform a pruning operation on the current layer; the retraining unit uses the training data to train the pruned neural network; the process of training the neural network using the iterative retraining method based on layer pruning sensitivity includes:

101. Start retraining and initialize the initial threshold and threshold increment; 102. Calculate the scaling factor of the current weight matrix based on the sum of the absolute values of the weight matrix to be updated and the depth of the weight matrix in the network, and initialize the pruning precision factor to 0. The process of obtaining the scaling factor of the weight matrix includes the following steps: Calculate the sum of the absolute values of each element in the current weight matrix; Get the depth value of the current weight matrix, where the closer the current weight matrix is to the input layer, the smaller the depth value, and the closer it is to the output layer, the larger the depth value; The pruning sensitivity is calculated based on the depth value and the sum of the absolute values of each element in the matrix, that is: Among them, st represents the pruning sensitivity of the matrix; sum represents the sum of the absolute values of the matrix; β is the balance factor, 0<β<1; x is the depth of the current weight matrix; The ratio of the sensitivity of the current weight matrix to the minimum value of the sum of the sensitivity values of all weight matrices of the neural network is used as the value of the scaling factor; 103. Update the initial threshold and the threshold increment using the scaling factor, and obtain the input and output of the current weight matrix before retraining as training data; 104. Calculate the pruning threshold based on the initial threshold and threshold increment obtained in step 103, as well as the precision factor.

105. Pruning the weight matrix according to the pruning threshold, and retraining the pruned weight matrix using the training data; 106. Determine whether the error of the current weight matrix after pruning is within the set range. If so, increment the pruning precision factor by 1, and then return to step 104.

107. If the accuracy does not meet the requirements, determine whether there are other matrices that need to be pruned. If so, switch to the pruned weight matrix and return to step 102.

2. An image processor according to claim 1, characterized in that: The calculation of the pruning threshold includes: Pruning threshold = initial threshold + threshold increment × j; Where j represents the pruning precision factor.

3. The image processor according to claim 1, wherein: The sum of the absolute values of each matrix is expressed as: Among them, sum represents the sum of the absolute values of the matrix; n and m represent the dimensions of the weight matrix.

4. The image processor according to claim 1, wherein: The calculation of the scaling factor involves: Where sf is the scaling factor; st i represents the sensitivity of the i-th weight matrix; n represents the number of weight matrices in the neural network.

5. The image processor according to claim 1, wherein: The process of updating the initial threshold and the threshold increment using the scaling factor in step 103 includes: Among them, ti is the initial threshold, td is the initial threshold applicable to the current weight matrix after scaling, sf is the scaling factor; adi is the initial threshold increment, and ad is the threshold increment applicable to the current weight matrix after scaling.

Images