Image classification method based on weight distribution adjustment of binarized neural network

Abstract

The present application relates to a kind of image classification methods based on weight distribution adjustment binary neural network, belong to computer vision technical field, including the following steps: S1: define the binary neural network for image classification;S2: in the binary neural network, configure the additional loss of activation dead weight, and train the approximate function with " convex " derivative of perception, obtain image classification model;S3: data set is collected and initialized, constructs training set and test set;S4: using training set to the image classification model is trained, using test set is tested;S5: using the image classification model of training completion carries out image classification.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to an image classification method based on a weight distribution-adjusted binary neural network. Background Technology

[0002] Computer vision is an interdisciplinary field involving computer systems' understanding and processing of images and videos, aiming to enable computers to mimic and interpret the functions of the human visual system. This includes a range of tasks such as image analysis, object detection, image segmentation, and image classification. Image classification is a core task in computer vision. It involves assigning images to different categories or labels based on their visual content. The key to this task is enabling computers to automatically learn to recognize images of different categories, rather than relying on manually designed rules. In image classification tasks, the application of deep neural networks (DNNs) has had a revolutionary impact. Among them, convolutional neural networks (CNNs), a key architecture within deep neural networks, progressively extract features from images through convolutional layers, pooling layers, and fully connected layers. This allows computers to automatically extract high-level and abstract features from images, improving classification accuracy and automation efficiency. Convolutional neural networks have achieved remarkable results in multiple fields, such as medical diagnosis, autonomous driving, security monitoring, and image search.

[0003] However, in environments with limited computing and storage resources, such as mobile phones, embedded systems, and IoT devices, convolutional neural networks (CNNs) are difficult to deploy due to their large number of parameters and high computational cost. Therefore, some lightweight methods have been proposed to eliminate redundancy in the network and compress it to maintain sufficient performance while reducing the number of parameters and computational cost. Network quantization methods are among these. Binarized neural networks are an extreme version of network quantization methods. By representing full-precision weights and activations in the network as 1 bit, the storage complexity of the model is greatly reduced. Simultaneously, thanks to the advantage that convolutional operations for binarizing weights and activations can be performed using Xnor and Pop-count, the inference speed of the model is greatly accelerated.

[0004] However, due to the 1-bit representation of weights and activations, the representational power of binarized networks is significantly insufficient, resulting in a marked performance degradation. In terms of accuracy, binarized neural networks lag far behind their corresponding full-precision networks. Furthermore, due to the discrete nature of their quantization functions, the classic backpropagation algorithm is difficult to directly apply to the training of binarized neural networks. The difficulty in training the network is another important reason for its reduced performance.

[0005] Many studies have focused on addressing the performance degradation of binary neural networks by improving training methods, further exploring training strategies to enhance network performance. Among these, in 2018, Bi-real Net proposed a weight-aware training method to improve the performance of binary networks; in 2020, IR-Net proposed a standard balancing operation to increase the information entropy of weights, and an EDE strategy to balance the contradiction between the information content and mismatch of weight pseudo-gradients during training, greatly improving the problem of insufficient training of binary network weights; in 2021, ReCU focused on solving the problem of "dead weights" that do not actively flip signs, designing the Recified Clamp Unit to eliminate dead weights, reduce quantization error, and further improve the performance of binary networks by using b... * Standardization is used to further improve the information entropy of the weights before quantization; however, the RecifiedClamp Unit designed by ReCU can lead to weight saturation. These methods improve the performance of binary neural networks by refining the training process. Besides the "dead weights" defined in ReCU, in binary neural networks, those weights with small amplitudes in the pre-quantized real-valued weight distribution are called sensitive weights. Due to their small amplitudes and sensitivity to gradients, they are prone to flipping, disrupting training stability. Dead weights and sensitive weights are not well addressed in binary networks. Both types of weights hinder the training of binary networks and reduce their performance. Dead weights and sensitive weights are as follows... Figure 1 As shown. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a new additional loss for activating dead weights and a novel training-aware approximation function with a “convex” derivative to solve the problem of dead weights and sensitive weights in the training process of binary neural networks, and to improve the performance of binary networks in image classification tasks.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An image classification method based on a weight distribution-adjusted binary neural network includes the following steps:

[0009] S1: Define a binary neural network for image classification;

[0010] S2: Configure an additional loss with dead weights in the binary neural network, and train a perceptual approximation function with a "convex" derivative to obtain an image classification model;

[0011] S3: Collect and initialize the dataset, and build the training and test sets;

[0012] S4: Train the image classification model using the training set and test it using the test set;

[0013] S5: Use the trained image classification model to classify images.

[0014] Furthermore, the additional loss for configuring dead weights in the binary neural network as described in step S2 specifically includes the following steps:

[0015] S211: Consider that the unquantized real-valued weights in a single convolutional layer are in vector form w = (ω1, ω2, ..., ω3). i ,...ω N ), where N is the number of real weights, ω i If w is the i-th weight in a real weight vector, then the additional loss L for activating dead weights is... D Calculate as follows:

[0016]

[0017] L D Used to provide additional gradient encouragement beyond the limit kμ |ω| The dead weights are moved to near the boundary value to reactivate them, and the maximum value of the weight amplitude in the distribution is constrained to k. |ω| Nearby; where |w| represents the new vector (|ω0|, |ω1|, ..., |ω) obtained by taking the absolute value of all elements in w. i |,...,|ω N |); For an indicator function, |w| is defined if and only if the derivative element |ω| ≥ kμ. |ω| The value is 1 if the condition is met, and 0 otherwise; kμ |ω| The set limit; k is the scaling factor, k > 1; μ |ω| μ is the average of the absolute values ​​of all elements in w. |ω| The definition is as follows:

[0018]

[0019] S212: Define an additional loss L for each convolutional layer in the network that activates dead weights independently. D And sum the sum with the task loss; consider a network containing M convolutional layers, For the additional loss of the dead weights of the l-th convolutional layer, L C If the task loss is due to network limitations, then the total loss L is... total for:

[0020]

[0021] Where λ DFor mission loss L C and L D A balancing factor for the additional loss from activating dead weights;

[0022] Define the total loss L total Used for network training, it constrains the weight distribution of all convolutional layers in the network and eliminates dead weights in the distribution.

[0023] Furthermore, the approximate function with a convex derivative trained in step S2 satisfies the following three specifications:

[0024] Standard 1: The derivative g′(ω) of the approximate function g(ω) is convex, and its value decreases as the magnitude of the weight ω, |ω|, increases;

[0025] Rule 2: The steepness of the "convexity" of the derivative g′(ω) of the approximate function g(ω) gradually increases with training;

[0026] Standard 3: The derivative g′(ω) of the approximation function g(ω) provides a sufficiently large update range to cover the additional loss constraints that activate dead weights, forming a compact distribution.

[0027] Furthermore, the "convex" derivative is:

[0028]

[0029] Where μ |ω| Let be the average of the absolute values ​​of all elements in w, and p be an adjustable hyperparameter; e and E be the current round and the total number of rounds, respectively; the updatable interval of g′(ω) is defined by (-pμ). |ω| ,pμ |ω| It is bounded by ) and is related to the average amplitude μ |ω| Proportional;

[0030] Let p be from the initial value p s Exponential decay to the final value p e p s and p e The configuration must conform to Specifications 2 and 3. First, gradually decrease the value of p to balance gradient information content and gradient quality (p). s >p e Secondly, let the updateable interval be larger than the compact distribution formed by the additional loss constraint of the activated dead weights, thus requiring pμ |ω| Greater than kμ |ω| (p s >k,p e >k).

[0031] The beneficial effects of this invention are as follows:

[0032] 1. This invention proposes an additional loss to activate dead weights in the training process of binary neural networks, which truly solves the problem of dead weights in the training process of binary networks.

[0033] 2. This invention proposes a training-aware approximation function with a "convex" derivative to reduce the number of sensitive weights, while meeting the different requirements of gradient information and gradient quality in the early and late stages of the network, thus avoiding the weight saturation problem.

[0034] 3. This invention enables binarized neural networks to perform better in image classification by simply and directly configuring additional losses for activating dead weights and training perceptual approximations with convex derivatives into the binarized network training process, without adding extra structure or computation. As a network performance enhancement technique, it is more direct and convenient to implement.

[0035] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0037] Figure 1 In the diagram, (a) represents dead weights and (b) represents sensitive weights.

[0038] Figure 2 A diagram illustrating the additional loss for activating dead weights and the process for eliminating dead weights;

[0039] Figure 3 Example of a convex derivative g+(ω) and a schematic diagram of a decrease in the number of sensitive weights. Detailed Implementation

[0040] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0041] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0042] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0043] This invention provides an image classification method based on a weight distribution-adjusted binarized neural network. Applying a weight distribution-adjusted binarized neural network in image classification tasks can be divided into the following three steps:

[0044] Step 1: Define a binary neural network for image classification;

[0045] Step 2 involves incorporating two novel weight distribution adjustment techniques proposed in this invention into the traditional training process of a binary neural network. The model is then trained. These two weight distribution adjustment techniques are an additional loss for activating dead weights and a training-aware approximation function with a convex derivative. The definitions and configuration processes of the two techniques are as follows:

[0046] Step 2.1, configure the additional loss for activating dead weights to recall dead weights and reactivate them. This involves two steps: 1. Calculate the additional loss for activating dead weights; 2. Add it to the task loss to obtain the total loss.

[0047] Consider the unquantized real-valued weights in a single convolutional layer as a vector form w = (ω1, ω2, ..., ω3). i ,...ω N ), where N is the number of real weights, ω i If w is the i-th weight in a real weight vector, then the additional loss L for activating dead weights is... D Calculate as follows:

[0048]

[0049] Here, represents the new vector (|ω0|, |ω1|, ..., |ωt) obtained by taking the absolute value of all elements in w. i |,...,|ω N |); For an indicator function, |w| is greater than or equal to kμ if and only if the derivative element |w| in |w| is greater than or equal to kμ. |ω| The value is 1 if the condition is met, and 0 otherwise; kμ |ω| The set limit; k is the scaling factor, which in practice must satisfy k > 1 to prevent the weight distribution from being shrunk too much, resulting in training non-convergence; μ |ω| μ is the average of the absolute values ​​(mean of amplitude) of all elements in w. |ω| The definition is as follows:

[0050]

[0051] L calculated by formula (1) D This will provide additional gradient encouragement for those whose magnitudes exceed the limit kμ. |ω| Dead weights are moved near the boundary value to reactivate them. Since dead weights are eliminated through regularization, weight saturation is avoided. Ultimately, the weight distribution is tightened, and the maximum value of the weight amplitude in the distribution is constrained to k. |ω| Nearby, such as Figure 2 As shown.

[0052] Furthermore, an additional loss L is independently defined for each convolutional layer in the network to activate dead weights. D And sum the results with the task loss. Consider a network containing M convolutional layers. For the additional loss of the dead weights of the l-th convolutional layer, L C Let L be the task loss of the network, specifically the cross-entropy loss for the classification task. The total loss L is... total for:

[0053]

[0054] Where λ D For mission loss L C and L D A balancing factor for the additional loss of activating dead weights.

[0055] Define the total loss L total Used for network training, it can constrain the weight distribution of all convolutional layers in the network, eliminate dead weights in the distribution, and thus improve the performance of the binarized network.

[0056] Step 2.2 In order to reduce the number of sensitive weights, this invention defines a training-aware approximation function with a "convex" derivative and configures it in the binarization quantization layer of the binarization network. The approximation function with a "convex" derivative is used to calculate the gradient during backpropagation.

[0057] The training-aware function, with a convex derivative approximation function g(ω), is primarily used to reduce the number of sensitive weights. Furthermore, the steepness of the derivative of g(ω) is adjusted through training-aware methods to meet the different needs of the network in terms of gradient information and gradient quality at different stages. Finally, weight saturation must be avoided. Therefore, it should meet three specifications as follows:

[0058] (i) The derivative g′(ω) of g(ω) should be “convex”, and its derivative value decreases as the magnitude of the weight ω, |ω|, increases.

[0059] (ii) The steepness of the “convex” shape of the derivative g′(ω) of g(ω) should gradually increase with training.

[0060] (iii) The derivative of g(ω) g′(ω) should provide a sufficiently large update range to cover the additional loss constraints that activate dead weights to form a compact distribution.

[0061] In Specification 1, the convex derivative g′(ω) encourages an increase in the magnitude of sensitive weights, thereby reducing the number of sensitive weights. Specifically, since a larger magnitude |ω| results in a smaller g′(ω), under gradient updates based on constantly changing directions, once the weight magnitude |ω| increases, it is difficult to reduce it again through updates based on new gradients because g′(ω) is relatively small; conversely, once the weight magnitude |ω| decreases, it is easy to increase it again because g′(ω) is relatively large. Therefore, the convex derivative g′(ω) always guides the weights to tend to increase their magnitude. The number of sensitive weights will decrease due to the increase in magnitude. Specification 2 makes the convex derivative flatter in the early stages compared to the later stages, thus providing more gradient information in the early stages, allowing for a wider range of weight updates. In the later stages, the gradient quality is better, and the weight updates are more accurate. In Specification 3, the derivative g′(ω) of g(ω) has a sufficiently large update range that covers the compact distribution formed by the additional loss of activating dead weights, meaning that all weights can be updated with gradients, thereby avoiding the weight saturation problem.

[0062] Below is a feasible example of the derivative g′(ω) that conforms to the above three specifications. Figure 3 Its formula is defined as follows:

[0063]

[0064] Where μ |ω|Let represent the average weight value calculated by formula (2), where p is an adjustable hyperparameter. e and E are the current round and the total number of rounds, respectively. The updatable interval of g′(ω) is defined as (-pμ). |ω| , Pμ |ω| It is bounded by ) and is related to the average amplitude μ |ω| Proportional. In practice, p is changed from the initial value p s Exponential decay to the final value p e p s and p e The configuration must conform to specifications two and three. First, according to the requirements of specification two, gradually decrease the value of p to balance gradient information content and gradient quality (p s >p e Secondly, to satisfy specification three, the updatable interval should be significantly larger than the compact distribution formed by the additional loss constraint of the activated dead weights, thus requiring Pμ |ω| Greater than kμ |ω| (p s >k,p e >k). Note that the decay of p also enhances the steepness of g′(ω), which helps to reduce the number of sensitive weights to a greater extent and achieve more stable training.

[0065] After adding the two weight distribution adjustment methods proposed in this invention in Step 2, training parameters, optimizers, learning rate adjusters, etc. can be configured directly according to the traditional binary neural network training process to obtain a binary neural network with better performance in image classification.

[0066] Step 3: After training, apply the trained binarized network to the image classification task and test its performance.

[0067] The two weight distribution adjustment techniques proposed in this invention can be easily and flexibly configured into the traditional binary neural network training process, solving the problems of dead weights and sensitive weights in the training process, so that the trained binary neural network performs better in image classification.

[0068] Example 1:

[0069] Using this invention, a weight distribution-adjusted binarized neural network based on ResNet20 is trained on the CIFAR-10 dataset. The steps are as follows:

[0070] Step 1: Define the ResNet20 model for training, and replace the full-precision convolutional layers with binary convolutional layers. The feedforward of the quantization layer is the sign function, and the feedback is the derivative of the approximate function. The model parameters can be initialized randomly.

[0071] Step 2: Set up the basic training configuration of the model, including learning rate, optimizer, task loss definition, number of training epochs, and basic parameter configuration (weight decay is set to 5e-4).

[0072] Step 3: Define an additional loss for activating dead weights independently for each binarized convolutional layer according to formula (1), and add it to the total loss according to formula (3). This will allow for the final implementation during training. Figure 2 The effect of constrained distribution is to truly eliminate inactive dead weights; in the hyperparameter part, k in formula (1) is set to 1.48, and λ in formula (3) is set to 1.48. D It is set to 4e-1.

[0073] Step 4: Replace the derivative of the approximation function fed back from the quantization layer with the derivative of the approximation function defined in Equation (4). In the hyperparameter part, (p) in Equation (4) s p e ) is set to (5, 2).

[0074] Step 5: After configuring the additional loss for activating dead weights and the derivative of the new approximation function, you can directly train to obtain a higher-performance ResNet20-based weight distribution-adjusted binarized neural network.

[0075] After training in Step 6, the trained ResNet20-based weight distribution adjustment network is applied to an image classification task, and its performance is tested.

[0076] Example 2:

[0077] Using this invention, a weight distribution-adjusted binarization neural network based on ResNet18 is trained on ImageNet2012. The steps are as follows:

[0078] Step 1: Define the ResNet18 model for training, and replace the full-precision convolutional layers with binary convolutional layers. The feedforward of the quantization layer is the sign function, and the feedback is the derivative of the approximate function. The model parameters can be initialized randomly.

[0079] Step 2: Set up the basic training configuration of the model, including learning rate, optimizer, task loss definition, number of training epochs, and basic parameter configuration (weight decay is set to 1e-4).

[0080] Step 3: Define an additional loss for activating dead weights independently for each binarized convolutional layer according to formula (1), and add it to the total loss according to formula (3). This will allow for the final implementation during training. Figure 2 The effect of constrained distribution is to truly eliminate inactive dead weights; in the hyperparameter part, k in formula (1) is set to 1.48, and λ in formula (3) is set to 1.48.D It is set to 4e-1.

[0081] Step 4: Replace the derivative of the approximation function fed back from the quantization layer with the derivative of the approximation function defined in Equation (4). In the hyperparameter part, (p) in Equation (4) s p e ) is set to (5, 2).

[0082] Step 5: After configuring the additional loss for activating dead weights and the derivative of the new approximation function, you can directly train to obtain a higher-performance ResNet18-based weight distribution-adjusted binarized neural network.

[0083] After training in Step 6, the trained ResNet18-based weight distribution-adjusted binarized neural network is applied to an image classification task, and its performance is tested.

[0084] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it can implement the steps of the method. The storage medium may be, for example, ROM / RAM, magnetic disk, optical disk, etc.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An image classification method based on a weight distribution-adjusted binary neural network, characterized in that: Includes the following steps: S1: Define a binary neural network for image classification; S2: Configure an additional loss with dead weights in the binary neural network, and train a perceptual approximation function with a "convex" derivative to obtain an image classification model; S3: Collect and initialize the dataset, and build the training and test sets; S4: Train the image classification model using the training set and test it using the test set; S5: Perform image classification using the trained image classification model; The additional loss for configuring dead weights in the binary neural network as described in step S2 specifically includes the following steps: S211: Consider that the real-valued weights in a single convolutional layer are in vector form before quantization. Where N is the number of real weights, For the real number weight vector The i-th weight is then subject to the additional loss for activating dead weights. Calculate as follows: Used to provide additional gradient encouragement beyond the limit The dead weights are moved near the boundary value to reactivate them, and the maximum value of the weight amplitude in the distribution is constrained to... Nearby; among them Indicates to The new vector obtained by taking the absolute value of all elements in the vector. ; For an indicator function, if and only if Middle derivative elements The value is 1 if the condition is met, and 0 otherwise. The boundaries set; Scaling factor ; for The average of the absolute values ​​of all elements in the set. The definition is as follows: S212: Define an additional loss for each convolutional layer in the network that activates dead weights independently. And sum the sum with the task loss; consider a network containing M convolutional layers, For the first The additional loss for activating dead weights in each convolutional layer, If the task loss is due to network issues, then the total loss is... for: in For mission losses and A balancing factor for the additional loss from activating dead weights; Defined total loss Used for network training, it constrains the weight distribution of all convolutional layers in the network and eliminates dead weights in the distribution.

2. The image classification method based on weight distribution-adjusted binary neural network according to claim 1, characterized in that: The approximate function with a convex derivative obtained through training in step S2 satisfies the following three criteria: Standard 1: Approximate Function derivative It is convex, and its derivative value increases with the weight. amplitude Increase and decrease; Standard 2: Approximate Functions derivative The steepness of the "convex" shape gradually increases with training; Standard 3: Approximate Functions derivative Provide a sufficiently large update range to cover the additional loss constraints that activate dead weights, forming a compact distribution.

3. The image classification method based on a weight distribution-adjusted binary neural network according to claim 2, characterized in that: The approximate function with a convex derivative, which is trained and perceived in step S2, has the following convex derivative: in for The average of the absolute values ​​of all elements in the set. It is an adjustable hyperparameter; and These represent the current round and the total number of rounds, respectively. The updatable range As a boundary, it is related to the average amplitude. Proportional; Will From the initial value Exponential decay to end value , and The configuration must comply with Specifications 2 and 3, and should be gradually reduced first. The value of is used to balance gradient information content and gradient quality, i.e. > Secondly, let the updateable interval be larger than the compact distribution formed by the additional loss constraint of the activated dead weights, thus requiring Greater than ,Right now > , > .

Images