A neural network binary quantization method and an image classification method

Abstract

The application discloses a kind of neural network binary quantization method and image classification method, belong to digital image processing technical field.The binary neural network of the application strengthens the ability of shallow feature to retain detail information by detail feature penalty term, and is supplemented with semantic information regular module, while avoiding overfitting, strengthen the effectiveness of advanced semantic information of deep feature, finally reach the effect of enhancing the information retention of input feature map, further improve the performance of quantized network.Simulation results prove that the application has higher accuracy than existing binary neural network-based image classification method, while maintaining faster inference speed and less memory usage;In addition, the binary neural network of the application can clearly observe the main contour of the picture object, retains enough discriminative semantic information, so it can better identify the target subject in the image classification task, thereby further ensuring the accuracy of image classification.

Description

Technical Field

[0001] This invention relates to a neural network binary quantization method and an image classification method, belonging to the field of digital image processing technology. Background Technology

[0002] With the development of science and technology, image classification technology has been applied to more and more fields. Neural networks have powerful computing capabilities and adaptability, and can be well applied to image classification technology.

[0003] Traditional neural networks are often over-parameterized. A full-precision neural network has 32-bit weights and activation parameters. Directly applying these to image classification, even if it achieves high classification accuracy, consumes a large amount of memory, and computational efficiency decreases as the number of network parameters increases. Furthermore, some of the redundant neurons do not improve the accuracy of the classification results. In addition, the application of neural networks in image classification is constrained by the limited computing power, strict latency requirements, and insufficient storage space of embedded devices.

[0004] To address the aforementioned issues, recent research has focused on compressing model size through methods such as pruning, distillation, quantization, low-rank decomposition, compact network design, and parameter sharing to solve deployment problems. The main idea of ​​model quantization is to compress the original network by reducing the number of bits required for network weights and activations. Compared to floating-point models, quantized networks can produce highly compact models, improving computational efficiency, reducing memory and storage usage, and simultaneously reducing energy consumption. However, the core challenge of model quantization is how to control network performance loss within an acceptable range while reducing parameter precision, i.e., finding the optimal balance between compression ratio and accuracy. Binary quantization methods can quantize the weights and activation parameters of a neural network from 32 bits to 1 bit, reducing memory access time during neural network inference and thus accelerating inference to improve the efficiency of image classification methods.

[0005] Patent CN113936169A discloses an image classification method and system based on a binary neural network. It applies binary methods to image classification tasks by inserting a feature amplification layer before binary convolution to increase the number of channels. This reduces computational cost while increasing the representational power of multi-channel binary convolution, fully utilizing the information in the floating-point feature maps. Furthermore, it adds segmented learnable scaling factors between blocks, allowing independent learning of the positive and negative half-axis of each channel. This adjusts the floating-point feature maps transmitted in the network, significantly improving overall network performance with minimal floating-point computation and achieving good image classification results. However, this research does not focus on feature extraction; it only redesigns the network structure and adjusts it using scaling factors, failing to effectively utilize the efficient information in the neural network. This hinders further improvement in classification accuracy.

[0006] Patent CN114358206A discloses a training method and system for binary neural network models, as well as an image processing method and system. It combines binary quantization and knowledge distillation to construct an online knowledge distillation-enhanced binary neural network training framework. Using the online distillation method, it trains the initial real-valued neural network model, the initial auxiliary neural network model, and the initial binary neural network model to obtain the target binary neural network model. This research effectively combines the advantages of quantization and distillation, but it relies heavily on auxiliary training between teacher and student networks to improve accuracy, without specifically extracting network information.

[0007] However, in real-valued networks, different layers represent different information. Shallow features are close to the input layer. Because shallow networks have smaller receptive fields and smaller overlapping areas, this setup ensures the network captures more information, but it also results in a relatively weaker feature extraction capability, only able to extract low-level detailed features. Deep features, on the other hand, are generated for specific tasks and are closer to the network output layer. The number of receptive fields increases, and the overlapping area increases accordingly, compressing image information and facilitating the extraction of overall abstract information from the image.

[0008] Existing neural network binary quantization methods perform poorly in binarizing shallow feature maps, making it very difficult for deep networks to learn high-level semantic information for tasks. Furthermore, deep networks exhibit more pronounced overfitting issues, leading to poor validation and testing results. Some image classification methods based on model compression focus on the final classification result while neglecting the utilization and preservation of shallow network features, whereas deeper features require abstraction and extraction based on shallow features. Consequently, when shallow details are not well preserved, deep semantic information is also difficult to obtain, ultimately hindering the improvement of image classification accuracy. This characteristic is widespread in image classification using binary quantized networks; for example, SD-BNN (Xue P, Lu Y, Chang J, et al. Self-distribution binary neural networks[J]. Applied Intelligence, 2022: 1-13.) is almost unable to extract any shallow binary activation features, further contributing to the decline in classification accuracy.

[0009] Furthermore, for neural networks, activation quantization has a more significant impact on the final performance than weight quantization. To enhance feature mapping, directly using activation functions with stronger fitting capabilities (e.g., Maxout) is highly susceptible to overfitting. Figure 2 As shown, the left image is a random image from CIRAR-10, and the right image is a schematic diagram of shallow binarization activation of the ResNet-18 network using the Adabin method. Figure 2As can be seen, binarization activation completely loses the discriminative information in the original image. This loss of discriminative information leads to low image recognition accuracy, making it difficult to correctly classify the image in subsequent operations. Therefore, it is hoped that the performance of the binary network can be improved by enhancing the deep and shallow features of the network. Summary of the Invention

[0010] To improve the accuracy of image classification methods based on binary neural networks, this invention provides a neural network binary quantization method and an image classification method, the technical solution of which is as follows:

[0011] The first objective of this invention is to provide a binary quantization method for neural networks, applied to image classification, comprising:

[0012] Step 1: Preprocess the image data in the training dataset;

[0013] Step 2: Set the initial parameters and training parameters of the neural network, and solve the problem of non-differentiability of the sign function during backpropagation of the network by using an error decay estimator;

[0014] Step 3: The neural network extracts features from each sample image in the training dataset;

[0015] Step 4: Using the i-th sample image x i ∈R 3×H×W and the shallowest features of the sample Calculate the detailed feature penalty loss;

[0016] Step 5: Analyze the outputs of the first, second, and third layers of the neural network. and Compared with the true label y of the sample i The loss is calculated using the semantic information regularization module;

[0017] Step 6: Calculate the total loss based on the detail feature penalty loss obtained in Step 4 and the semantic information regularization loss obtained in Step 5, and complete the training of the neural network according to the total loss.

[0018] Optionally, the process of calculating the detail feature penalty loss in step 4 includes:

[0019] Step 41: For the i-th sample image x i The image is converted to grayscale to obtain a single-channel grayscale image G, whose width and height are proportional to x. i Consistent;

[0020] Step 42: Adjust the grayscale image G of size 1×H×W to G' of size 1×H0×W0 through the resize operation, and perform L2 normalization on G'.

[0021] Step 43: Reduce the channel dimension of the shallowest feature F0 of the sample to obtain F0' with size 1×H0×W0, and perform standardization processing in the same way;

[0022] Step 44: Calculate the L2 distance of the normalized features obtained in steps 42 and 43 as the detail feature penalty loss L. dfp The expression is:

[0023] L dfp =||normal(G')-normal(F0')||

[0024] The normal() function represents the normalization operation. The actual calculation method is: new data = (original data - minimum value) / (maximum value - minimum value).

[0025] Optionally, step 5 includes:

[0026] Step 51: The input to the semantic information regularization module is the feature maps of m intermediate layers. The semantic information regularization module calculates for each layer of feature maps, specifically for the j-th layer feature map F. j The semantic information of the feature maps from the (j+1)th to the mth layers is used to supervise the jth layer of the current training.

[0027] Step 52: Calculate the fully connected output for the corresponding feature map using the auxiliary classification branch of each layer. The auxiliary classification branch includes, in sequence, a convolutional layer, a global average pooling layer, and a fully connected layer. The feature map F of the j-th layer... j z is obtained through the calculation of the auxiliary classification branch. j ;

[0028] Step 53: Calculate the true label y of the sample using the Label Smooth method. i The provided soft classification loss will determine the true label y of the sample. i Converting to a soft-label probability distribution, its expression is:

[0029]

[0030] Where K represents the number of categories, ε represents any minimum value, and k represents the index position of the probability distribution;

[0031] Step 54: Calculate the soft classification loss, expressed as:

[0032]

[0033] Where, p j For z j The probability distribution of p j,kLet k be the probability value, and its expression is:

[0034]

[0035] The semantic information transfer loss of the j-th layer is expressed as:

[0036]

[0037] Where KL(·) is the Kullback-Leibler divergence function, σ(·) is the softmax function, and τ is the smoothing coefficient;

[0038] Step 55: Obtain the semantic information regularization loss, expressed as:

[0039]

[0040] The hyperparameter β is used to balance the contributions of the soft classification loss and the semantic information transfer loss to the semantic information regularization loss.

[0041] Optionally, the total loss in step 6 is:

[0042] L = L ce +αL dfp +L sir

[0043] Where α is the loss L that balances the detailed feature penalty. dfp and the semantic information regularization loss L sir The weighting coefficients for the two losses.

[0044] Optionally, the preprocessing in step 1 includes: randomly flipping the image horizontally, randomly cropping it, filling the boundary parts with zeros, adjusting the resolution to 32*32 and performing normalization processing.

[0045] A second objective of this invention is to provide a neural network for image classification, trained using the binary quantization method described in any of the preceding claims.

[0046] A third objective of this invention is to provide an image classification method, comprising: inputting an image to be classified into the aforementioned neural network, wherein the neural network for image classification processes the input image and outputs a classification result.

[0047] Optionally, the neural network includes, but is not limited to, VGG networks and ResNet networks.

[0048] A fourth object of the present invention is to provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the binary quantization method of the neural network described in any of the preceding claims and / or the image classification method described in any of the preceding claims.

[0049] A fifth objective of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the binary quantization method of the neural network described in any of the preceding claims and / or the image classification method described in any of the preceding claims.

[0050] The beneficial effects of this invention are:

[0051] During training, the binarized neural network of this invention enhances the ability of shallow features to retain detailed information through a detail feature penalty term, and is supplemented by a semantic information regularization module. This avoids overfitting while enhancing the effectiveness of high-level semantic information of deep features, ultimately improving the information retention of the input feature map and further improving the performance of the quantized network.

[0052] Compared to existing methods that only improve network structure or enhance classification accuracy through other means, this invention utilizes both shallow and deep network features more directly and with a degree of interpretability. It effectively extracts easily overlooked shallow network features and combines them with deep features to improve binarization performance. A detail feature penalty term extracts shallow network information, followed by Maxout activation combined with a semantic information regularization module to extract deep semantic information and mitigate overfitting. Finally, a hybrid loss function further reduces the difference between the network before and after quantization. The detail feature penalty term applies to the shallow network to preserve shallow network details including color, shape, and texture information, and uses the grayscale image of the original image to supervise the binary activation features. The semantic information regularization module uses label information and deeper auxiliary branch output information to supervise the auxiliary branch output information of the current layer, and uses Label Smooth to mitigate overfitting to some extent.

[0053] Simulation results demonstrate that this invention achieves higher accuracy compared to existing image classification methods based on neural network binary quantization. Furthermore, this invention only optimizes the network training process without altering the network's fundamental structure, thus maintaining the complexity of the image processing flow and preserving faster inference speed and lower memory usage. In addition, visualization experiments show that the binary neural network of this invention can clearly observe the main outline of the image object, retaining sufficient discriminative semantic information compared to existing binary neural networks. Therefore, it can better identify the target subject in image classification tasks, further ensuring the accuracy of image classification. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 It is a fully connected network structure with Maxout added.

[0056] Figure 2 This is a CIFAR-10 image and a schematic diagram of its shallow binarization activation in a ResNet-18 network.

[0057] Figure 3 This is an overall framework diagram of the method of the present invention during ResNet-20 network training.

[0058] Figure 4 This is a diagram showing the configuration of the penalty items in the detailed features of the present invention.

[0059] Figure 5 This is a diagram illustrating the semantic information regularization module structure of the present invention.

[0060] Figure 6 This is a comparison chart of the validation accuracy of VGG-Small with different values ​​on the CIFAR-10 dataset.

[0061] Figure 7 These are visualization comparison images; where (a) and (d) are randomly selected images from the TinyImagenet dataset, (b) and (e) are Adabin feature maps, and (c) and (f) are feature maps of this invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0063] The basic knowledge involved in this invention is introduced as follows:

[0064] Maxout function:

[0065] Suppose the input of the l-th layer of the network contains d neurons, and its feature vector is: The formula for calculating each neuron in the Maxout hidden layer is:

[0066]

[0067] Where k is the number of neurons in the Maxout layer. In the formula, z... i,j The calculation formula is:

[0068] z i,j =X T W i,j +b i,j

[0069] Where W∈R d×m×k b∈R m×k d represents the number of input layer nodes, m represents the number of hidden layer nodes, and k represents the number of "hidden hidden layer" nodes corresponding to each hidden layer node. The number of convolutional kernels (input channels) and output channels are also considered. When parameter k is set to 1, the network resembles a typical fully connected network.

[0070] In a fully connected network, layers i through i+1 have only one set of parameters. By introducing Maxout, this layer will train k sets of parameters simultaneously, and then select the largest activation value as the activation value for the next layer's neurons. Figure 1 The network structure after adding Maxout is shown.

[0071] The Maxout function is a learnable piecewise linear function, which is less prone to gradient vanishing and its weights can be continuously learned and changed during training. Any convex function can be approximated by a piecewise linear function, demonstrating Maxout's extremely strong fitting ability, but it is also very susceptible to overfitting.

[0072] Example 1:

[0073] This embodiment provides a binary quantization method for neural networks, applied to image classification, including:

[0074] Step 1: Preprocess the image data in the training dataset;

[0075] Step 2: Set the initial and training parameters of the neural network, and solve the problem of the non-differentiability of the sign function during backpropagation of the network by using an error decay estimator;

[0076] Step 3: The neural network extracts features from each sample image in the training dataset;

[0077] Step 4: Using the i-th sample image x i ∈R 3×H×W and the shallowest features of the sample Calculate the detailed feature penalty loss;

[0078] Step 5: Analyze the outputs of the first, second, and third layers of the neural network. and Compared with the true label y of the sample i The loss is calculated using the semantic information regularization module;

[0079] Step 6: Calculate the total loss based on the detail feature penalty loss obtained in Step 4 and the semantic information regularization loss obtained in Step 5, and complete the training of the neural network according to the total loss.

[0080] Example 2:

[0081] This embodiment provides a binary quantization method for neural networks, such as... Figure 3 The diagram shown illustrates the overall structure of the proposed neural network feature enhancement method during ResNet-20 network training. This embodiment primarily includes the design of the detail feature penalty term and the semantic information regularization module. It is important to emphasize that since this embodiment only adds features during the training phase, the complexity during the testing phase will not increase. The following sections will describe each part in detail:

[0082] Step 1: Read in the image data from the dataset, perform data augmentation on the images, adjust the resolution to 32*32, and normalize them. Specifically, this includes:

[0083] Step 11: Read in the image data from the dataset, and randomly flip, crop, and pad the edges of the images with zeros.

[0084] Step 12: Adjust the resolution to 32*32 and normalize it.

[0085] Step 2: Set the batch size for network training to 128. During backpropagation, the Error Decay Estimator (EDE) proposed by IR-Net is used to address the issue of the non-differentiability of the sign function. This includes the following steps:

[0086] Step 21: Set the batch size to 128, initialize the learning rate to 0.1, use the SGD optimizer, set the momentum to 0.9, and use cosine annealing to adjust the learning rate;

[0087] Step 22: During the backpropagation process, the Error Decay Estimator (EDE) method was used to solve the problem of the non-differentiability of the sign function.

[0088] Step 3: Perform feature extraction for each sample in the dataset, specifically including:

[0089] Step 31: Since the main application of this invention is image classification datasets, each input sample should be an image. For the i-th sample x... i ∈R 3×H×W In other words;

[0090] Step 32: The samples are input into the neural network for convolution calculation. In this embodiment, the shallowest layer features are used. And take the outputs of layer1, layer2, and layer3. and

[0091] Step 33: Subsequent operations are performed using sample input x. i The detailed feature penalty loss is calculated using F0, and F1, F2, and F3 are compared with the true label y of the sample. i The loss is calculated using the semantic information regularization module.

[0092] Step 4: Calculate the detail feature penalty term, which includes:

[0093] Step 41: The structure of the detail feature penalty term is as follows Figure 4 As shown, it is mainly through x i The learning of shallow feature map F0 is supervised by grayscale images;

[0094] Step 42: Enter x i First, grayscale conversion is performed to obtain a single-channel grayscale image G, whose width and height are proportional to x. i Consistent;

[0095] Step 43: In order to supervise the subsequent output features (F0') of F0, in order to make G match the dimensions of the subsequent output features of F0, the grayscale image of G with size 1×H×W is adjusted to G' with size 1×H0×W0 through the resize operation.

[0096] Step 44: Perform L2 normalization on G';

[0097] Step 45: In order to match the dimension of G', a convolutional block is needed to reduce the dimension of the F0 channel;

[0098] Step 46: Obtain F0' with dimensions 1×H0×W0, and perform standardization processing as well;

[0099] Step 47: Calculate the L2 distance between the standardized features of the upper and lower branches as the penalty term loss L. dfp Its expression is:

[0100] L dfp =||normal(G')-normal(F0')||

[0101] Step 48: Note that you need to input x. i The branches no longer need to retain gradients; only the F0 branch convolutional blocks need to retain gradients for training. Normalized blocks can operate normally and the weights can be trained.

[0102] Step 5: Input semantic information regular expression module, specifically including:

[0103] The semantic information regularization module needs to calculate for each layer of feature maps. Figure 5 The calculation process was demonstrated using the j-th layer as an example.

[0104] Step 51: The input is the intermediate feature maps of m layers, using the current training F... j Taking the loss calculation of the j-th layer feature map as an example, since the semantic information contained in the j-th layer feature map is less than that contained in the deeper layer feature maps, the semantic information of the feature maps from the (j+1)-th to the m-th layers is used to supervise the j-th layer currently being trained.

[0105] Step 52: Calculate the fully connected output for the feature map corresponding to each auxiliary classification branch of each layer, such as based on the feature map F of the j-th layer. j z was calculated j ;

[0106] Step 53: Calculate the data label y using the Label Smooth method i The provided soft classification loss will determine the true label y of the sample. i Converting to a soft-label probability distribution, its expression is:

[0107]

[0108] Where K represents the number of categories, ε represents any minimum value, and k represents the index position of the probability distribution.

[0109] Step 54: If the true label of the current sample is y i If the probability value of the k-th position in the soft label probability distribution is at most 1-ε, then it is set to ε / (K-1) otherwise. The soft classification loss can be expressed by the formula:

[0110]

[0111] Where, p j For z j The probability distribution of p can be calculated using the softmax activation function. j,k Let k be the probability value, and its expression is:

[0112]

[0113] The semantic information transfer loss of the j-th layer can be expressed as:

[0114]

[0115] Where KL(·) is the Kullback-Leibler (KL) divergence function, σ(·) is the softmax function, and τ is the smoothing coefficient. The hyperparameter β is used to balance the contributions of the two losses to the semantic information regularization loss.

[0116] Step 55: The total loss of the semantic information regularization module is obtained as follows:

[0117]

[0118] Therefore, the total loss of the method in this embodiment is:

[0119] L = L ce +αL dfp +L sir

[0120] Where α is the equilibrium L dfp and L sir The weighting coefficients for the two losses.

[0121] Step 6: Complete training and test / deploy the model.

[0122] Step 61: Train for 400 epochs using CIFAR-10 and 100 epochs using TinyImageNet, with hyperparameters α = 100, β = 0.01, ε = 0.2, and τ = 3.0, and complete the training as described above;

[0123] Step 62: Deployment complete, put into testing.

[0124] Example 3:

[0125] This example demonstrates the use of a ResNet20 network to identify all 10 categories on the CIFAR10 dataset.

[0126] Step 1: Read in the image data from the dataset, perform data augmentation on the images, adjust the resolution to 32*32, and normalize them. Specifically, this includes:

[0127] Step 11: Read in the image data from the dataset, and randomly flip, crop, and pad the edges of the images with zeros.

[0128] Step 12: Adjust the resolution to 32*32 and normalize it.

[0129] Step 2: Set the batch size for network training to 128. During backpropagation, the Error Decay Estimator (EDE) proposed by IR-Net is used to address the issue of the non-differentiability of the sign function. Specifically, this includes:

[0130] Step 21: Set the batch size to 128, initialize the learning rate to 0.1, use the SGD optimizer, set the momentum to 0.9, and use cosine annealing to adjust the learning rate;

[0131] Step 22: During the backpropagation process, the Error Decay Estimator (EDE) method was used to solve the problem of the non-differentiability of the sign function.

[0132] Step 3: Perform feature extraction for each sample in the dataset, specifically including:

[0133] Step 31: Since the main application of this method is image classification datasets, each input sample should be an image. For the i-th sample x... i ∈R 3×H×W In other words;

[0134] Step 32: The samples are input into the neural network for convolution calculation. This method uses the shallowest layer features. And take the outputs of layer1, layer2, and layer3. and

[0135] Step 33: Subsequent operations are performed using sample input x. i The detailed feature penalty loss is calculated using F0, and F1, F2, and F3 are compared with the true label y of the sample. i The loss is calculated using the semantic information regularization module.

[0136] Step 4: Calculate the detail feature penalty term, which includes:

[0137] Step 41: The structure of the detail feature penalty term is as follows Figure 4 As shown, it is mainly through x i The learning of shallow feature map F0 is supervised by grayscale images;

[0138] Step 42: Enter x i First, grayscale conversion is performed to obtain a single-channel grayscale image G, whose width and height are proportional to x. i Consistent;

[0139] Step 43: In order to supervise the subsequent output features (F0') of F0, in order to make G match the dimensions of the subsequent output features of F0, the grayscale image of G with size 1×H×W is adjusted to G' with size 1×H0×W0 through the resize operation.

[0140] Step 44: Perform L2 normalization on G';

[0141] Step 45: In order to match the dimension of G', a convolutional block is needed to reduce the dimension of the F0 channel;

[0142] Step 46: Obtain F0' with dimensions 1×H0×W0, and perform standardization processing as well;

[0143] Step 47: Calculate the L2 distance between the standardized features of the upper and lower branches as the penalty term loss L. dfp Its expression is:

[0144] L dfp =||normal(G')-normal(F0')||

[0145] Step 48: Note that you need to input x. i The branches no longer need to retain gradients; only the F0 branch convolutional blocks need to retain gradients for training. Normalized blocks can operate normally and the weights can be trained.

[0146] Step 5: Input semantic information regular expression module, specifically including:

[0147] The semantic information regularization module needs to calculate for each layer of feature maps. Figure 5 The calculation process was demonstrated using the j-th layer as an example.

[0148] Step 51: The input is the intermediate feature maps of m layers, using the current training F... j Taking the loss calculation of the j-th layer feature map as an example, since the semantic information contained in the j-th layer feature map is less than that contained in the deeper layer feature maps, the semantic information of the feature maps from the (j+1)-th to the m-th layers is used to supervise the j-th layer currently being trained.

[0149] Step 52: Calculate the fully connected output for the feature map corresponding to each auxiliary classification branch of each layer, such as based on the feature map F of the j-th layer. j z was calculated j ;

[0150] Step 53: Calculate the data label y using the Label Smooth method i The provided soft classification loss will determine the true label y of the sample. i Converting to a soft-label probability distribution, its expression is:

[0151]

[0152] Where K represents the number of categories, ε represents any minimum value, and k represents the index position of the probability distribution.

[0153] Step 54: If the true label of the current sample is y i If the probability value of the k-th position in the soft label probability distribution is at most 1-ε, then it is set to ε / (K-1) otherwise. The soft classification loss can be expressed by the formula:

[0154]

[0155] Where, p j For z j The probability distribution of p can be calculated using the softmax activation function. j,k Let k be the probability value, and its expression is:

[0156]

[0157] The semantic information transfer loss of the j-th layer can be expressed as:

[0158]

[0159] Where KL(·) is the Kullback-Leibler (KL) divergence function, σ(·) is the softmax function, and τ is the smoothing coefficient. The hyperparameter β is used to balance the contributions of the two losses to the semantic information regularization loss.

[0160] Step 55: The total loss of the semantic information regularization module is obtained as follows:

[0161]

[0162] Therefore, the total loss of the improved method is:

[0163] L = L ce +αL dfp +L sir

[0164] Where α is the equilibrium L dfp and L sir The weighting coefficients for the two losses.

[0165] Step 6: Complete training and test / deploy the model, which includes:

[0166] Step 61: Train for 400 epochs using CIFAR-10 and 100 epochs using TinyImageNet, with hyperparameters α = 100, β = 0.01, ε = 0.2, and τ = 3.0, and complete the training as described above;

[0167] Step 62: Deployment complete, put into testing.

[0168] Example 4:

[0169] This embodiment uses the ResNet20 network on the TinyImageNet dataset to identify all 200 categories.

[0170] Step 1: Read in the image data from the dataset, perform data augmentation on the images, adjust the resolution to 32*32, and normalize them. This includes the following steps:

[0171] Step 11: Read in the image data from the dataset, and randomly flip, crop, and pad the edges of the images with zeros.

[0172] Step 12: Adjust the resolution to 32*32 and normalize it.

[0173] Step 2: Set the batch size for network training to 128. During backpropagation, the Error Decay Estimator (EDE) proposed by IR-Net is used to address the issue of the non-differentiability of the sign function. Specifically, this includes:

[0174] Step 21: Set the batch size to 128, initialize the learning rate to 0.1, use the SGD optimizer, set the momentum to 0.9, and use cosine annealing to adjust the learning rate;

[0175] Step 22: During the backpropagation process, the Error Decay Estimator (EDE) method was used to solve the problem of the non-differentiability of the sign function.

[0176] Step 3: Perform feature extraction for each sample in the dataset, specifically including:

[0177] Step 31: Since the main application of this method is image classification datasets, each input sample should be an image. For the i-th sample x... i ∈R 3×H×W In other words;

[0178] Step 32: The samples are input into the neural network for convolution calculation. This method uses the shallowest layer features. And take the outputs of layer1, layer2, and layer3. and

[0179] Step 33: Subsequent operations are performed using sample input x. i The detailed feature penalty loss is calculated using F0, and F1, F2, and F3 are compared with the true label y of the sample. i The loss is calculated using the semantic information regularization module.

[0180] Step 4: Calculate the detail feature penalty term, which includes:

[0181] Step 41: The structure of the detail feature penalty term is as follows Figure 4As shown, it is mainly through x i The learning of shallow feature map F0 is supervised by grayscale images;

[0182] Step 42: Enter x i First, grayscale conversion is performed to obtain a single-channel grayscale image G, whose width and height are proportional to x. i Consistent;

[0183] Step 43: In order to supervise the subsequent output features (F0') of F0, in order to make G match the dimensions of the subsequent output features of F0, the grayscale image of G with size 1×H×W is adjusted to G' with size 1×H0×W0 through the resize operation.

[0184] Step 44: Perform L2 normalization on G';

[0185] Step 45: In order to match the dimension of G', a convolutional block is needed to reduce the dimension of the F0 channel;

[0186] Step 46: Obtain F0' with dimensions 1×H0×W0, and perform standardization processing as well;

[0187] Step 47: Calculate the L2 distance between the standardized features of the upper and lower branches as the penalty loss term. Ldfp， Its expression is:

[0188] L dfp =||normal(G')-normal(F0')||

[0189] Step 48: Note that you need to input x. i The branches no longer need to retain gradients; only the F0 branch convolutional blocks need to retain gradients for training. Normalized blocks can operate normally and the weights can be trained.

[0190] Step 5: Input semantic information regular expression module, specifically including:

[0191] The semantic information regularization module needs to calculate for each layer of feature maps. Figure 5 The calculation process was demonstrated using the j-th layer as an example.

[0192] Step 51: The input is the intermediate feature maps of m layers, using the current training F... j Taking the loss calculation of the j-th layer feature map as an example, since the semantic information contained in the j-th layer feature map is less than that contained in the deeper layer feature maps, the semantic information of the feature maps from the (j+1)-th to the m-th layers is used to supervise the j-th layer currently being trained.

[0193] Step 52: Calculate the fully connected output for the feature map corresponding to each auxiliary classification branch of each layer, such as based on the feature map F of the j-th layer.j z was calculated j ;

[0194] Step 53: Calculate the data label y using the Label Smooth method i The provided soft classification loss will determine the true label y of the sample. i Converting to a soft-label probability distribution, its expression is:

[0195]

[0196] Where K represents the number of categories, ε represents any minimum value, and k represents the index position of the probability distribution.

[0197] Step 54: If the true label of the current sample is y i If the probability value of the k-th position in the soft label probability distribution is at most 1-ε, then it is set to ε / (K-1) otherwise. The soft classification loss can be expressed by the formula:

[0198]

[0199] Where, p j For z j The probability distribution of p can be calculated using the softmax activation function. j,k Let k be the probability value, and its expression is:

[0200]

[0201] The semantic information transfer loss of the j-th layer can be expressed as:

[0202]

[0203] Where KL(·) is the Kullback-Leibler (KL) divergence function, σ(·) is the softmax function, and τ is the smoothing coefficient. The hyperparameter β is used to balance the contributions of the two losses to the semantic information regularization loss.

[0204] Step 55: The total loss of the semantic information regularization module is:

[0205]

[0206] Therefore, the total loss of the improved method is:

[0207] L = L ce +αL dfp +L sir

[0208] Where α is the equilibrium L dfp and L sir The weighting coefficients for the two losses.

[0209] Step 6: Complete training and test / deploy the model, which includes:

[0210] Step 61: Train for 400 epochs using CIFAR-10 and 100 epochs using TinyImageNet, with hyperparameters α = 100, β = 0.01, ε = 0.2, and τ = 3.0, and complete the training as described above;

[0211] Step 62: Deployment complete, put into testing.

[0212] To further illustrate the beneficial effects of the present invention, an experiment was conducted, the process of which is as follows:

[0213] 1. Dataset and Evaluation Criteria

[0214] This invention uses two benchmark classification datasets to evaluate the effectiveness of the proposed method: CIFAR-10 and TinyImageNet. All experiments were implemented using PyTorch and an NVIDIA GTX 1080Ti. For data augmentation, random horizontal flipping, random cropping, and normalization were applied. The input image size for CIFAR-10 was 32×32, and for TinyImageNet it was 64×64. During backpropagation, the Error Decay Estimator (EDE) proposed by IR-Net was used to address the non-differentiability of the sign function. During training, the batch size was set to 128, the initial learning rate was 0.1, the SGD optimizer was used, the momentum was set to 0.9, and cosine annealing was used to adjust the learning rate. Training was performed for 400 epochs on CIFAR-10 and 100 epochs on TinyImageNet, with hyperparameters α = 100, β = 0.01, ε = 0.2, and τ = 3.0.

[0215] 2. Accuracy Comparison

[0216] To verify the effectiveness of the method of the present invention, experiments were conducted on the CIFAR-10 and TinyImageNet datasets, and the results were compared with existing BNN methods.

[0217] Table 1 shows a performance comparison of various methods on CIFAR-10, with * indicating that these methods use a Bi-RealNet architecture. The method proposed in this invention achieves the best accuracy in all cases.

[0218] Table 1. Validation of different novel binary convolutional neural networks on the CIFAR-10 classification dataset.

[0219]

[0220]

[0221] Among them, the method of this invention using VGG-Small achieved an accuracy of 93.1%, surpassing all current state-of-the-art methods, with a difference of only 1% from the full-precision accuracy. It also achieved the highest accuracy in ResNet-20 and ResNet-18, exceeding other mainstream state-of-the-art methods.

[0222] Table 2 shows the performance of different state-of-the-art binarization algorithms using ResNet-18 on TinyImageNet. All experiments were rerun under the same settings, and top-1 accuracy results are presented. Due to the more challenging task, the performance results show significant differences. It can be seen that the proposed method improves upon Adabin by 0.45% and surpasses ReActNet's 11.79%. Experimental results demonstrate that the proposed method outperforms other state-of-the-art methods.

[0223] Table 2. Validation of different novel binary convolutional neural networks using ResNet-18 on the TinyImageNet classification dataset.

[0224]

[0225] 3. Ablation Experiment and Analysis

[0226] Ablation experiments were conducted to further demonstrate the effectiveness of the proposed method and to explain the design rationale behind some of the structures. This ablation study comprises five parts: the impact of different modules on model performance, the sensitivity of hyperparameters α and β to model accuracy, a comparison of algorithm complexity and speed, and visualization analysis.

[0227] (1) The impact of different modules on model performance.

[0228] To further verify the improvement effect of the proposed module, ablation experiments were conducted by adding a detail feature penalty term L to the baseline loss function. dfp With semantic information regularization loss L sir And add L at the same time dfp and L sir Conduct a comparative experiment.

[0229] Table 3 Ablation results of different network structures for different modules on the CIFAR-10 dataset.

[0230]

[0231] As shown in Table 3, only add L dfp Compared to adding only L sir The accuracy is improved compared to the baseline, and L is added.dfp and L sir This can further improve accuracy. Specifically, adding L to ResNet-18 dfp It can improve accuracy by 0.08%, while adding L sir This improved accuracy by 0.25%, while adding L. dfp With L sir The accuracy improved by 0.33%. Therefore, the two proposed modules are effective in improving the performance of the binary quantization method.

[0232] (2) The sensitivity of the size of the hyperparameter α to the model accuracy.

[0233] This ablation experiment used a VGG-Small structure for comparison on CIFAR-10, where α is the equilibrium detail feature penalty term L. dfp and semantic information regularization loss L sir The weighting coefficients for the two losses are shown; a larger α value indicates a higher proportion of shallow features in the total loss. To further demonstrate the impact of α on performance, [the following is used as an example]. Figure 6 Show the specific results.

[0234] like Figure 6 It can be seen that the composition of the loss function can have a certain impact on the accuracy. When the α value is small, the correction of shallow features to the loss function is not obvious, while as the α value gradually increases, L... dfp The regularization effect on the total loss gradually increases, reaching an accuracy of 93.02% at α = 100. Subsequently, as the value of α further increases, L... dfp The regularization effect on the total loss is too strong, resulting in an excessively high cross-entropy loss L. ce and semantic information regularization loss L sir The contribution to the total loss decreases, the model's ability to learn high-level semantic information from deep features diminishes, and the accuracy decreases.

[0235] (3) The sensitivity of the hyperparameter β to the model accuracy.

[0236] This ablation experiment uses the ResNet-18 architecture for comparison on CIFAR-10 and TinyImageNet, as shown in Equation (8), where β is the loss L used to balance semantic information transfer. sit and soft tag loss L smooth The larger the β value, the greater the contribution of the semantic information regularization loss to the soft label loss L. smooth In semantic information regularization total loss L sir The higher the proportion, the better. To further illustrate the impact of β value on performance, specific results are presented in Table 4.

[0237] Table 4 shows the validation accuracy of ResNet-18 with different β values ​​on the CIFAR-10 and TinyImageNet datasets.

[0238]

[0239] Analysis shows that the composition of the loss function can have a certain impact on accuracy. When the β value is small, the soft label loss has a significant impact on the total loss L of semantic information regularization. sir The effect is not obvious, but as the β value gradually increases, the soft tag loss L... smooth The total loss L for semantic information regularization sir The contribution of β gradually increases, and the model performance gradually improves. Deeper features pay more attention to learning soft labels, reducing the influence of noise from deeper features on the current layer features. When β = 0.1, the method achieves an accuracy of 93.38% on CIFAR-10; when β = 0.05, the method achieves an accuracy of 53.51% on TinyImageNet.

[0240] (4) Comparison of complexity and speed.

[0241] The method of this invention introduces only two improved modules during the training process, increasing the training complexity only. The complexity during validation and inference is unaffected by these improved modules, making the proposed method very deployment-friendly for networks. Table 5 shows the complexity of ResNet18 networks using different methods on different datasets.

[0242] Table 5. Comparison of the complexity of ResNet18 networks using different methods on the CIFAR-10 and TinyImageNet datasets.

[0243]

[0244] The complexity of ResNet18 networks using different methods was compared on two datasets. Table 5 shows the parameter count and inference speed results of the compared methods. As can be seen from the results in Table 5, the Adabin method and the method of this invention have the same parameter count and speed.

[0245] To enhance features at both shallow and deep layers, the method of this invention introduces an additional number of parameters, which is equivalent to the number of parameters of IR-Net, which has a relatively small number of parameters in the current field of binary quantization, on the CIFAR-10 and TinyImageNet datasets. At the same time, this method is about 1.51 FPS faster than the BNN-BN method in terms of inference speed.

[0246] On the other hand, compared to Adabin, our method has the same number of parameters and inference speed on both datasets. Therefore, compared to Adabin, our method achieves higher accuracy performance without sacrificing inference speed.

[0247] (5) Visual analysis.

[0248] Figure 7 The visualization effects of the method of this invention and Adabin are illustrated. Eighteen images of different categories were randomly selected from the TinyImageNet dataset and input into the ResNet-18 BNN of both. In Adabin, sufficient discriminative semantic information is almost lost, and the target subject cannot be identified. However, the method of this invention retains useful semantic features due to the feature enhancement from shallow to deep layers, and the main outline of the image object can be clearly observed.

[0249] Some steps in the embodiments of the present invention can be implemented using software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.

[0250] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A binary quantization method for a neural network, wherein the neural network is applied to image classification, characterized in that, The binary quantization method includes: Step 1: Preprocess the image data in the training dataset; Step 2: Set the initial parameters and training parameters of the neural network, and solve the problem of non-differentiability of the sign function during backpropagation of the network by using an error decay estimator; Step 3: The neural network extracts features from each sample image in the training dataset; Step 4: Using the i-th sample image x i ∈R 3×H×W and the shallowest features of the sample Calculate the detailed feature penalty loss; Step 5: Analyze the outputs of the first, second, and third layers of the neural network. and Compared with the true label y of the sample i The loss is calculated using the semantic information regularization module; Step 6: Calculate the total loss based on the detail feature penalty loss obtained in Step 4 and the semantic information regularization loss obtained in Step 5, and complete the training of the neural network according to the total loss; The process of calculating the detail feature penalty loss in step 4 includes: Step 41: For the i-th sample image x i The image is converted to grayscale to obtain a single-channel grayscale image G, whose width and height are proportional to x. i Consistent; Step 42: Adjust the grayscale image G of size 1×H×W to G' of size 1×H0×W0 through the resize operation, and perform L2 normalization on G'. Step 43: Reduce the channel dimension of the shallowest feature F0 of the sample to obtain F0' with size 1×H0×W0, and perform standardization processing in the same way; Step 44: Calculate the L2 distance of the normalized features obtained in steps 42 and 43 as the detail feature penalty loss L. dfp The expression is: L dfp =||normal(G')-normal(F0')|| The normal() function represents the normalization operation, and the actual calculation method is: new data = (original data - minimum value) / (maximum value - minimum value); Step 5 includes: Step 51: The input to the semantic information regularization module is the feature maps of m intermediate layers. The semantic information regularization module calculates for each layer of feature maps, specifically for the j-th layer feature map F. j The semantic information of the feature maps from the (j+1)th to the mth layers is used to supervise the jth layer of the current training. Step 52: Calculate the fully connected output for the corresponding feature map using the auxiliary classification branch of each layer. The auxiliary classification branch includes, in sequence, a convolutional layer, a global average pooling layer, and a fully connected layer. The feature map F of the j-th layer... j z is obtained through the calculation of the auxiliary classification branch. j ; Step 53: Calculate the true label y of the sample using the Label Smooth method. i The provided soft classification loss will determine the true label y of the sample. i Converting to a soft-label probability distribution, its expression is: Where K represents the number of categories, ε Let represent any minimum value, and k represent the index position of the probability distribution; Step 54: Calculate the soft classification loss, expressed as: Where, p j For z j The probability distribution of p j,k Let k be the probability value, and its expression is: The semantic information transfer loss of the j-th layer is expressed as: Where KL(·) is the Kullback-Leibler divergence function, σ(·) is the softmax function, and τ is the smoothing coefficient; Step 55: Obtain the semantic information regularization loss, expressed as: The hyperparameter β is used to balance the contributions of the soft classification loss and the semantic information transfer loss to the semantic information regularization loss.

2. The binary quantization method for neural networks according to claim 1, characterized in that, The total loss in step 6 is: L=L ce +αL dfp +L sir Where α is the loss L that balances the detailed feature penalty. dfp and the semantic information regularization loss L sir The weighting coefficients for the two losses.

3. The binary quantization method for neural networks according to claim 1, characterized in that, The preprocessing in step 1 includes: randomly flipping the image horizontally, randomly cropping it, filling the boundary parts with zeros, adjusting the resolution to 32*32 and performing normalization processing.

4. A neural network for image classification, characterized in that, The neural network is trained using the binary quantization method described in any one of claims 1-3.

5. An image classification method, characterized in that, The image classification method includes: inputting the image to be classified into the neural network of claim 4, wherein the neural network for image classification processes the input image and outputs the classification result.

6. The image classification method according to claim 5, characterized in that, The neural networks include, but are not limited to: VGG networks and ResNet networks.

7. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the binary quantization method of the neural network as described in any one of claims 1-3 and / or the image classification method as described in any one of claims 5-6.

8. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a binary quantization method for a neural network as claimed in any one of claims 1-3 and / or an image classification method as claimed in any one of claims 5-6.

Images