A vehicle classification method based on self-distillation binary neural network

Abstract

The application relates to a vehicle classification method based on a self-distillation binary neural network and belongs to the technical field of model compression. First, a vehicle picture dataset containing N similar label pairs is acquired, and a training set and a test set are obtained according to a division ratio of (4+N):1; the training set pictures are input into a dynamic approximate gradient binary neural network built to obtain category output prediction; the network of the current iteration training becomes a teacher, the category output prediction is filtered and averaged to obtain a soft label library indicating correctness; the network of the next iteration training becomes a student, the soft label library provides soft label teacher knowledge, and self-distillation is performed at the tail of the binary neural network; the iteration of the teacher-student self-distillation training process is continuously carried out, the iteration number of the distillation effect is 2 to SumE, so that the classification precision of the binary neural network is improved; and the vehicle test set pictures are classified and predicted according to the trained binary classification model to obtain a classification result.

Description

Technical Field

[0001] This invention relates to a vehicle classification method based on a self-distillation binary neural network, belonging to the field of model compression technology. Background Technology

[0002] In recent years, PLC equipment meeting the IEC 61499 standard has developed rapidly. Meanwhile, deep convolutional neural networks have achieved great success in computer vision fields such as image classification, object detection, and semantic segmentation, providing an opportunity for more effective applications in intelligent transportation systems. Binary neural networks represent the ultimate case of parameter quantization. They restrict the values ​​of convolutional parameters in the network to -1 and +1, and use inexpensive XOR and bitwise operations instead of expensive floating-point convolution operations, significantly improving computational and storage efficiency. This makes them highly suitable for intelligent transportation PLC equipment with limited computing power and memory storage resources, enabling the effective deployment of high-performance neural networks. However, the accuracy of binary neural networks is significantly lower than that of full-precision neural networks with the same structure because parameter binarization leads to severe information loss. Therefore, there is an urgent need to propose appropriate methods to provide more information to the training process of binary neural networks to improve accuracy. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a vehicle classification method based on a self-distillation binary neural network, which can provide the binary neural network with more information containing intra-class and inter-class similarity knowledge, so that the binary neural network with low accuracy can obtain better vehicle classification results.

[0004] The technical solution of this invention is: a vehicle classification method based on a self-distillation binary neural network. First, a vehicle image dataset containing N similar label pairs is obtained, divided into a training set and a test set according to a ratio of (4+N):1. The training set images are input into a dynamically approximate gradient binary neural network to obtain category output predictions. The network trained in the current iteration becomes the teacher, filtering and averaging the category output predictions to obtain a soft label library indicating correctness. The network trained in the next iteration becomes the student, with the soft label library providing soft label teacher knowledge, and self-distillation is performed at the tail of the binary neural network. The teacher-student self-distillation training process is continuously iterated, with the number of iterations for distillation ranging from 2 to SumE, to improve the classification accuracy of the binary neural network. Based on the trained binary classification model, the vehicle test set images are classified and predicted to obtain the classification result.

[0005] The specific steps are as follows:

[0006] Step 1: Obtain a vehicle image dataset containing N similar label pairs, and divide it into training and test sets according to a ratio of (4+N):1.

[0007] Step 2: Input the training set images into the constructed dynamic approximate gradient binary neural network to obtain the category output prediction.

[0008] Step 3: The network trained in this round of iterations becomes the teacher, filtering and averaging the category output predictions to obtain a soft label library that indicates the correctness.

[0009] Step 4: The network trained in the next iteration becomes the student, and the soft label library provides soft label teacher knowledge, which is then self-distilled at the tail of the binary neural network.

[0010] Step 5: Continuously conduct self-distillation training with teachers and students to improve the classification accuracy of the binary neural network.

[0011] The aforementioned teacher-student self-distillation process always uses the network of the current iteration as the teacher and the network of the next iteration as the student. Distillation is performed on the student network, and the number of iterations for distillation is 2 to SumE, meaning that the first iteration of training only generates teacher knowledge.

[0012] Step 6: Based on the trained binary classification model, perform classification prediction on the vehicle test set images to obtain the classification results.

[0013] Step 1 specifically refers to:

[0014] Step 1.1: The dataset contains images of M types of vehicles in a specific scene. It must contain at least one of the following similar label pairs: bus-truck, van-car, electric vehicle-motorcycle. The number of similar label pairs contained therein is set to N.

[0015] Step 1.2: Divide the training set and the test set into a ratio of (4+N):1, and perform one-to-one annotation on the original images and labels.

[0016] Step 2 specifically includes:

[0017] Step 2.1: For the constructed convolutional neural network, except for the first and last layers, all other convolutional layer parameters are binarized using the Sign function, and binary convolution operations are performed using inexpensive XOR and bitwise operations, replacing expensive traditional floating-point full-precision convolution operations.

[0018] Step 2.2: During backpropagation, the Sign function is replaced with a dynamic approximation function, DynamicHardTanh, and the resulting dynamic approximation gradient is backpropagated to the full-precision weights to address the non-differentiability problem of the Sign function. The formula for the dynamic approximation function, DynamicHardTanh, is as follows.

[0019]

[0020] In the formula, Y is a dynamic threshold that changes with the number of training iterations. If E is the current iteration number and SumE is the total number of iterations, then:

[0021]

[0022] During backpropagation, the actual backpropagated dynamic approximation gradient DyGradient after approximating the Sign function by the dynamic approximation function DynamicHardTanh is:

[0023]

[0024] Step 2.3: For the input image, by stacking N convolutional blocks and using the final linear classification layer, the category output prediction is obtained. The process can be formulated as follows:

[0025]

[0026] In the formula, This represents the nth convolutional block, typically consisting of binary convolutions (except for the first convolutional block), batch normalization, and a non-linear activation function. For an input image I, intermediate features are extracted by stacking N convolutional blocks, and finally, a linear classifier is used. Obtain the output prediction P, where P indicates the confidence of the binary neural network for each class, and assumes that the highest score is the correct class.

[0027] Step 3 specifically refers to:

[0028] Step 3.1: Filter all output predictions for the input image, remove incorrect output predictions, and keep only the correct output predictions.

[0029] Step 3.2: Averaging the correct predictions for the same category yields a soft label library with correct indications. The size of the resulting soft label library is M×M. This process is formalized as follows:

[0030]

[0031] In the formula, This represents the correct prediction output for category j after filtering. It is its quantity. The process of assembling a soft tag library involves averaging M categories to obtain M vectors of M elements each, which are then stacked in order of category to form the final soft tag library. As the number of training iterations increases, the accuracy of the binary neural network rises, and the number of incorrect predictions decreases. While maintaining accuracy, it will further enhance the ability to generalize knowledge about category relationships.

[0032] Step 4 specifically refers to:

[0033] Step 4.1: Based on the true category of the input image, extract the corresponding category label from the soft label library to form a one-to-one annotation, which serves as the soft label teacher knowledge. This means that a student model becomes a teacher model, using its own knowledge to form soft labels, supplementing the lack of information in the true labels, and completing the self-distillation process of the binary neural network to provide more information during training.

[0034] Step 4.2: At the tail of the binary neural network, both soft labels and real labels are used for additional supervised training. The loss term corresponding to the soft labels is:

[0035]

[0036] In the formula, This represents the output prediction of the student network. This refers to soft labels for teachers. Representing the Kullback–Leibler divergence function:

[0037]

[0038] For the Softmax function:

[0039]

[0040] In the formula, T is the temperature coefficient. As it increases, the soft label distribution produced by the Softmax function becomes softer, which helps to provide more information to the student network parameters during backpropagation; T=4 is set.

[0041] The final total loss function is:

[0042]

[0043] In the formula, This represents the cross-entropy function, which uses the output prediction P to match the true label. Calculate the loss term for the image classification task.

[0044] The formula for the cross-entropy function is:

[0045]

[0046] In the formula, It is a trade-off hyperparameter used to balance the two loss terms.

[0047] The beneficial effects of this invention are as follows: This invention uses its own output prediction to generate soft-label teacher knowledge for self-distillation supervised training. The proposed soft-label library provides rich intra-class and inter-class similarity knowledge for binary neural networks, which can help student models converge to the minimum value, thus leading to better generalization. It can also prevent the gradient vanishing problem in student binary neural networks. In addition, it avoids various shortcomings caused by external teachers in traditional techniques. It improves the classification accuracy of binary neural networks without increasing any additional inference cost, making it more efficient and practical, and alleviating the problem of low accuracy caused by information scarcity during the training process of binary neural networks. Attached Figure Description

[0048] Figure 1 This is a flowchart of the present invention;

[0049] Figure 2 This is a schematic diagram illustrating the binary convolution operation process in the binary neural network provided by the present invention;

[0050] Figure 3 This is a schematic diagram illustrating the process of implementing the through-estimator using the dynamic approximation function of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the change of the dynamic threshold in the dynamic approximation function of the present invention;

[0052] Figure 5 This is a structural diagram of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0054] Example 1: As Figure 1 As shown, a vehicle classification method based on a self-distillation binary neural network comprises the following steps:

[0055] Step 1: Obtain a vehicle image dataset containing N similar label pairs, and divide it into training and test sets according to a ratio of (4+N):1.

[0056] Step 1.1: The dataset contains images of M types of vehicles in a specific scene. It must contain at least one of the following similar label pairs: bus-truck, van-car, electric vehicle-motorcycle. The number of similar label pairs contained therein is set to N.

[0057] Step 1.2: Divide the training set and the test set into a ratio of (4+N):1, and perform one-to-one annotation on the original images and labels.

[0058] Step 2: Input the training set images into the constructed dynamic approximate gradient binary neural network to obtain the category output prediction.

[0059] Step 2.1: For the constructed convolutional neural network, except for the first and last layers, all parameters of other convolutional layers are binarized using the Sign function. The binarization formula is as follows:

[0060] (1)

[0061] In the formula, This indicates full-precision weights. This represents the full-precision activation calculated from the previous network layers, such as batch normalized layers. and This represents the calculated binary weights and binary activations.

[0062] Binary convolution operations are constructed using inexpensive XOR and bitwise operations to replace expensive traditional floating-point full-precision convolution operations. The specific formula is as follows:

[0063] (2)

[0064] In the formula, This represents the traditional convolution operation using multiplication and addition. This represents a binary convolution operation using XOR and counter operations. The replaced binary convolution operation is as follows: Figure 2 As shown.

[0065] Step 2.2: As Figure 3 As shown, during backpropagation, the Sign function is replaced by a dynamic approximation function, DynamicHardTanh, and the resulting dynamic approximation gradient is backpropagated to the full-precision weights to address the non-differentiability problem of the Sign function. The formula for the dynamic approximation function, DynamicHardTanh, is:

[0066] (3)

[0067] In the formula, Y is a dynamic threshold that changes with the number of training iterations. If E is the current iteration number and SumE is the total number of iterations, then:

[0068] (4)

[0069] Specific function value changes are as follows Figure 4 As shown.

[0070] During backpropagation, the actual backpropagated dynamic approximation gradient DyGradient after approximating the Sign function by the dynamic approximation function DynamicHardTanh is:

[0071] (5)

[0072] Step 2.3: For the input image, by stacking N convolutional blocks and using the final linear classification layer, the category output prediction is obtained. The process can be formulated as follows:

[0073] (6)

[0074] In the formula, This represents the nth convolutional block, typically consisting of binary convolutions (except for the first convolutional block), batch normalization, and a non-linear activation function. For an input image I, intermediate features are extracted by stacking N convolutional blocks, and finally, a linear classifier is used. Obtain the output prediction P, where P indicates the confidence of the binary neural network for each class, and assumes that the highest score is the correct class.

[0075] Step 3: As Figure 5 As shown, in this round of iterative training, the network becomes the teacher, filtering and averaging the category output predictions to obtain a soft label library that indicates the correctness.

[0076] Step 3.1: Filter all output predictions for the input image, remove incorrect output predictions, and keep only the correct output predictions;

[0077] Step 3.2: Averaging the correct predictions for the same category yields a soft label library of correct indications. The size of the resulting soft label library is M×M. This process can be formalized as follows:

[0078] (7)

[0079] In the formula This represents the correct prediction output for category j after filtering. It refers to its quantity. The process of assembling a soft tag library involves averaging M categories to obtain M vectors of M elements each, which are then stacked in order of category to form the final soft tag library. As the number of training iterations increases, the accuracy of the binary neural network rises, and the number of incorrect predictions decreases. While maintaining accuracy, it will further enhance the ability to generalize knowledge about category relationships.

[0080] Step 4: As Figure 5 As shown, the network trained in the next iteration becomes the student, and the soft-label library provides soft-label teacher knowledge, which is then self-distilled at the tail of the binary neural network.

[0081] Step 4.1: Based on the true category of the input image, extract the corresponding category label from the soft label library to form a one-to-one annotation, which serves as the soft label teacher knowledge. This means that a student model becomes a teacher model, using its own knowledge to form soft labels, supplementing the information-poor true labels, and completing the self-distillation process of the binary neural network to provide more information during training. Soft labels provide the binary neural network with more knowledge of category relationships. For example, soft labels suggest that trucks are more similar to buses than motorcycles, and give the degree of similarity (labeled as 30% for trucks, 11% for motorcycles, compared to 59% for buses), while the true labels only indicate the category of buses, but rigidly assume that trucks and motorcycles have the same degree of similarity to buses.

[0082] Step 4.2: At the tail of the binary neural network, additional supervised training is performed using both soft labels and ground truth labels. The loss term corresponding to the soft labels is:

[0083] (8)

[0084] In the formula, This represents the output prediction of the student network. This indicates a soft label for teachers. Representing the Kullback–Leibler divergence function:

[0085] (9)

[0086] For the Softmax function:

[0087] (10)

[0088] T is a temperature coefficient. As it increases, the soft label distribution produced by the Softmax function becomes softer, which helps to provide more information to the student network parameters during backpropagation. We set T=4.

[0089] The final total loss function is:

[0090] (11)

[0091] In the formula, This represents the cross-entropy function, which uses the output prediction P to match the true label. Calculate the loss term for the image classification task. The cross-entropy function formula is:

[0092] (12)

[0093] It is a trade-off hyperparameter used to balance the two loss terms.

[0094] Step 5: Continuously perform teacher-student self-distillation training to improve the classification accuracy of the binary neural network. The teacher-student self-distillation process always uses the network of the current iteration as the teacher and the network of the next iteration as the student, with distillation occurring in the student network. The number of iterations for distillation is 2 to SumE, meaning the first iteration only generates teacher knowledge.

[0095] Step 6: Based on the trained binary classification model, perform classification prediction on the vehicle test set images to obtain the classification results.

[0096] In existing technologies, the combination of binary neural networks and knowledge distillation has the following two common drawbacks: Before training begins, a complete external full-precision teacher model always needs to be pre-trained, resulting in two-stage training and a sharp increase in time and memory costs; Due to the architectural differences between teachers and students, ability matching between teachers and students is required, leading to multi-step progressive distillation or the addition of additional network computation branches, which again drastically increases training time and causes an increase in inference computation costs.

[0097] This invention utilizes the output prediction of a binary neural network to provide soft-label teacher knowledge, forming a self-distillation process. It eliminates the need for externally pre-trained teacher models; a single-stage iterative distillation based solely on the network's own output prediction improves classification accuracy. Furthermore, since the teacher and student networks originate from the same source, they exhibit stronger capability adaptability. During the binary neural network inference stage, we discard the additional soft-label library distillation loss auxiliary component, thus incurring no additional inference cost compared to a standard binary neural network.

[0098] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A vehicle classification method based on a self-distillation binary neural network, characterized in that: Step 1: Obtain a vehicle image dataset containing N similar label pairs, and divide it into training and test sets according to a ratio of (4+N):

1. Step 2: Input the training set images into the constructed dynamic approximate gradient binary neural network to obtain the category output prediction; Step 3: The network trained in this round of iterations becomes the teacher, filtering and averaging the category output predictions to obtain a soft label library with correct indications; Step 4: The network trained in the next iteration becomes the student, and the soft label library provides soft label teacher knowledge, which is then self-distilled at the tail of the binary neural network; Step 5: Continuously conduct self-distillation training with teachers and students to improve the classification accuracy of the binary neural network; The aforementioned teacher-student self-distillation process always uses the network of the current iteration as the teacher and the network of the next iteration as the student. Distillation is performed on the student network, and the number of iterations for distillation is 2 to SumE, meaning that the first iteration of training only generates teacher knowledge. Step 6: Based on the trained binary classification model, perform classification prediction on the vehicle test set images to obtain the classification results; Step 2 specifically refers to: Step 2.1: For the constructed convolutional neural network, except for the first and last layers of the network, all other convolutional layer parameters are binarized using the Sign function, and binary convolution operations are performed using inexpensive XOR and bit-counting operations. Step 2.2: During backpropagation, the Sign function is replaced by the dynamic approximation function DynamicHardTanh, and the generated dynamic approximation gradient is backpropagated to the full-precision weights. The formula for the dynamic approximation function DynamicHardTanh is: ； In the formula, Y is a dynamic threshold that changes with the number of training iterations. If E is the current iteration number and SumE is the total number of iterations, then: ； During backpropagation, the actual backpropagated dynamic approximation gradient DyGradient after approximating the Sign function by the dynamic approximation function DynamicHardTanh is: ； Step 2.3: For the input image, by stacking N convolutional blocks and using the final linear classification layer, the category output prediction is obtained. The process can be formulated as follows: ； In the formula, This represents the nth convolutional block. For an input image I, intermediate features are extracted by stacking N convolutional blocks, and finally a linear classifier is used. Obtain the output prediction P.

2. The vehicle classification method based on a self-distillation binary neural network according to claim 1, characterized in that, Step 1 specifically refers to: Step 1.1: The dataset contains images of M types of vehicles in a specific scene, and must contain at least one of the following similar label pairs: bus-truck, van-car, electric vehicle-motorcycle, and the number of similar label pairs contained therein is set to N; Step 1.2: Divide the training set and the test set into a ratio of (4+N):1, and perform one-to-one annotation on the original images and labels.

3. The vehicle classification method based on a self-distillation binary neural network according to claim 1, characterized in that, Step 3 specifically refers to: Step 3.1: Filter all output predictions for the input image, remove incorrect output predictions, and keep only the correct output predictions; Step 3.2: Averaging the correct predictions for the same category yields a soft label library with correct indications. The size of the resulting soft label library is M×M. This process is formalized as follows: ； In the formula, This represents the correct prediction output for category j after filtering. It is its quantity. The process of assembling a soft tag library involves averaging M categories to obtain M vectors of M elements each, which are then stacked in order of category to form the final soft tag library. .

4. The vehicle classification method based on a self-distillation binary neural network according to claim 1, characterized in that, Step 4 specifically refers to: Step 4.1: Based on the actual category of the input image, extract the corresponding category tags from the soft tag library to form a one-to-one annotation, which serves as soft tag teacher knowledge; Step 4.2: At the tail of the binary neural network, both soft labels and real labels are used for additional supervised training. The loss term corresponding to the soft labels is: ； In the formula, This represents the output prediction of the student network. This refers to soft labels for teachers. Represents the Kullback–Leibler divergence function; The final total loss function is: ； In the formula, This represents the cross-entropy function, which uses the output prediction P to match the true label. Calculate the loss term for the image classification task.

Images