Construction method of target recognition binary neural network in data-free scene

Abstract

The invention relates to a method for constructing a target recognition binary neural network in a data-free scene, and belongs to the technical field of model compression. According to the method, random noise is sampled to generate a false sample, the false sample is used as a drive, and a data-free adversarial distillation framework is adopted to train a target recognition binary neural network. In the framework, a binary neural network is constructed through binarization weight and activation, and category distribution of generated samples is balanced by utilizing category information entropy maximization. In addition, the intra-class feature clustering constraint is applied to enhance the heterogeneity of the same class of samples, so that the generalization performance of the network is improved. And finally, through training, constructing a target identification binary neural network based on a data-free adversarial distillation framework. According to the method, the problems of data privacy and non-direct accessibility can be effectively solved, meanwhile, the challenge that the computing power of edge equipment is limited in a target identification task is overcome, and the optimization balance of efficiency is realized while the precision is ensured.

Description

Technical Field

[0001] The invention relates to a method for constructing a binary neural network for target recognition in a data-free scenario, and belongs to the technical field of model compression. Background Art

[0002] Deep neural networks have achieved remarkable results in tasks such as image classification, object detection, and semantic segmentation. However, deep neural networks usually rely on high-computing resources and have high computing speed and memory requirements for hardware devices, which limits their application on resource-constrained devices. As an extreme parameter quantization scheme, binary neural networks quantize activation values ​​and weight values ​​into 1 bit, greatly reducing the amount of calculation and parameters, and are extremely suitable for resource-constrained environments.

[0003] However, in general, some special data sets often involve personal privacy issues. Therefore, their data may not be directly used for training. Data-free adversarial distillation can effectively solve this problem. It does not directly use the data set to train the target model, but trains the model through knowledge transfer. Data-free adversarial distillation is very suitable for scenarios with extremely high privacy and security requirements.

[0004] Based on the above, the present invention takes advantage of the binary neural network and the advantages of no-data adversarial distillation, and proposes a method for constructing a binary neural network for target recognition in a no-data scenario. At the same time, the maximization of category information entropy and intra-category feature clustering constraints are introduced to optimize samples. Summary of the invention

[0005] The purpose of the present invention is to provide a method for constructing a binary neural network for target recognition in a data-free scenario, aiming to solve the technical problems that many current scenarios cannot access the original data set to train the neural network and that the neural network cannot be deployed on edge devices due to insufficient computing power.

[0006] To achieve the above objectives, the present invention provides a method for constructing a binary neural network for target recognition in a data-free scenario, which can implement efficient and fast target recognition tasks in scenarios where the original data set is unavailable. The specific steps are as follows:

[0007] Step 1: Binarize the weights and activations of the baseline network in the full-precision object recognition network, build a binary student network, introduce the pre-trained full-precision object recognition model as the teacher network, introduce the generator, and generate fake samples through random noise;

[0008] Step 2: Take the fake samples generated by the generator as the network input, and use the characteristics of the binary neural network weights and activation values ​​taking discrete values ​​to calculate the difference between the continuous output of the full-precision teacher network and the discrete output of the binary student network. By minimizing the difference, update the weight parameters of the binary network, and optimize the generator by maximizing the difference, so as to achieve difference-driven adversarial optimization under the binary network structure.

[0009] Step 3: According to the output of the binary student network, the category information entropy is calculated and maximized, and the opposite of the information entropy is taken as the additional loss function of the generator. By utilizing the characteristics of the discretized output of the binary neural network and maximizing the category information entropy, the generator generates fake samples with diverse categories and achieves a balanced category distribution in the binary output space.

[0010] Step 4: Perform intra-class clustering on the output features of the intermediate layer of the target recognition binary neural network and apply it to the generator as a constraint loss term;

[0011] Step 5: Repeatedly iterate and optimize Step 2, Step 3, and Step 4 and update the gradient until the model converges, and finally output a fully trained target recognition binary neural network model.

[0012] The Step 2 is specifically as follows:

[0013] The mean absolute error function is used to calculate the difference between the continuous output of the teacher network and the discrete output of the binary student network, and reducing the difference is used as the training goal of the binary neural network. The adversarial distillation loss function of the target recognition binary neural network is:

[0014]

[0015] Where T F represents the full-precision target recognition network, The feature map output by the full-precision network baseline network ResNet18 is pre-trained in advance, W r(n) represents the weight of the nth layer, F r(n-1) Represents the activation of layer n-1, through full-precision convolution The normalization layer BN and the nonlinear function δ obtain the output feature map. The specific network is as follows:

[0016]

[0017] The full-precision feature map representing the output of the baseline network is used to select the candidate region position coordinates r through the region candidate network i , and then through the candidate region pooling layer RoI Pooling, the fully connected layer FC, and the classification layer σ clsand the bounding box regression layer f reg The output prediction result generated, r i ={r1, r2, ..., r n} represents the location coordinates of a candidate region, generated by the region candidate network, and the RoL pooling layer proposes r for each region i , and map it to the feature map F r(n) The feature is transformed by the fully connected layer FC, and finally, the classification layer σ cls The classification prediction result is obtained through the softmax function, and the border regression layer f reg The border position adjustment parameters of the candidate area are given for precise positioning. The specific network is as follows:

[0018]

[0019] where r i ={r1, r2, ..., r n};

[0020] S B represents the target recognition network based on binarization, The feature map output by the binary baseline network ResNet-18 is binarized, and its activation and features are binarized. b(n) represents the binary weight of the nth layer, F b(n-1) Represents the binary activation of the n-1th layer. The specific network is as follows:

[0021]

[0022] The binary feature map representing the output of the baseline network is used to select the candidate region position coordinates r through the region candidate network i , candidate region pooling layer RoI Pooling, fully connected layer FC, classification layer σ cls and the bounding box regression layer f reg The output prediction result generated, F b(n) Represents the binary activation of the nth layer. The specific network is as follows:

[0023]

[0024] where r i ={r1, r2, ..., r n};

[0025] The generator is introduced to form data-free adversarial distillation, and a binary-based target recognition network is constructed in the data-free scenario. The training is divided into two stages:

[0026] The first stage: fix the binary neural network, iterate the generator, and maximize the model difference to train the generator;

[0027] Increase the absolute value distance between the feature maps output by the full-precision baseline network and the binary baseline network, and increase the absolute value distance between the output prediction values ​​of the full-precision recognition network and the binary recognition network, including the classification prediction value and the bounding box prediction value;

[0028] The second stage: fix the generator, iterate the binary neural network, minimize the model difference to train the binary neural network, and the training process is opposite to the first stage.

[0029] The Step 3 is specifically as follows:

[0030] For the output prediction of the binary network, the maximum information entropy of the category is used to constrain the quantitative difference between its output prediction values, and finally constrain the generator to make the probability of generating samples of each category equal; the information entropy of the prediction probability of each category in each batch is calculated, and its information entropy is increased during the generator update process. The generator adjusts the data distribution of the generated samples so that the output prediction of the final binary network meets the constraints. The final loss calculation formula for maximizing the category information entropy is:

[0031]

[0032] As shown in the above formula, n is the set batch size, log is the logarithmic function with base 2, P(x i ) is the predicted probability value of the teacher network, P(y i ) is the predicted probability value of the binary network; the information entropy is calculated for each category prediction probability and summed up, and then the opposite of the information entropy is taken as the loss to increase the category information entropy.

[0033] The Step 4 is specifically as follows:

[0034] Use intra-class feature clustering constraints to increase the distance between the features of the generated fake samples and their cluster centers. Count the classification prediction values ​​in each batch, classify each category, and divide it by the number of samples to obtain the average value of the feature distribution of samples in this category, which is recorded as the cluster center. The cluster center calculation formula is:

[0035]

[0036] As shown in the above formula, d represents a certain category of false samples, F t (d) represents the feature map obtained by inputting the generated sample into the binary neural network, c represents a certain category, k represents the number of samples in this category, and for the above cluster center, the cosine distance between each feature of this category and the cluster center is calculated. The calculation formula is as follows:

[0037]

[0038] After calculating the cosine distance between each feature and its cluster center, the cosine distance is adjusted according to the settings of the upper and lower limit parameters. The generator increases the cosine distance between the generated sample features by updating to increase the intra-class heterogeneity. The loss function formula of the clustering constraint is as follows:

[0039] L C =max(λ1-D cos ,0)+max(D cos -λ u ,0)

[0040] λ l is the lower limit, λ u is the upper limit, and max means taking the maximum value;

[0041] The cosine distance between each category feature and its cluster center is constrained between the lower limit and the upper limit through the intra-category feature clustering constraint. cos >λ l When l -D cos The value of max(λ1-D cos , 0)=0, indicating that the generated sample heterogeneity is high enough, and the generator is cos , 0) This sub-loss is not updated; when D cos <λ l When l -D cos The value is positive, max(λ1-D cos , 0) = λ l -D cos , the generator increases D cos To reduce losses; when D cos <λ u When max(D cos -λ u , 0)=0, indicating that the cosine distance is within the normal range, and the generator is cos -λ u , 0) This sub-loss is not updated; when D cos >λ u When max(D cos -λ u , 0) = D cos -λ u , the generator reduces D cos To reduce losses.

[0042] The beneficial effects of the present invention are as follows: compared with the prior art, the present invention not only solves the challenge of not being able to access the original data set to train the neural network in many current scenarios, but also solves the problem of being unable to deploy the neural network on edge devices due to insufficient computing power. In the specific optimization scheme, the problem of loss of generated sample category information and high intra-class similarity is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0044] Figure 2 It is a detailed diagram of the network framework of the present invention;

[0045] Figure 3 This is a schematic diagram of maximizing the category information entropy of the present invention;

[0046] Figure 4 It is a schematic diagram of intra-class feature clustering constraints of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0048] Example 1: A method for constructing a binary neural network for target recognition in a data-free scenario, see Figures 1 to 4 . Figure 1 The overall framework diagram is shown in Figure 1. Figure 2 This is a detailed diagram of the network framework. The specific steps are as follows:

[0049] Step 1: Figure 2 As shown, the weights and activations of the baseline network in the full-precision target recognition network are binarized, a binary student network is constructed, a pre-trained full-precision target recognition model is introduced as the teacher network, and a generator is introduced to generate fake samples through random noise.

[0050] Step 2, such as Figure 2 As shown in the figure, the fake samples generated by the generator are used as network inputs. The difference between the continuous output of the full-precision teacher network and the discrete output of the binary student network is calculated by taking advantage of the characteristics that the weights and activation values ​​of the binary neural network take discrete values ​​(1 and -1). By minimizing the difference, the weight parameters of the binary network are updated. By maximizing the difference, the generator is optimized, thereby realizing difference-driven adversarial optimization under the binary network structure.

[0051] The mean absolute error function is used to calculate the difference between the continuous output of the teacher network and the discrete output of the binary student network, and reducing the difference is used as the training goal of the binary neural network. The adversarial distillation loss function of the target recognition binary neural network is:

[0052]

[0053] Where T F represents the full-precision target recognition network, The feature map output by the full-precision network baseline network ResNet18 is pre-trained in advance, W r(n) represents the weight of the nth layer, F r(n-1) Represents the activation of layer n-1, through full-precision convolution The normalization layer BN and the nonlinear function δ obtain the output feature map. The specific network is as follows:

[0054]

[0055] The full-precision feature map representing the output of the baseline network is used to select the candidate region position coordinates r through the region candidate network i , and then through the candidate region pooling layer RoI Pooling, the fully connected layer FC, and the classification layer σ cls and the bounding box regression layer f reg The output prediction result generated, r i ={r1, r2, ..., r n} represents the location coordinates of a candidate region, generated by the region candidate network, and the RoL pooling layer proposes r for each region i , and map it to the feature map F r(n) The feature is transformed by the fully connected layer FC, and finally, the classification layer σ cls The classification prediction result is obtained through the softmax function, and the border regression layer f reg The border position adjustment parameters of the candidate area are given for precise positioning. The specific network is as follows:

[0056]

[0057] where r i ={r1, r2, ..., r n};

[0058] S B represents the target recognition network based on binarization, The feature map output by the binary baseline network ResNet-18 is binarized, and its activation and features are binarized. b(n) represents the binary weight of the nth layer, F b(n-1) Represents the binary activation of the n-1th layer. The specific network is as follows:

[0059]

[0060] The binary feature map representing the output of the baseline network is used to select the candidate region position coordinates r through the region candidate network i , candidate region pooling layer RoI Pooling, fully connected layer FC, classification layer σ cls and the bounding box regression layer f reg The output prediction result generated, F b(n) Represents the binary activation of the nth layer. The specific network is as follows:

[0061]

[0062] where r i ={r1, r2, ..., r n};

[0063] The generator is introduced to form data-free adversarial distillation, and a binary-based target recognition network is constructed in the data-free scenario. The training is divided into two stages:

[0064] Phase 1: Fixed the binary neural network, iterated the generator 5 times, and maximized the model difference to train the generator;

[0065] Increase the absolute value distance between the feature maps output by the full-precision baseline network and the binary baseline network, and increase the absolute value distance between the output prediction values ​​of the full-precision recognition network and the binary recognition network, including the classification prediction value and the bounding box prediction value;

[0066] The second stage: fix the generator, iterate the binary neural network once, minimize the model difference to train the binary neural network, and the training process is opposite to the first stage.

[0067] Step 3: Figure 3 As shown, according to the output of the binary student network, the category information entropy is calculated and maximized, and the opposite of the information entropy is taken as the additional loss function of the generator. By utilizing the characteristics of the discretized output of the binary neural network and maximizing the category information entropy, the generator generates false samples with category diversity and achieves a balance in the category distribution in the binary output space.

[0068] For the output prediction of the binary network, the maximum information entropy of the category is used to constrain the quantitative difference between its output prediction values, and finally constrain the generator to make the probability of generating samples of each category equal; the information entropy of the prediction probability of each category in each batch is calculated, and its information entropy is increased during the generator update process. The generator adjusts the data distribution of the generated samples so that the output prediction of the final binary network meets the constraints. The final loss calculation formula for maximizing the category information entropy is:

[0069]

[0070] As shown in the above formula, n is the set batch size, log is the logarithmic function with base 2, P(x i ) is the predicted probability value of the teacher network, P(y i ) is the predicted probability value of the binary network; the sum of the two is divided by two. The purpose is to optimize the predicted probability distribution of the student network output while introducing the prior knowledge of the teacher network, so as to achieve better optimization of the probability distribution. The information entropy of each category prediction probability is calculated and summed, and then the opposite of the information entropy is taken as the loss to increase the category information entropy.

[0071] Step 4: Perform intra-class clustering on the output features of the intermediate layer of the target recognition binary neural network and apply it to the generator as a constraint loss term. By controlling the distance between features of different categories in the binary network, the intra-class heterogeneity is enhanced, so that the fake samples generated by the generator have greater diversity within the same category.

[0072] Using intra-class feature clustering constraints to increase the distance between the features of the generated fake samples and their cluster centers can improve intra-class heterogeneity, such as Figure 4 As shown. The classification prediction values ​​in each batch are counted, each category is classified, and divided by its sample number to obtain the average value of the sample feature distribution of this category, which is recorded as the cluster center. The cluster center calculation formula is:

[0073]

[0074] As shown in the above formula, d represents a certain category of false samples, F t (d) represents the feature map obtained by inputting the generated sample into the binary neural network, c represents a certain category, k represents the number of samples in this category, and for the above cluster center, the cosine distance between each feature of this category and the cluster center is calculated. The calculation formula is as follows:

[0075]

[0076] After calculating the cosine distance between each feature and its cluster center, the cosine distance is adjusted according to the settings of the upper and lower limit parameters. The generator increases the cosine distance between the generated sample features by updating to increase the intra-class heterogeneity. The loss function formula of the clustering constraint is as follows:

[0077] L C =max(λ1-D cos ,0)+max(D cos -λ u ,0)

[0078] Lower limit λ l is 0.1, the upper limit λ uIt is 0.8, and max means taking the maximum value. After testing and verification, this range setting is the most reasonable and can maximize the degree of approximation to the real data.

[0079] By constraining the cosine distance between each category feature and its cluster center to be between the lower limit and the upper limit through intra-class feature clustering constraints, the correctness of each category of samples can be guaranteed while increasing the intra-class heterogeneity. cos >λ l When l -D cos The value of max(λ l -D cos , 0)=0, indicating that the generated sample heterogeneity is high enough, and the generator is cos , 0) This sub-loss is not updated; when D cos <λ l When l -D cos The value of is positive, max(λ l -D cos , 0) = λ l -D cos , the generator increases D cos To reduce losses; when D cos <λ u When max(D cos -λ u , 0)=0, indicating that the cosine distance is within the normal range, and the generator is cos -λ u , 0) This sub-loss is not updated; when D cos >λ u When max(D cos -λ u , 0) = D cos -λ u , the generator reduces D cos To reduce losses.

[0080] Step 5: Repeatedly iterate and optimize Step 2, Step 3, and Step 4 and update the gradient until the model converges, and finally output a fully trained target recognition binary neural network model.

[0081] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for constructing a binary neural network for target recognition in a data-free scenario, characterized in that: The specific steps of the method are as follows: Step 1: Binarize the weights and activations of the baseline network in the full-precision object recognition network, build a binary student network, introduce the pre-trained full-precision object recognition model as the teacher network, introduce the generator, and generate fake samples through random noise; Step 2: Take the fake samples generated by the generator as the network input, and use the characteristics of the binary neural network weights and activation values ​​taking discrete values ​​to calculate the difference between the continuous output of the full-precision teacher network and the discrete output of the binary student network. By minimizing the difference, update the weight parameters of the binary network, and optimize the generator by maximizing the difference, so as to achieve difference-driven adversarial optimization under the binary network structure. Step 3: According to the output of the binary student network, the category information entropy is calculated and maximized, and the opposite of the information entropy is taken as the additional loss function of the generator. By utilizing the characteristics of the discretized output of the binary neural network and maximizing the category information entropy, the generator generates fake samples with diverse categories and achieves a balanced category distribution in the binary output space. Step 4: Perform intra-class clustering on the output features of the intermediate layer of the target recognition binary neural network and apply it to the generator as a constraint loss term; Step 5: Repeatedly iterate and optimize Step 2, Step 3, and Step 4 and update the gradient until the model converges, and finally output a fully trained target recognition binary neural network model.

2. The method for constructing a binary neural network for target recognition in a data-free scenario according to claim 1, characterized in that: The Step 2 is specifically as follows: The mean absolute error function is used to calculate the difference between the continuous output of the teacher network and the discrete output of the binary student network, and reducing the difference is used as the training goal of the binary neural network. The adversarial distillation loss function of the target recognition binary neural network is: Where T F represents the full-precision target recognition network, The feature map output by the full-precision network baseline network ResNet18 is pre-trained in advance, W r(n) represents the weight of the nth layer, F r(n-1) Represents the activation of layer n-1, through full-precision convolution The normalization layer BN and the nonlinear function δ obtain the output feature map. The specific network is as follows: The full-precision feature map representing the output of the baseline network is used to select the candidate region position coordinates r through the region candidate network i , and then through the candidate region pooling layer RoI Pooling, the fully connected layer FC, and the classification layer σ cls and the bounding box regression layer f reg The output prediction result generated, r i ={r1, r2, ..., r n } represents the position coordinates of a candidate region, which is generated by the region candidate network. The RoL pooling layer proposes r for each region. i , and map it to the feature map F r(n) The feature is transformed by the fully connected layer FC, and finally, the classification layer σ cls The classification prediction result is obtained through the softmax function, and the border regression layer f reg The border position adjustment parameters of the candidate area are given for precise positioning. The specific network is as follows: where r i ={r1, r2, ..., r n }; S B represents the target recognition network based on binarization, The feature map output by the binary baseline network ResNet-18 is binarized, and its activation and features are binarized. b(n) represents the binary weight of the nth layer, F b(n-1) Represents the binary activation of the n-1th layer. The specific network is as follows: The binary feature map representing the output of the baseline network is used to select the candidate region position coordinates r through the region candidate network i , candidate region pooling layer RoI Pooling, fully connected layer FC, classification layer σ cls and the bounding box regression layer f reg The output prediction result generated, F b(n) Represents the binary activation of the nth layer. The specific network is as follows: where r i ={r1, r2, ..., r n }; The generator is introduced to form data-free adversarial distillation, and a binary-based target recognition network is constructed in the data-free scenario. The training is divided into two stages: The first stage: fix the binary neural network, iterate the generator, and maximize the model difference to train the generator; Increase the absolute value distance between the feature maps output by the full-precision baseline network and the binary baseline network, and increase the absolute value distance between the output prediction values ​​of the full-precision recognition network and the binary recognition network, including the classification prediction value and the bounding box prediction value; The second stage: fix the generator, iterate the binary neural network, minimize the model difference to train the binary neural network, and the training process is opposite to the first stage.

3. The method for constructing a binary neural network for target recognition in a data-free scenario according to claim 1, characterized in that: The Step 3 is specifically as follows: For the output prediction of the binary network, the maximum information entropy of the category is used to constrain the quantitative difference between its output prediction values, and finally constrain the generator to make the probability of generating samples of each category equal; the information entropy of the prediction probability of each category in each batch is calculated, and its information entropy is increased during the generator update process. The generator adjusts the data distribution of the generated samples so that the output prediction of the final binary network meets the constraints. The final loss calculation formula for maximizing the category information entropy is: As shown in the above formula, n is the set batch size, log is the logarithmic function with base 2, P(x i ) is the predicted probability value of the teacher network, P(y i ) is the predicted probability value of the binary network; the information entropy is calculated for each category prediction probability and summed up, and then the opposite of the information entropy is taken as the loss to increase the category information entropy.

4. The method for constructing a binary neural network for target recognition in a data-free scenario according to claim 1, characterized in that: The Step 4 is specifically as follows: Use intra-class feature clustering constraints to increase the distance between the features of the generated fake samples and their cluster centers. Count the classification prediction values ​​in each batch, classify each category, and divide it by the number of samples to obtain the average value of the feature distribution of samples in this category, which is recorded as the cluster center. The cluster center calculation formula is: As shown in the above formula, d represents a certain category of false samples, F t (d) represents the feature map obtained by inputting the generated sample into the binary neural network, c represents a certain category, k represents the number of samples in this category, and for the above cluster center, the cosine distance between each feature of this category and the cluster center is calculated. The calculation formula is as follows: After calculating the cosine distance between each feature and its cluster center, the cosine distance is adjusted according to the settings of the upper and lower limit parameters. The generator increases the cosine distance between the generated sample features by updating to increase the intra-class heterogeneity. The loss function formula of the clustering constraint is as follows: L C =max(λ1-D cos ,0)+max(D cos -l u ,0) λ l is the lower limit, λ u is the upper limit, and max means taking the maximum value; The cosine distance between each category feature and its cluster center is constrained between the lower limit and the upper limit through the intra-category feature clustering constraint. cos >λ l When l -D cos The value of max(λ1-D cos，0 )=0, indicating that the generated sample heterogeneity is high enough, and the generator is cos，0 ) This sub-loss is not updated; when D cos <λ l When l -D cos The value is positive, max(λ1-D cos，0 )=λ l -D cos , the generator increases D cos To reduce losses; when D cos <λ u When max(D cos -λ u , 0)=0, indicating that the cosine distance is within the normal range, and the generator is cos -λ u , 0) This sub-loss is not updated; when D cos >λ u When max(D cos -λ u , 0) = D cos -λ u , the generator reduces D cos To reduce losses.

Images