Target detection distillation method based on weighted mask guidance

Abstract

The invention discloses a target detection distillation method based on weighted mask guidance. The method comprises the following steps: training a second neural network through a first neural network; calculating a loss function when the second neural network fits the global features of the first neural network image; fitting of the second neural network to the local features is trained and a loss function when the second neural network fits the local features is calculated through the predicted bounding box, the prior box and truth phase marking optimization; forming a feature distillation loss function based on a weighted mask in a second neural network according to the loss function of the global features and the loss function of the local features; according to the method, the feature map with relatively high discrimination capability learned by the large network can be fully utilized to guide learning of the small network; a global feature map and local region features between the first neural network and the second neural network are effectively fitted, so that distillation is more targeted, and convergence of the second neural network is accelerated; the precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and classification, and in particular to a target detection distillation method based on weighted mask guidance. Background Art

[0002] Currently, in the field of image recognition and target detection, target detection classifiers are often needed. Target detection classifiers use neural networks to identify, detect and classify targets in images. Detectors with better performance usually rely on deeper and larger capacity basic networks, resulting in extremely high computational complexity and large memory usage for the entire detector, which greatly limits their application in resource-constrained scenarios.

[0003] In the prior art, the knowledge distillation method is used to solve the problem of large computational complexity and memory usage. Usually, the student network is used to fit the teacher network to improve the performance of the student network and promote the deployment of the student network. However, the knowledge distillation method cannot distinguish between the global feature fitting and the local feature fitting during the fitting process, that is, it cannot automatically control the global feature fitting attention and the local feature fitting attention.

[0004] Since the target region contains more information and should be paid more attention in the feature distillation stage, this method focuses on the features of the target region while minimizing the distance between the feature maps of the student network and the teacher network. Summary of the invention

[0005] The purpose of the present invention is to provide a target detection distillation method based on weighted mask guidance to solve the problem that the knowledge distillation method in the prior art cannot distinguish between the fitting of global features and the fitting of local features during the fitting process, that is, it cannot automatically control the fitting attention of global features and the fitting attention of local features.

[0006] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a target detection distillation method based on weighted mask guidance, which is applied to a target detection classifier, and the method includes: training a second neural network through a first neural network, wherein the first neural network is a large-capacity network and the second neural network is a small-capacity network; calculating the loss function when the second neural network fits the global features of the first neural network image; optimizing the training of the second neural network for the fitting of local features through predicted bounding boxes, prior boxes and truth annotations and calculating the loss function when the second neural network fits the local features, wherein the predicted bounding box is the prediction result output by the second neural network model, the prior box is a preset local feature box, and the truth annotation is supervision information; forming a feature distillation loss function based on a weighted mask in the second neural network according to the loss function of the global features and the loss function of the local features.

[0007] As a further improvement of the present invention, the method also includes: the "training the second neural network" includes: using a preset training set to train the first neural network; using the distillation temperature τ in the first neural network to output the probability distribution in the output layer; training the second neural network by adjusting the distillation temperature τ; controlling the distillation temperature τ=1, and calculating the detection method of the first neural network for the training set in the second neural network.

[0008] As a further improvement of the present invention, the method further includes: the calculation formula of the probability distribution is: the calculation formula of the probability distribution is:

[0009]

[0010] Where zi represents the logical output of the i-th category; pi represents the corresponding probability; τ is the distillation temperature coefficient; and j represents the total number of categories.

[0011] As a further improvement of the present invention, the method further includes: when training the second neural network, dynamically fitting the first neural network by calculating parameters of the second neural network.

[0012] As a further improvement of the present invention, the method further includes: the parameter calculation formula of the second neural network is:

[0013]

[0014] in, is the parameter of the second neural network; H(y * ,P S ) is the cross entropy loss of the true label; H(P T τ ,P S τ ) is the cross entropy loss of the probability distribution of the first neural network and the second neural network; α is the parameter for adjusting the two cross entropy losses; is the probability distribution of the first neural network; is the probability distribution of the second neural network.

[0015] As a further improvement of the present invention, the method further includes: the “loss function when the second neural network fits the global features of the first neural network image” is:

[0016]

[0017] in, is the loss function when the global features in the first neural network are fitted to the second neural network; ui is the feature map of the i-th prediction module in the first neural network; vi is the feature map corresponding to the i-th prediction module in the second neural network, r represents the dimension adjustment layer in the second neural network; M represents the number of feature maps of the prediction module that need to be distilled; is the weight parameter of the global distillation loss of each prediction module; 1 / 2N means that the number of filters in each layer of the second neural network is only 1 / 2N of the corresponding layer of the first neural network; D i Indicates the size of the spatial dimension in the feature map module; 1 / D i Indicates that the result of the feature map of i samples is normalized.

[0018] As a further improvement of the present invention, the method further includes: "optimizing the training of the second neural network for fitting local features by predicting the bounding box, the priori box and the truth annotation" includes: when fitting and training the second neural network by adjusting the distillation temperature τ, setting the local feature weight parameter λ l ; The second neural network outputs the predicted bounding box, compares it with the preset prior box and uses the truth annotation for proofreading; dynamically adjust the local feature weight parameter λ during the training process 1 .

[0019] As a further improvement of the present invention, the method further includes: the “loss function when the second neural network fits the local features” is:

[0020]

[0021] in, is the loss function when the local features in the first neural network are fitted to the second neural network; R represents the selected sample set; represents the regional features of the kth sample of the i-th prediction module in the first neural network; is the regional feature corresponding to the second neural network; r is the dimension adjustment layer; M represents the number of prediction module feature maps that need to be distilled; is the preset weight parameter of the local distillation loss of each prediction module; 1 / 2N means that the number of filters in each layer of the second neural network is only 1 / 2N of the corresponding layer of the first neural network; D () Indicates the size of the spatial dimension in the feature map module; 1 / D () Indicates that the results of the feature maps of k samples are normalized.

[0022] As a further improvement of the present invention, the method further comprises: the “dynamically adjusting the local feature weight parameter λ l" is: Use the loss size of the regression task in each optimization training and the local adjustment distillation loss size to adjust the weight parameter λ of the local distillation loss 1 :

[0023]

[0024] Among them, Loss r Represents the loss of the regression task in target detection; Loss d represents the loss of local distillation.

[0025] As a further improvement of the present invention, the method further includes: presetting a hyperparameter γ for reducing the loss of local features; the feature distillation loss function based on weighted mask The formula is:

[0026]

[0027] In the second neural network, the loss function is distilled based on the weighted mask-based features. Output the target image.

[0028] Compared with the prior art, the present invention has the following beneficial effects: by using the method of training the second neural network through the first neural network, the feature map with strong discrimination ability learned by the large network can be fully utilized under this framework to guide the learning of the small network; the global feature map and local area features between the first neural network and the second neural network can be effectively fitted at the same time, so that the distillation is more targeted and the convergence of the second neural network is accelerated; finally, the accuracy of the target detection classifier is improved while the computational complexity and occupancy rate are small. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the object detection distillation method guided by weighted mask according to the present invention.

[0030] Figure 2 It is a flow chart of the method for training the second neural network of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0032] In the first embodiment of the present invention, a method for target detection distillation based on weighted mask guidance is provided. Although the present application provides the following implementation manner or process Figure 1The method operation steps shown are based on routine or no creative labor, and the execution order of the steps in the method that do not have a necessary causal relationship in logic is not limited to the execution order provided in the implementation mode of this application. Figure 1 As shown in Figure 1, a weighted mask-guided object detection distillation method is applied to an object detection classifier, which is used for image recognition and classification. The steps are as follows:

[0033] S1: The second neural network is trained by the first neural network, wherein the first neural network is a large-capacity network and the second neural network is a small-capacity network; specifically, the second neural network can be trained by using knowledge distillation technology, wherein the first neural network can be regarded as a teacher network and the second neural network can be regarded as a student network.

[0034] Specifically, “training the second neural network” includes:

[0035] The first neural network is trained using a preset training set, wherein the preset training set is a plurality of images set in advance. The trained first neural network, i.e., the teacher network, focuses on all features of the image.

[0036] In the first neural network, the distillation temperature τ is used to output the probability distribution in the output layer; the output layer is represented as a SoftMax layer in the first neural network, and the probability distribution is represented as a soft-target, that is, a predicted value, in the first neural network.

[0037] During the distillation process of fitting the first neural network to the second neural network, the distillation temperature τ is adjusted; the second neural network is trained by adjusting the distillation temperature τ; τ is the temperature parameter in the neural network, when τ=1, it is the ordinary SoftMax formula; when τ→o, zi / τ→0, all probability values ​​pi will tend to be equal, that is, high temperature will eliminate the differences originally caused by different numerical values. By continuously adjusting the distillation temperature τ, the second neural network is trained to fit the first neural network, thereby continuously learning the first neural network.

[0038] After learning the first neural network by adjusting the distillation temperature τ as described above, the distillation temperature τ is controlled to be 1, and the detection method of the first neural network for the training set is calculated in the second neural network.

[0039] S2: Calculate the loss function when the second neural network fits the global features of the first neural network image. Specifically: "The loss function when the second neural network fits the global features of the first neural network image" is:

[0040]

[0041] in, is the loss function when the global features in the first neural network are fitted to the second neural network; ui is the feature map of the i-th prediction module in the first neural network; vi is the feature map corresponding to the i-th prediction module in the second neural network, r represents the dimension adjustment layer in the second neural network; M represents the number of feature maps of the prediction module that need to be distilled; is the weight parameter of the global distillation loss of each preset prediction module; 1 / 2N means that the number of filters in each layer of the second neural network is only 1 / 2N of the corresponding layer of the first neural network; i i Indicates the size of the spatial dimension in the feature map module; 1 / D i Indicates that the result of the feature map of i samples is normalized.

[0042] S3: Optimize the training of the second neural network for fitting local features through the predicted bounding box, the prior box and the ground truth annotation, and calculate the loss function of the second neural network when fitting local features, wherein the predicted bounding box is the prediction result output by the second neural network model, the prior box is a preset local feature box, and the ground truth annotation is supervision information.

[0043] Specifically, the method of "optimizing the training of the second neural network for fitting local features through the predicted bounding box, the prior box and the ground truth annotation" includes:

[0044] When fitting and training the second neural network by adjusting the distillation temperature τ, set the local feature weight parameter λ l ;

[0045] The second neural network outputs a predicted bounding box, which is compared with a preset priori box and proofread using the groundtruth annotation; in an embodiment of the present invention, the ground truth annotation is represented by self-supervised learning. The predicted bounding box is the prediction result output by the second neural network model, the priori box is a preset local feature box, and the ground truth annotation is the supervisory information; specifically, self-learning is: given a training set, and the priori box is marked by the thought, the prior box is annotated on the local features of the image in the learning of local features in this embodiment, that is, it can be understood as the "correct answer" marked by the thought, and in the learning process, the predicted value, i.e., the bounding box, is output from the second neural network, and the authenticity and accuracy of the prediction result is confirmed by the ground truth annotation.

[0046] In summary, based on the predicted bounding box, the prior box and the ground truth annotation, deep learning of local features is achieved in the second neural network fitting process.

[0047] Dynamically adjust the local feature weight parameter λ during training l By setting the weight parameter λ l , it can better identify local features and focus on more important features during self-supervised learning.

[0048] Furthermore, the “loss function when the second neural network fits local features” is:

[0049]

[0050] in, is the loss function when the local features in the first neural network are fitted to the second neural network; R represents the selected sample set; represents the regional features of the kth sample of the i-th prediction module in the first neural network; is the regional feature corresponding to the second neural network; r is the dimension adjustment layer; M represents the number of prediction module feature maps that need to be distilled; is the preset weight parameter of the local distillation loss of each prediction module; 1 / 2N means that the number of filters in each layer of the second neural network is only 1 / 2N of the corresponding layer of the first neural network; D () Indicates the size of the spatial dimension in the feature map module; 1 / D () Indicates that the results of the feature maps of k samples are normalized.

[0051] S4: A feature distillation loss function based on a weighted mask is formed in the second neural network according to the loss function of the global feature and the loss function of the local feature.

[0052] Furthermore, the calculation formula of the probability distribution soft-target is:

[0053]

[0054] Where zi represents the logical output of the i-th category; pi represents the corresponding probability; τ is the distillation temperature coefficient; and j represents the total number of categories.

[0055] In the above knowledge distillation process, the size of τ is adjusted to output different pi, and the second neural network learns to fit the dynamic process of the first neural network.

[0056] The temperature τ changes the degree of attention paid to negative labels during the training of the second neural network. When the temperature τ is low, less attention is paid to negative labels, especially those negative labels that are significantly lower than the average value; when the temperature τ is high, the value related to negative labels will increase relatively, and the second neural network model will pay more attention to negative labels.

[0057] Furthermore, it also includes dynamically fitting the first neural network by calculating the parameters of the second neural network when training the second neural network. Specifically: the parameter calculation formula of the second neural network is:

[0058]

[0059] in, is the parameter of the second neural network; H(y * ,P S ) is the cross entropy loss of the true label; H(P T τ ,P S τ ) is the cross entropy loss of the probability distribution of the first neural network and the second neural network; α is the parameter for adjusting the two cross entropy losses; is the probability distribution of the first neural network; is the probability distribution of the second neural network.

[0060] It should be noted that It can be understood as the loss function of the second neural network in predicting the true label. The loss function is used to dynamically calculate the parameters of each layer of the second neural network, so as to output the predicted value more accurately.

[0061] It should be further explained that the value of the distillation loss is usually related to the sampling strategy. Different strategies will result in different values ​​of the distillation loss. Too large a loss value will cause the network to diverge, while too small a loss value will result in insufficient network training. In order to prevent the problem of the loss value of the distillation item being too large or too small, and to ensure that the network does not diverge during the distillation process and can converge to the optimal value, it is often necessary to manually adjust the weight of the distillation loss to ensure that the value of the distillation loss is appropriate. However, manual adjustment is often cumbersome, and it is difficult to adjust to the optimal weight, so an adaptive dynamic adjustment method is adopted.

[0062] In the detection task, for the classification task, both positive and negative samples are used for training, while for the regression task, only positive samples are used for training. Since the negative samples are diverse, their changes in each training batch cannot be guaranteed. Therefore, the classification loss value is unstable relative to the regression loss value. In order to ensure the stability of the distillation loss value, the loss size of the regression task can be used as the target of the loss value generated by distillation. The details are as follows:

[0063] In one embodiment of the present invention, “dynamically adjusting the local feature weight parameter λ l " is: Use the loss size of the regression task in each optimization training and the local adjustment distillation loss size to adjust the weight parameter λ of the local distillation loss 1 :

[0064]

[0065] Among them, Loss r Represents the loss of the regression task in target detection; Loss d represents the loss of local distillation.

[0066] Therefore, the use of Realize adaptive adjustment of weight parameters.

[0067] In one embodiment of the present invention, it also includes a pre-set hyperparameter y for reducing the loss of local features; the feature distillation loss function based on weighted mask The formula is:

[0068]

[0069] In the second neural network, the loss function is distilled based on the weighted mask-based features. Output the target image. The target image focuses more on local features.

[0070] In summary, the present invention aims to solve the problem in the prior art that in the field of target detection, the knowledge distillation technology is usually used to fit a large network with a small network. However, the knowledge distillation method cannot distinguish between the fitting of global features and the fitting of local features during the fitting process, that is, it cannot automatically control the fitting attention of global features and the fitting attention of local features.

[0071] The present invention mainly adopts a weighted mask method. In the process of fitting the first neural network with the second neural network, the local features are weighted, and the weight parameters in the local feature loss function are dynamically controlled by learning; and the focus on global features is reduced, that is, the focus on image background information is reduced.

[0072] Through the above technical solution, the recognition technology of fitting a large network with a small network is used to better reduce memory usage, while reducing the attention to unnecessary features, improving recognition efficiency, and reducing the disadvantages of resource usage. The weight parameters can be dynamically adjusted according to the learning process, which can more effectively identify the required local features and accelerate the convergence of the small network, i.e., the second neural network; ultimately, the accuracy of the target detection classifier can be improved while reducing the amount of calculation and occupancy.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and modules can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0074] In several embodiments provided in the present application, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system implementation described above is only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, systems or modules, which can be electrical, mechanical or other forms. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0075] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of hardware plus software functional modules. The above-mentioned integrated module implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above-mentioned software functional modules are stored in a storage medium, including a number of instructions for a computer system (which can be a personal computer, a server, or a network system, etc.) or a processor (processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0076] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned implementation modes, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned implementation modes, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various implementation modes of the present application.

Claims

1. A weighted mask-guided object detection distillation method, characterized in that: Applied in a target detection classifier, the target detection distillation method based on weighted mask guidance includes: Training a second neural network through the first neural network, wherein the first neural network is a large-capacity network and the second neural network is a small-capacity network; Calculate the loss function when the second neural network fits the global features of the first neural network image; The second neural network is trained to fit the local features through the predicted bounding box, the priori box and the truth annotation, and the loss function of the second neural network when fitting the local features is calculated, wherein the predicted bounding box is the prediction result output by the second neural network model, the priori box is the preset local feature box, and the truth annotation is the supervision information; According to the loss function of the global feature and the loss function of the local feature, a feature distillation loss function based on the weighted mask is formed in the second neural network.

2. The object detection distillation method based on weighted mask guidance according to claim 1, characterized in that: The “training the second neural network” includes: Using a preset training set to train the first neural network; In the first neural network, a probability distribution is outputted in an output layer using a distillation temperature τ; The second neural network is trained by adjusting the distillation temperature τ; the distillation temperature τ is controlled to be 1, and the detection method of the first neural network for the training set is calculated in the second neural network.

3. The object detection distillation method based on weighted mask guidance according to claim 2, characterized in that: The calculation formula of the probability distribution is: Where zi represents the logical output of the i-th category; pi represents the corresponding probability; τ is the distillation temperature coefficient; and j represents the total number of categories.

4. The object detection distillation method based on weighted mask guidance according to claim 1, characterized in that: When training the second neural network, the first neural network is dynamically fitted by calculating the parameters of the second neural network.

5. The object detection distillation method based on weighted mask guidance according to claim 4, characterized in that: The parameter calculation formula of the second neural network is: in, is the parameter of the second neural network; H(y * ,P S ) is the cross entropy loss of the true label; is the cross entropy loss of the probability distribution of the first neural network and the second neural network; α is the parameter for adjusting the two cross entropy losses; is the probability distribution of the first neural network; is the probability distribution of the second neural network.

6. The object detection distillation method based on weighted mask guidance according to claim 1, characterized in that: The "loss function when the second neural network fits the global features of the first neural network image" is: in, is the loss function when the global features in the first neural network are fitted to the second neural network; ui is the feature map of the i-th prediction module in the first neural network; vi is the feature map corresponding to the i-th prediction module in the second neural network, r represents the dimension adjustment layer in the second neural network; M represents the number of feature maps of the prediction module that need to be distilled; is the weight parameter of the global distillation loss of each prediction module; 1 / 2N means that the number of filters in each layer of the second neural network is only 1 / 2N of the corresponding layer of the first neural network; i i Indicates the size of the spatial dimension in the feature map module; 1 / D i Indicates that the result of the feature map of i samples is normalized.

7. The object detection distillation method based on weighted mask guidance according to claim 6, characterized in that: The method of "optimizing the training of the second neural network for fitting local features by predicting the bounding box, the prior box and the truth annotation" includes: When fitting and training the second neural network by adjusting the distillation temperature τ, set the local feature weight parameter λ l ; The second neural network outputs the predicted bounding box, which is compared with the preset prior box and proofread using the truth annotation; Dynamically adjust the local feature weight parameter λ during training l .

8. The object detection distillation method based on weighted mask guidance according to claim 7, characterized in that: The "loss function when the second neural network fits local features" is: in, is the loss function when the local features in the first neural network are fitted to the second neural network; R represents the selected sample set; represents the regional features of the kth sample of the i-th prediction module in the first neural network; is the regional feature corresponding to the second neural network; r is the dimension adjustment layer; M represents the number of prediction module feature maps that need to be distilled; is the preset weight parameter of the local distillation loss of each prediction module; 1 / 2N means that the number of filters in each layer of the second neural network is only 1 / 2N of the corresponding layer of the first neural network; D () Indicates the size of the spatial dimension in the feature map module; 1 / D () Indicates that the results of the feature maps of k samples are normalized.

9. The object detection distillation method based on weighted mask guidance according to claim 7, characterized in that:The "dynamic adjustment of local feature weight parameter λ l " is: Use the loss size of the regression task in each optimization training and the local adjustment distillation loss size to adjust the weight parameter l of the local distillation loss λ: Among them, Loss r Represents the loss of the regression task in target detection; Loss d represents the loss of local distillation.

10. The object detection distillation method based on weighted mask guidance according to claim 8, characterized in that: It also includes a preset hyperparameter γ for reducing the loss of local features; the feature distillation loss function based on weighted mask The formula is: In the second neural network, the loss function is distilled based on the weighted mask-based features. Output the target image.

Images