Person detection method and system based on transfer learning

Abstract

The present invention proposes a person detection method and system based on transfer learning, constructing a teacher network for image target detection and its corresponding student network, obtaining a picture data set marked with person category labels as a training data set; according to the amount of target object information contained in the features of each layer of target detection FPN of the teacher network and the student network, respectively, the teacher feature richness score of each layer of target detection FPN of the teacher network and the student feature richness score of each layer of target detection FPN of the student network are obtained; based on the teacher feature richness score and the student feature richness score, the classification head constraint loss is obtained; the loss of the teacher network, the loss of the student network and the classification head constraint loss are summed to obtain the distillation loss; after the distillation loss converges, the student network is used to detect people in the picture.

Description

[0001] This application is a divisional application of the invention patent application with application date of September 18, 2021, application number 202111103913.2, and invention name “Person detection method and system based on transfer learning”. Technical Field

[0002] The present invention relates to the technical field of knowledge distillation in target detection and transfer learning, and in particular to a person detection method, system, storage medium and client based on transfer learning. Background Art

[0003] Large-scale deep models have achieved tremendous success in recent years, but their enormous computational complexity and massive storage requirements make their deployment on resource-constrained devices a significant challenge. Knowledge distillation, a model compression and acceleration method, effectively improves the performance of small models by transferring dark knowledge from the teacher detector—information implicit in the teacher network that is useful to the student network. Most existing object detection methods based on knowledge distillation primarily task the student network with mimicking features from the teacher network that overlap with the bounding box, assuming that foreground features selected from the bounding box are important.

[0004] Existing knowledge distillation methods for object detection place too much emphasis on the information in the bounding box area, while ignoring the useful information in other areas. First, the foreground features selected from the bounding box only contain the categories in the dataset, while ignoring the categories of objects outside the dataset, resulting in the omission of some important features. For example, the COCO dataset does not contain a human model category, but does contain a person category. Since models are visually similar to humans, the features of human models contain many useful human features, which are conducive to improving the detection effect of distillation detectors on humans. Secondly, using only the prior knowledge of the bounding box to select features for distillation ignores the defects of the teacher network. Imitating features that are mistakenly identified as background by the teacher network will mislead the student network. Summary of the Invention

[0005] This paper proposes a person detection method based on transfer learning, which includes:

[0006] Step 1: Build a teacher network and its corresponding student network for image object detection, and obtain a dataset of images with labeled person categories as a training dataset;

[0007] Step 2: According to the target object information amount contained in the features of each layer of target detection FPN of the teacher network and the student network, the teacher feature richness score of each layer of target detection FPN of the teacher network and the student feature richness score of each layer of target detection FPN of the student network are obtained;

[0008] Step 3: Based on the teacher's feature richness score and the student's feature richness score, obtain the classification head constraint loss;

[0009] Step 4: Sum the loss of the teacher network, the loss of the student network, and the classification head constraint loss to obtain the distillation loss.

[0010] Step 5: Loop step 2 to step 4 until the distillation loss converges or reaches the preset number of iterations, terminate the training, use the current student network as a person detection model, and use the person detection model to detect the person in the image to be identified, and obtain the person category of the person in the image to be identified as the recognition result.

[0011] The person detection method based on transfer learning, wherein the classification head constrains the loss:

[0012]

[0013] Where M represents the number of FPN layers, W and H represent the width and height of the feature map, They represent the teacher feature richness score and student feature richness score corresponding to the lth layer of the teacher network and the student network respectively, and φ is the binary cross entropy function.

[0014] The present invention also proposes a person detection system based on feature richness knowledge distillation, which includes:

[0015] Module 1 is used to build a teacher network and its corresponding student network for image object detection, and obtain a dataset of images with labeled person categories as a training dataset;

[0016] Module 2 is used to obtain the teacher feature richness score of each layer of target detection FPN of the teacher network and the student feature richness score of each layer of target detection FPN of the student network according to the amount of target object information contained in the features of each layer of target detection FPN of the teacher network and the student network respectively;

[0017] Module 3 is used to obtain the classification head constraint loss based on the teacher's feature richness score and the student's feature richness score;

[0018] Module 4 is used to sum the loss of the teacher network, the loss of the student network and the classification head constraint loss to obtain the distillation loss;

[0019] Module 5 is used to repeatedly iteratively call module 2 to module 4 until the distillation loss converges or reaches a preset number of repeated iterations, terminate the training, use the current student network as a person detection model, and use the person detection model to detect the person in the image to be identified, and obtain the person category of the person in the image to be identified as the recognition result.

[0020] The person detection system based on transfer learning, wherein the classification head constrains the loss:

[0021]

[0022] Where M represents the number of FPN layers, W and H represent the width and height of the feature map, They represent the teacher feature richness score and student feature richness score corresponding to the lth layer of the teacher network and the student network respectively, and φ is the binary cross entropy function.

[0023] The present invention also proposes a storage medium for storing a program for executing the person detection method based on transfer learning.

[0024] The present invention also proposes a client for use in the transfer learning-based person detection system.

[0025] From the above scheme, it can be seen that the advantage of the present invention is that it improves the accuracy of target detection. When the evaluation indicator is mAP on the COCO dataset: Faster-Resnet50 increases by 2.1%, FCOS-Resnet50 increases by 2.4%, and GFL-Resnet50 increases by 3.4%. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a system block diagram of the present invention;

[0027] Figure 2 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0028] To address the above issues, this paper proposes a feature richness scoring (FRS) method to select important features that are beneficial for distillation. Feature richness refers to the amount of information about an object contained in a feature, and can also be expressed as the probability that these features are objects. Extracting features with high feature richness rather than features in the bounding box area can effectively address the two limitations mentioned above: ignoring features of objects outside the bounding box that are not included in the dataset category; and over-emphasizing features that are misclassified by the teacher detector.

[0029] First, the features of objects not included in the dataset categories have high feature richness. Therefore, using feature richness, we can retrieve important features outside the bounding box, which can guide the student network to learn the generalized detectability of the teacher network. For example, the features of human models with high feature richness can promote the student detector to improve its generalized detectability of people.

[0030] Secondly, the features in the bounding box but misclassified by the teacher detector have low feature richness. Therefore, using feature richness can remove the misleading features of the teacher detector in the bounding box.

[0031] Therefore, the importance of a feature is closely related to its richness, i.e., feature richness is suitable for selecting important features for distillation. Since the aggregation of classification scores of all categories is an approximation of the probability that a feature is an object, the present invention uses the aggregated classification score as the criterion for feature richness.

[0032] In practice, the present invention utilizes the classification score corresponding to each FPN layer in the teacher network as a feature mask, which is used as a feature richness map to guide the learning of the student network.

[0033] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0034] 1. Feature mask matrix S:

[0035] Based on the classification branch of the target detection FPN in each layer of the teacher network, a four-dimensional matrix (NCHW) is obtained. The matrix is ​​then summed along the C channel direction to obtain a feature matrix (NHW). This matrix is ​​then used as the feature mask matrix S. When the student network is ResNet50, the teacher network can be a larger network such as ResNet101 or Resnext101. In NCHW, N represents the size of a batch of images, for example, how many images are processed at a time. C represents the channel dimension of the image, and HW represents the height and width of the image or feature map, respectively.

[0036]

[0037] y t is the output of the classification branch of the teacher network, c' refers to the channel, l is the number of layers of the corresponding FPN, and t refers to the teacher network.

[0038] 2. Constrained Loss:

[0039] 2.1. Feature map constraint Loss:

[0040]

[0041] M represents the number of FPN layers, W, H represent the width and height of the feature map, l is the number of layers corresponding to FPN, i, j represent a specific point on the feature map, F lijc' It means that different weights are given to each feature in the HW dimension of each FPN layer through the feature mask; the teacher network is more complex than the student network. The main purpose of this invention is to improve the detection performance of the student network through knowledge distillation. The number of FPN layers of the two is the same. Represent the FPN feature maps corresponding to the Lth layer of the teacher network and the student network respectively. adapt This paper adopts feature map constraint loss to make full use of the feature mask corresponding to each FPN layer in the teacher network, and guides the learning of the student network more efficiently.

[0042] 2.2. Classification Head Constrained Loss

[0043]

[0044] M represents the number of FPN layers, W and H represent the width and height of the feature map, Represent the teacher and student feature richness scores for the teacher and student networks at layer l, respectively. The classification branch scores are based on existing techniques and are used in image detection to determine the accuracy of the classification. φ is a binary cross-entropy function. This paper employs a classification head constraint loss to fully utilize the information (richness) of the target object in the features, promoting the student network to improve its generalized detectability of the target.

[0045] In specific applications, based on the above content:

[0046] Step 1: Get the COCO dataset training data.

[0047] The second step is to build a teacher network and train it using the COCO dataset.

[0048] The third step is to build the student network. Based on the original training loss, the corresponding distillation loss (feature map constraint loss and / or classification head constraint loss) is added to all FPN (feature pyramid network) layers. The network is then trained on the COCO dataset.

[0049] The last step is to use the trained student network to detect the people in the picture.

[0050] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0051] The present invention also proposes a person detection system based on transfer learning, which is characterized by comprising:

[0052] Module 1 is used to build a teacher network and its corresponding student network for image object detection, and obtain a dataset of images with labeled person categories as a training dataset;

[0053] Module 2 is used to extract the classification branch output of each layer of target detection FPN of the teacher network when training the teacher network and the student network with the training data set, and obtain a four-dimensional matrix including the output results of the four channels of NCHW;

[0054] Module 3 is used to obtain the NHW dimension feature matrix as the feature mask matrix by summing the C channel direction of the four-dimensional matrix, and obtain the feature map constraint loss based on the feature mask matrix, the FPN feature map of the teacher network and the student network;

[0055] Module 4 is used to sum the loss of the teacher network, the loss of the student network and the feature map constraint loss to obtain the distillation loss;

[0056] Module 5: Repeat the iterations of Module 2 to Module 4 until the distillation loss converges or reaches a preset number of iterations, terminate the training, use the current student network as a person detection model, and use the person detection model to detect people in the picture.

[0057] The person detection system based on transfer learning is characterized in that:

[0058] The module 2 includes: obtaining a teacher feature richness score of each layer of target detection FPN of the teacher network and a student feature richness score of each layer of target detection FPN of the student network according to the amount of target object information contained in the features of each layer of target detection FPN of the teacher network and the student network respectively;

[0059] The module 3 includes: obtaining a classification head constraint loss based on the teacher's feature richness score and the student's feature richness score;

[0060] Module 4 includes: summing the loss of the teacher network, the loss of the student network, the feature map constraint loss, and the classification head constraint loss to obtain the distillation loss.

[0061] The person detection system based on transfer learning is characterized in that the feature map constraint loss is:

[0062]

[0063] Where M represents the total number of FPN layers, W, H represent the width and height of the feature map respectively, l is the number of layers corresponding to FPN, and i, j represent the coordinates of the points on the feature map; Represent the FPN feature maps corresponding to the Lth layer of the teacher network and the student network, φ adapt is the adaptive convolution layer.

[0064] The person detection system based on transfer learning is characterized in that the classification head constrains the loss:

[0065]

[0066] Where M represents the number of FPN layers, W and H represent the width and height of the feature map, They represent the teacher feature richness score and student feature richness score corresponding to the lth layer of the teacher network and the student network respectively, and φ is the binary cross entropy function.

[0067] The present invention also proposes another person detection system based on feature richness knowledge distillation. The difference between this system and the above-mentioned person detection system is that the above-mentioned embodiment uses feature map constraint loss, feature map constraint loss, and classification head constraint loss for training, while this embodiment only uses the classification head constraint loss. Specifically, it includes:

[0068] Module 1 is used to build a teacher network and its corresponding student network for image object detection, and obtain a dataset of images with labeled person categories as a training dataset;

[0069] Module 2 is used to obtain the teacher feature richness score of each layer of target detection FPN of the teacher network and the student feature richness score of each layer of target detection FPN of the student network according to the amount of target object information contained in the features of each layer of target detection FPN of the teacher network and the student network respectively;

[0070] Module 3 is used to obtain the classification head constraint loss based on the teacher's feature richness score and the student's feature richness score;

[0071] Module 4 is used to sum the loss of the teacher network, the loss of the student network and the classification head constraint loss to obtain the distillation loss;

[0072] Module 5 is used to repeatedly iteratively call module 2 to module 4 until the distillation loss converges or reaches a preset number of repeated iterations, terminate the training, use the current student network as a person detection model, and use the person detection model to detect the person in the image to be identified, and obtain the person category of the person in the image to be identified as the recognition result.

[0073] The present invention also proposes a client for use in the above-mentioned person detection system based on transfer learning.

[0074] The present invention also proposes a storage medium for storing a program for executing the transfer learning-based person detection.

Claims

1. A person detection method based on transfer learning, characterized in that: include: Step 1: Build a teacher network and its corresponding student network for image object detection, and obtain a dataset of images with labeled person categories as a training dataset; Step 2: According to the target object information amount contained in the features of each layer of target detection FPN of the teacher network and the student network, the teacher feature richness score of each layer of target detection FPN of the teacher network and the student feature richness score of each layer of target detection FPN of the student network are obtained; Step 3: Based on the teacher's feature richness score and the student's feature richness score, obtain the classification head constraint loss; Step 4: Sum the loss of the teacher network, the loss of the student network, and the classification head constraint loss to obtain the distillation loss. Step 5: Loop step 2 to step 4 until the distillation loss converges or reaches a preset number of iterations, terminate the training, use the current student network as a person detection model, and use the person detection model to detect the person in the image to be identified, and obtain the person category of the person in the image to be identified as the recognition result; The classification head constrains the loss: Where M represents the number of FPN layers, W and H represent the width and height of the feature map, They represent the teacher feature richness score and student feature richness score corresponding to the lth layer of the teacher network and the student network respectively, φ is the binary cross entropy function, and C represents the channel dimension of the image.

2. A person detection system based on transfer learning, characterized in that: include: Module 1 is used to build a teacher network and its corresponding student network for image object detection, and obtain a dataset of images with labeled person categories as a training dataset; Module 2 is used to obtain the teacher feature richness score of each layer of target detection FPN of the teacher network and the student feature richness score of each layer of target detection FPN of the student network according to the amount of target object information contained in the features of each layer of target detection FPN of the teacher network and the student network respectively; Module 3 is used to obtain the classification head constraint loss based on the teacher's feature richness score and the student's feature richness score; Module 4 is used to sum the loss of the teacher network, the loss of the student network and the classification head constraint loss to obtain the distillation loss; Module 5 is used to repeatedly iteratively call module 2 to module 4 until the distillation loss converges or reaches a preset number of iterations, terminate the training, use the current student network as a person detection model, and use the person detection model to detect the person in the image to be identified, and obtain the person category of the person in the image to be identified as the recognition result; The classification head constrains the loss: Where M represents the number of FPN layers, W and H represent the width and height of the feature map, They represent the teacher feature richness score and student feature richness score corresponding to the lth layer of the teacher network and the student network respectively, φ is the binary cross entropy function, and C represents the channel dimension of the image.

3. A storage medium for storing a program for executing the person detection method based on transfer learning according to claim 1.

4. A client, used in the person detection system based on transfer learning as described in claim 2.

Images