Object detector for head detection

Abstract

This invention relates to a target detector for head detection, comprising: an input layer for inputting an image to be detected; an input end of a backbone network connected to the output end of the input layer for receiving the image to be detected and generating multi-scale features accordingly; an input end of an encoder connected to the output end of the backbone network for receiving the multi-scale features and generating a multi-scale feature map accordingly; input ends of an auxiliary detection head and a decoder detection head, both connected to the output end of the encoder; during training, the multi-scale feature map output by the encoder is input into the auxiliary detection head and the decoder detection head for joint training; the output end of the auxiliary detection head is also connected to the input end of the decoder detection head for training, wherein positive sample queries generated by the auxiliary detection head and contrast-denoised sample queries generated by ground truth bounding boxes are input into the decoder detection head for auxiliary training; and an output layer connected to the decoder detection head for outputting the head detection result.

Description

Technical Field

[0001] This invention relates to the field of target detection, and in particular to a target detector for human head detection. Background Technology

[0002] Head detection is a subtask within object detection, widely used in crowd counting, face tracking and recognition, video surveillance and security, autonomous driving, and intelligent transportation. Unlike face detection and pedestrian detection, head detection can detect heads facing away from the camera and is more effective than pedestrian detection in densely occluded scenes. Therefore, from both a practical application and technical perspective, head detection is a crucial and challenging object detection task.

[0003] Existing head detectors are primarily based on convolutional neural networks (CNNs). To address the issue of varying head sizes, they utilize backbone networks like ResNet50 to extract multi-scale features from images, and then employ networks such as Feature Pyramid Networks (FPNs) to fuse these multi-scale features, thereby detecting heads with varying scales on the multi-scale feature maps. To cope with interference from occlusion, blurring, and similar objects, these head convolutional detectors enhance the perception of global and local contextual information to distinguish heads from the background, especially backgrounds similar to heads, thus improving the accuracy of head detection. However, the limited receptive field of convolutional neural networks makes it difficult for them to capture global contextual information.

[0004] Convolutional object detection algorithms are weak in modeling global contextual relationships, which prevents the network from detecting objects based on global feature information. To address this issue, many works have applied Transformers to object detection. DETR treats object detection as a set prediction problem, using Transformer and bipartite graph matching loss to perform one-to-one matching between predicted and ground truth boxes, eliminating many handcrafted components of convolutional detectors, such as non-maximum suppression and anchor box generation. However, its performance on small object detection tasks is worse than convolutional detectors, and it requires nearly 500 epochs to converge. Subsequent works have improved upon DETR. Deformable DETR borrows the sparse spatial sampling of deformable convolutions and the relational modeling capabilities of Transformers, proposing a multi-scale deformable attention module for global sparse interaction of multi-scale feature information. Due to the fusion of multi-scale features, Deformable DETR achieves a significant improvement in small object detection performance compared to DETR. DAB-DETR uses four-dimensional anchor boxes as the target query in the decoder and dynamically updates them layer by layer, equivalent to performing soft ROI pooling. DN-DETR introduces denoising training, which accelerates model convergence, improves model detection accuracy, and stabilizes training. DINO further improves the model's convergence speed and small object detection accuracy by comparing denoising training and hybrid query selection. Co-DETR improves the encoder's discriminative feature learning ability in the end-to-end detector by training multiple parallel detection heads under one-to-many label assignment supervision, thereby enhancing object detection performance.

[0005] However, existing work mainly focuses on general target detection. How to design a specific target detector for small targets such as human heads is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a target detector for human head detection, comprising: an input layer, a backbone network, an encoder, an auxiliary detection head, a decoder detection head, and an output layer;

[0007] The input layer is used to input the image to be detected;

[0008] The input end of the backbone network is connected to the output end of the input layer to receive the image to be detected and generate multi-scale features accordingly.

[0009] The input end of the encoder is connected to the output end of the backbone network to receive multi-scale features and generate multi-scale feature maps accordingly.

[0010] Both the input terminals of the auxiliary detection head and the decoder detection head are connected to the output terminal of the encoder; during training, the multi-scale feature map output by the encoder is input into the auxiliary detection head and the decoder detection head for joint training.

[0011] The output of the auxiliary detection head is also connected to the input of the decoder detection head, so that during training, the positive sample query generated by the auxiliary detection head and the contrast-denoised sample query generated by the ground truth box are input into the decoder detection head for auxiliary training;

[0012] The output layer, connected to the decoder's detection head, is used to output the head detection results.

[0013] Furthermore, the encoder includes: N encoder layers and a feature fusion module; each encoder layer is connected to the feature fusion module to fuse the feature maps output by each encoder layer layer by layer, so as to obtain the multi-scale feature map output by the entire encoder.

[0014] Furthermore, each encoder layer includes: a multi-scale deformable attention module and a feedforward network.

[0015] Furthermore, the feature fusion module specifically assumes that the multi-scale features output by the backbone network are F. b The feature map output by the i-th encoder layer is F i The feature map of a single fusion is denoted as F. f i Then it is represented as:

[0016]

[0017] Finally, the feature map F output by the last layer of the feature fusion module is used. f N This is the multi-scale feature map that is the final output of the encoder.

[0018] Furthermore, the auxiliary detection heads include a dynamic region convolution detection head and a zoom network detection head, the input ends of which are connected to the output end of the encoder, and the output ends of which are connected to the input end of the decoder detection head.

[0019] Furthermore, the encoder outputs a multi-scale feature map with 5 layers. A 3x3 convolution operation is used to downsample the 64x downsampled feature layer to generate a multi-scale feature map with 6 layers, which serves as the feature input for the dynamic region convolutional detection head and the zoom network detection head.

[0020] Furthermore, the dynamic region convolutional detection head includes an RPN head, an RoI pooling layer, and a classification and regression head connected in sequence. The multi-scale feature map is processed by the RPN head to classify the pre-defined anchor boxes to determine positive and negative samples, and at the same time, the detection box offset is regressed to obtain the proposal boxes. The feature map output by the encoder and the proposal boxes output by the RPN head are fed into the RoI pooling layer to generate a fixed 7x7 size proposal box feature map, which is then classified and regressed by the classification and regression head formed by the fully connected layer.

[0021] Furthermore, the zoom network detection head includes a classification branch, a detection box regression branch, and a detection box refinement branch; the multi-scale feature map output by the encoder is used to predict the four-dimensional distance vector through the detection box regression branch; the distance scaling factor is predicted through the detection box refinement branch; the category is predicted through the classification branch, and an IoU-aware classification loss function is used to ensure that the training is focused on high-quality positive samples.

[0022] Furthermore, the decoder detection head uses a GIoU-aware classification loss function during training:

[0023]

[0024] Where α and γ are hyperparameters, p t Let represent the predicted probability of the t-th query, BCE represent the binary cross-entropy loss, and q be the coefficient of GIoU between 0 and 1.

[0025] Furthermore,

[0026]

[0027] Where *neg* indicates that the t-th predicted box is a negative sample, and *pos* indicates that the t-th predicted box is a positive sample.

[0028] This invention relates to a target detector for head detection, which is an end-to-end target detector. Its key difference from existing technologies is that traditional head detection, due to the one-to-one matching between the target query and the ground truth bounding box in the decoder, compared with the one-to-many matching of convolutional detectors, does not effectively activate the feature map output by the encoder, making it more prone to missed detections and false detections. At the same time, it has been found that small targets are more sensitive to the accuracy of the predicted bounding box than large targets. Even a slight deviation in the predicted bounding box will significantly reduce the IoU. Therefore, a more accurate predicted detection bounding box is very helpful in improving the performance of small target detection. Therefore, this invention takes generating prediction boxes with high classification confidence and high localization quality as its starting point. During training, an auxiliary detection head that is more effective for detecting small objects is selected to enrich the label assignment paradigm, effectively supervise the encoder output, and better activate the features of the head region. The auxiliary detection head and the decoder detection head are jointly trained, and the positive sample queries of the auxiliary detection head and the contrast-denoised sample queries generated by the ground truth boxes are input into the decoder detection head for auxiliary training. This accelerates the convergence speed in the early stage of model training, improves the detection performance of the head detector, and reduces the occurrence of false detections and false negatives during the inference process. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of an embodiment of the target detector for human head detection according to the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0031] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of the present invention involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the order of method execution. Those skilled in the art will understand that anything that does not violate the inventive concept and is within the scope of the present invention should be included in the protection scope of the present invention.

[0032] like Figure 1 As shown, the present invention provides a target detector for human head detection, comprising: an input layer, a backbone network, an encoder, an auxiliary detection head, a decoder detection head, and an output layer;

[0033] The input layer is used to input the image to be detected;

[0034] The input end of the backbone network is connected to the output end of the input layer to receive the image to be detected and generate multi-scale features accordingly.

[0035] The input end of the encoder is connected to the output end of the backbone network to receive multi-scale features and generate multi-scale feature maps accordingly.

[0036] Both the input terminals of the auxiliary detection head and the decoder detection head are connected to the output terminal of the encoder; during training, the multi-scale feature map output by the encoder is input into the auxiliary detection head and the decoder detection head for joint training.

[0037] The output of the auxiliary detection head is also connected to the input of the decoder detection head, so that during training, the positive sample query generated by the auxiliary detection head and the contrast-denoised sample query generated by the ground truth box are input into the decoder detection head for auxiliary training;

[0038] The output layer, connected to the decoder's detection head, is used to output the head detection results.

[0039] In this embodiment, a head detection target detector of the present invention is presented. It is an end-to-end target detector. Its key difference from the prior art is that, in traditional head detection, the target query in the decoder is matched one-to-one with the ground truth box. Compared with the one-to-many matching of the convolutional detector, the feature map output by the encoder is not effectively activated, which makes it more prone to missed detections and false detections. At the same time, it was found that small targets are more sensitive to the accuracy of the predicted box than large targets. Even a slight deviation in the predicted box will significantly reduce the IoU. Therefore, a more accurate predicted detection box is very helpful in improving the performance of small target detection. Therefore, this invention takes generating prediction boxes with high classification confidence and high localization quality as its starting point. During training, an auxiliary detection head that is more effective for detecting small objects is selected and connected to the encoder and decoder detection heads to enrich the label allocation paradigm, effectively supervise the encoder output, and better activate the features of the head region. The auxiliary detection head and decoder detection head are used for joint training, and the positive sample queries from the auxiliary detection head and the contrast-denoised sample queries generated by the ground truth boxes are input into the decoder detection head for auxiliary training to accelerate the convergence speed in the early stage of model training, improve the detection performance of the head detector, and reduce the occurrence of false detections and false negatives during the inference process.

[0040] Preferred:

[0041] A: Input layer:

[0042] Specifically, the input image to be detected can be, but is not limited to, any scenario requiring head detection, such as images or video data from scenarios like crowd counting, face tracking and recognition, video surveillance and security, autonomous driving, and intelligent transportation. For example,... Figure 1 The input image shown.

[0043] B: Backbone Network

[0044] Specifically, the backbone network can employ any existing backbone network module to extract multi-scale features from the image to be detected. For example, ... Figure 1 As shown, the use of the SwinL backbone network is optional, but not limited to, outputting multi-scale features of S1-S5.

[0045] C: Encoder:

[0046] Specifically, the encoder may optionally include, but is not limited to, N encoder layers and a feature fusion module; each encoder layer is connected to the feature fusion module to fuse the feature maps output by each encoder layer layer by layer, as the multi-scale feature map output by the entire encoder.

[0047] Preferably, each encoder layer may optionally include, but is not limited to, a multi-scale deformable attention module and a feedforward network;

[0048] More preferably, the feature fusion module may be represented as, but is not limited to, as:

[0049] Assume the multi-scale features output by the backbone network are F b The feature map output by the i-th encoder layer is F i The feature map of a single fusion is denoted as F. f i Then, it can be, but is not limited to, represented as:

[0050]

[0051] Finally, the feature map F output by the last layer of the feature fusion module is used. f N The multi-scale feature map F is the final output of the encoder. e .

[0052] For example, such as Figure 1 As shown, the encoder in the model consists of 6 stacked encoder layers. After the first encoder layer outputs F1, the feature fusion module uses F... b Combined with F1 to obtain F f 1After the second encoder layer outputs F2, the feature fusion module uses F2 and F... f 1 F is obtained by fusion f 2 After the third encoder layer outputs F3, the feature fusion module uses F3 and F... f 2 F is obtained by fusion f 3 ...After the 6th encoder layer outputs F6, the feature fusion module uses F6 and F... f 5 F is obtained by fusion f 6 The feature map F output by the last layer of the feature fusion module f 6 The feature map F, which is the final output of the encoder e .

[0053] This embodiment presents a preferred embodiment of the encoder of the present invention. As features interact within the encoder layers, the semantic information of multi-scale features becomes richer layer by layer, but spatial detail information is easily lost, which can affect localization accuracy. Therefore, the present invention adds a feature map fusion branch, namely a feature fusion module, to the multi-layer encoder layers. This module fuses the feature maps output from each encoder layer together as the feature map output of the entire encoder layer. Since the feature fusion module allows the output of shallow encoder layers to be directly passed to the output of the last encoder layer, it provides richer feature representation capabilities, mitigates information loss during feature map transmission layer by layer in the encoder, further improves the stability of model training, and accelerates model convergence.

[0054] D: Auxiliary detection head

[0055] Specifically, the auxiliary detection heads may include, but are not limited to, the Dynamic R-CNN detection head and the Varifocalnet detection head; the encoder outputs a multi-scale feature map with 5 layers, and a 3x3 convolution operation is used to downsample the 64x downsampled feature layer to generate a multi-scale feature map F with 6 layers, which serves as the feature input for the two auxiliary detection heads.

[0056] More preferably, the Dynamic R-CNN detection head may optionally include, but is not limited to, an RPN head, a RoI pooling layer, and a classification and regression head connected in sequence; for example, it consists of a standard RPN head (Region Proposal Network), a RoI pooling layer, and a classification and regression head.

[0057] This embodiment presents a preferred implementation of the Dynamic R-CNN detection head. Multi-scale feature maps are passed through a standard RPN head to classify pre-defined anchor boxes to determine positive and negative samples. Simultaneously, regression of the detection box offsets is performed to obtain accurate proposals. The feature map F output by the encoder and the proposals output by the RPN head are fed into the RoI pooling layer to generate a fixed 7x7 size proposal feature map, which is then classified and regressed using a classification and regression head formed by fully connected layers. Simultaneously, sampled positive samples are fed into the decoder for training.

[0058] Unlike Faster R-CNN, the Dynamic R-CNN detector head automatically adjusts the IoU threshold based on the statistical data of the proposed bounding boxes during training. This adapts to the IoU threshold size requirements at different training stages, helping the detector generate more high-quality predicted bounding boxes. Preferably, during training, this invention uses the bounding box loss of the Dynamic R-CNN detector head, optionally but not limited to FocalLoss, to improve training stability.

[0059] More preferably, the VarifocalNet detector head may optionally include, but is not limited to, a classification branch, a bounding box regression branch, and a bounding box refinement branch. The multi-scale feature map F output by the encoder is used by the bounding box regression branch to predict the four-dimensional distance vector; by the bounding box refinement branch to predict the distance scaling factor to further improve localization accuracy; and by the classification branch to predict the category, using an IoU-aware classification loss function, so that training is focused on high-quality positive samples, prompting the detector to predict bounding boxes with high confidence and high localization quality. More preferably, but not limited to, based on the FCOS architecture, positive and negative samples are defined using ATSS, and training is performed using IoU-aware classification scores.

[0060] In this embodiment, several preferred embodiments of the auxiliary detection head are given. In addition to being supervised by classification and localization loss functions when making predictions, it also generates a certain number of positive queries, which are sent to the decoder for prediction, increasing the weight of positive samples to alleviate the imbalance between the number of positive and negative samples.

[0061] E: Decoder detection head

[0062] Specifically, the decoder detection head may optionally include, but is not limited to, N decoder layers; preferably, it may also optionally include, but is not limited to, a query selection module to select the top 90% confidence positive queries and send them to the decoder for prediction.

[0063] Preferably, during training, a DETR-based model, such as DINO, is used, and FocalLoss is used as the classification loss function to optimize the model. The expression for FocalLoss is as follows:

[0064]

[0065] Where p t Let represent the predicted probability of the t-th query, q represent whether the category is correct, BCE represent the binary cross-entropy loss, and γ is a hyperparameter, which is usually set to 2.

[0066] However, due to the inconsistency between the distributions of classification confidence and localization accuracy, in the early stages of training, it is easy to encounter situations where predicted boxes with high classification confidence have poor localization quality, or predicted boxes with high localization quality have low classification confidence. The former leads to inaccurate object locations detected during inference, while the latter causes high-quality detection boxes to be filtered out by the confidence threshold during inference, resulting in missed detections. To alleviate this inconsistency between the distributions of classification and localization confidence, this invention proposes a GIoU-aware classification loss function to better reflect the overlap between predicted and ground truth boxes and achieve better supervision:

[0067]

[0068] Where α and γ are hyperparameters, which can be, but are not limited to, 0.75 and 2 respectively, and q is an expression for GIoU:

[0069]

[0070] When the t-th predicted bounding box is a negative sample, q is 0. When the t-th predicted bounding box matches the ground truth bounding box, q is a coefficient of GIoU between 0 and 1. In this case, the larger the q of the positive sample predicted bounding box, the greater its loss weight, and the optimization focuses on locating more accurate predicted bounding boxes.

[0071] In this embodiment, several preferred embodiments of the decoder detection head of the present invention are given, and its own structure and loss function during training are further improved.

[0072] In summary, the beneficial effects of the present invention include at least the following:

[0073] (1) Two high-quality auxiliary detection heads were introduced to help the model learn more accurate classification and precise localization feature information;

[0074] (2) A GIoU-aware classification loss function was designed to encourage more accurate detection boxes to have higher classification confidence.

[0075] (3) A feature fusion module was introduced into the encoder to alleviate the loss of information in the deep network and accelerate the convergence of the model.

[0076] (4) Applying the visual Transformer model to the head detection task improved the AP50 and AP75 performance metrics on the Brainwash dataset.

[0077] On the other hand, the present invention provides a detection method for head detection, comprising:

[0078] Construct and train any of the above target detectors;

[0079] The system acquires the current image to be detected, inputs it into the target detector, and outputs the head detection results.

[0080] The above detection method is based on the above target detector, and the combination of its technical features and technical effects will not be elaborated here. The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A target detector oriented to human head detection, characterized in that, include: Input layer, backbone network, encoder, auxiliary detection head, decoder detection head, and output layer; The input layer is used to input the image to be detected; The input end of the backbone network is connected to the output end of the input layer to receive the image to be detected and generate multi-scale features accordingly. The input end of the encoder is connected to the output end of the backbone network to receive multi-scale features and generate multi-scale feature maps accordingly. Both the input terminals of the auxiliary detection head and the decoder detection head are connected to the output terminal of the encoder; during training, the multi-scale feature map output by the encoder is input into the auxiliary detection head and the decoder detection head for joint training. The output of the auxiliary detection head is also connected to the input of the decoder detection head, so that during training, the positive sample query generated by the auxiliary detection head and the contrast-denoised sample query generated by the ground truth box are input into the decoder detection head for auxiliary training; The output layer, connected to the decoder detection head, is used to output the head detection results; The auxiliary detection heads include a dynamic region convolution detection head and a zoom network detection head, the input ends of which are connected to the output end of the encoder, and the output ends of which are connected to the input end of the decoder detection head. The dynamic region convolutional detection head includes a sequentially connected RPN head, RoI pooling layer, and classification and regression heads. Multi-scale feature maps are passed through the RPN head to classify pre-defined anchor boxes to determine positive and negative samples, while simultaneously regressing the detection box offsets to obtain proposal boxes. The feature maps output by the encoder and the proposal boxes output by the RPN head are fed into the RoI pooling layer to generate a fixed 7x7 size proposal box feature map, which is then classified and regressed by the classification and regression heads formed by fully connected layers.

2. The object detector of claim 1, wherein, The encoder includes: N encoder layers and a feature fusion module; each encoder layer is connected to the feature fusion module to fuse the feature maps output by each encoder layer layer by layer, so as to form a multi-scale feature map output by the entire encoder.

3. The object detector of claim 2, wherein, Each encoder layer includes: a multi-scale deformable attention module and a feedforward network.

4. The target detector according to claim 2, characterized in that, The feature fusion module specifically is: assuming that multi-scale features output by a backbone network are , a feature map output by an i-th layer encoder layer is , and a single fused feature map is denoted as , and is represented as: ， Finally, the feature map output by the last layer of the feature fusion module is a multi-scale feature map output by the encoder.

5. The target detector according to claim 1, characterized in that, The encoder outputs a multi-scale feature map with 5 layers. A 3x3 convolution operation is used to downsample the 64x downsampled feature layer to generate a multi-scale feature map with 6 layers, which serves as the feature input for the dynamic region convolutional detection head and the zoom network detection head.

6. The target detector according to claim 1, characterized in that, The zoom network detection head includes a classification branch, a bounding box regression branch, and a bounding box refinement branch; the multi-scale feature map output by the encoder is used by the bounding box regression branch to predict the four-dimensional distance vector; The distance scaling factor is predicted by refining the branches of the detection box; The classification branch is used to predict the class, and the GIoU-aware classification loss function is used to focus the training on high-quality positive samples.

7. The target detector according to any one of claims 1-6, characterized in that, The decoder detection head uses a GIoU-aware classification loss function during training. ， where a and y are hyperparameters, p t denotes the predicted probability of the t-th query, BCE denotes the binary cross-entropy loss, and q is a coefficient between 0 and 1 with respect to GIoU.

8. The target detector according to claim 7, characterized in that, ， Where *neg* indicates that the t-th predicted box is a negative sample, and *pos* indicates that the t-th predicted box is a positive sample.

Images