End-to-end target detection method and system based on dynamic hybrid label distribution
By using a dynamic hybrid label allocation method that combines one-to-many and one-to-one label allocation rules, the dependency and redundant detection problems of NMS in existing technologies are solved, achieving robust feature representation and efficient end-to-end detection results.
Patent Information
- Application Number
- CN202411154081.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-21
AI Technical Summary
Existing object detection methods rely on manually defined non-maximum suppression (NMS) steps during training, which makes them sensitive to hyperparameters and prone to missing complex overlapping targets. Furthermore, one-to-many label assignment rules lead to redundant detection, making it impossible to achieve efficient end-to-end detection.
A dynamic hybrid label allocation method is adopted, which combines one-to-many and one-to-one label allocation rules. Through a hybrid sample selection module and a stage-aware soft weight adjustment module, the contribution of foreground samples in the training process is dynamically adjusted to achieve robust feature representation and efficient end-to-end detection.
During training, robust feature representations are learned, and efficient end-to-end detection is achieved during inference, improving detection accuracy and robustness.
Smart Images

Figure CN118968034A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an end-to-end target detection method and system based on dynamic hybrid label allocation. Background Art
[0002] Object detection is a long-standing task in the field of computer vision, which aims to detect the category and location of predetermined objects in images. It has broad application prospects in many fields such as autonomous driving, video surveillance, face recognition, and medical image analysis. The current mainstream detectors predict objects with different scales, aspect ratios, and categories at each feature location in a regular dense sampling manner. These dense detectors are inseparable from an artificially defined component, namely non-maximum suppression (NMS), to eliminate duplicate predictions caused by dense detection in the post-processing step. However, this process makes these detectors sensitive to the hyperparameters of NMS and may lead to suboptimal solutions. In addition, in the face of complex dense overlapping objects, the detector can easily lose the target and cause missed detection. The existence of these problems prompted researchers to remove NMS to achieve end-to-end detection.
[0003] In early studies, some recurrent neural network (RNN) frameworks with long short-term memory (LSTM) units decoded image content into a set of variable-length prediction boxes. Although they achieved end-to-end detection, they only demonstrated their effectiveness on limited small datasets. CenterNet (X. Zhou, D. Wang, and P. Krahenbuhl, "Objects as points," arXiv preprint arXiv:1904.07850, 2019.) and its predecessor (H. Law and J. Deng, "Cornernet: Detecting objects as paired keypoints," in Proc. Eur. Conf. Comp. Vis., 2018, pp. 734–750.) achieved differentiable end-to-end training by modeling the ground truth as a heat map and converting object detection into a keypoint estimation problem. However, it requires more training time than the baseline detector. Recently, Hu et al. (H.Hu, J.Gu, Z.Zhang, J.Dai, and Y.Wei, "Relation networks for object detection," in Proc.IEEE Conf.Comp.Vis.Patt.Recogn., 2018, pp.3588–3597.) introduced an attention mechanism to model the pairwise relationship between different predictions and used bipartite matching to solve the problem of duplicate predictions. DETR (N.Carion, F.Massa, G.Synnaeve, N.Usunier, A.Kirillov, and S.Zagoruyko, "End-to-end object detection with transformers," in Proc.Eur.Conf.Comp.Vis., 2020, pp.213–229.) was inspired by the above methods and used a set of learnable query objects to interact with the global context. By adopting the bipartite matching rule and the Hungarian algorithm, DETR can directly output sparse results during inference without NMS. However, DETR lacks a multi-scale feature fusion mechanism, performs poorly on small objects, and requires longer training time than mainstream detectors.
[0004] Recently, Sun et al. (P.Sun et al., “What makes for end-to-end object detection?” in Proc. Int. Conf. Mach. Learn., 2021, pp. 9934–9944.) pointed out that the one-to-many label assignment rule adopted by mainstream detectors is the main reason for duplicate predictions during inference. Label assignment refers to the selection of anchors as foreground or background samples for training. It is considered to be a key factor affecting the performance of detectors. Usually, during training, the one-to-many label assignment rule adopted by mainstream detectors assigns multiple predictions to each ground-truth as a foreground sample to obtain robust feature representation capabilities. However, this rule will lead to redundant detection, which inevitably requires NMS to remove duplicate predictions and cannot perform end-to-end detection. To solve the above problem, researchers have tried to adopt a one-to-one label assignment rule to get rid of the dependence on NMS. Unlike the one-to-many label assignment rule, the one-to-one assignment rule only matches one foreground sample with a single ground-truth during training. Wang et al. (J. Wang, L. Song, Z. Li, H. Sun, J. Sun, and N. Zheng, “End-to-end object detection with fully convolutional network,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021, pp. 15849–15858.) used this rule to directly output detection results without NMS while achieving competitive detection performance. However, the one-to-one label assignment provides fewer foreground samples for supervision, exacerbating the imbalance of foreground-background classes, resulting in poor feature representation and slow network convergence. In addition, under the same conditions, the performance of the detector using the one-to-one assignment rule is not as good as the original detector trained using the one-to-many assignment rule. Summary of the invention
[0005] The purpose of the present invention is to provide an end-to-end target detection method and system based on dynamic hybrid label allocation, which can not only learn robust feature representation, but also perform efficient end-to-end detection during the reasoning process.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is: an end-to-end target detection method based on dynamic hybrid label allocation, comprising the following steps:
[0007] 1) Obtain training data set and training set annotation files;
[0008] 2) constructing an end-to-end target detection network model based on dynamic hybrid label allocation, wherein the end-to-end target detection model includes a preliminary feature extraction module, a multi-scale feature fusion module, a detection head module, a hybrid sample selection module, and a stage-aware soft weight adjustment module; training the end-to-end target detection network model through a training data set and a training set annotation file to obtain generalizable model parameters;
[0009] 3) Use the trained end-to-end target detection network model to detect targets and achieve end-to-end target detection.
[0010] Furthermore, in step 1), a training data set and a corresponding training set annotation file are obtained from an official open source data set, and the training data set is a MSCOCO object detection data set.
[0011] Furthermore, in step 2), the implementation method of training the end-to-end target detection network model by using the training data set and the training set annotation file is:
[0012] A) Inputting the training data set and the training set annotation files into the preliminary feature extraction module in batches to extract feature maps, and then inputting the extracted feature maps into the multi-scale feature fusion module at the same time;
[0013] B) the multi-scale feature fusion module extracts multi-scale features from the feature map through multiple feature fusion stages of different scales, and then samples the low-scale features into high-scale features through upsampling layer by layer and fuses them with the high-scale features;
[0014] C) inputting the multi-scale feature map obtained by the multi-scale feature fusion module into the detection head module, and the detection head module converts the high-dimensional feature map output by the multi-scale feature fusion module into a specific target detection result; the detection head divides the target detection task into a classification task and a positioning task; in the classification task, the detection head classifies each position in the feature map to determine whether the position belongs to the set target category or the background, and obtains a classification feature map; in the positioning task, the detection head regresses the bounding box coordinates of the target to obtain a positioning feature map;
[0015] D) inputting the obtained classification feature map and positioning feature map into a hybrid sample selection module, wherein the hybrid sample selection module calculates the consistency of the classification and positioning tasks at each position on the feature map and uses it as a selection metric for label assignment; the hybrid sample selection module assigns two groups of foreground samples to each instance, one group is selected by a one-to-many label assignment rule, and multiple foreground samples are used to strengthen representation learning; and the other group is selected by a one-to-one label assignment rule with a unique deduplication sample to achieve repeated prediction removal;
[0016] E) The two groups of foreground samples obtained from the mixed sample selection module are input into a stage-aware soft weight adjustment module, which dynamically adjusts the contribution of the foreground samples in the network training during the network training stage, so as to fully perform representation learning in the early stage of training and learn to remove repeated predictions in the later stage of network training.
[0017] Furthermore, in step A), the preliminary feature extraction module includes five preliminary feature extraction stages, namely Stage-1, Stage-2, Stage-3, Stage-4 and Stage-5; the input image undergoes feature extraction through the five preliminary feature extraction stages respectively; the sizes of the feature maps C3, C4 and C5 extracted by Stage-3, Stage-4 and Stage-5 are 256×100×128, 512×50×64 and 1024×25×32 respectively.
[0018] Further, in step B), the multi-scale feature fusion module includes five multi-scale feature fusion stages of FPN-1, FPN-2, FPN-3, FPN-4 and FPN-5; after the input image passes through the preliminary feature extraction module, the feature maps C3, C4 and C5 of the last three preliminary feature extraction stages are obtained; the feature map C3 is input into the first multi-scale feature fusion stage FPN-1 to obtain the F3 layer feature map with a size of 256×100×128; the feature map C4 is input into the second multi-scale feature fusion stage FPN-2 to obtain the F4 layer feature map with a size of 256×50×64; the feature map C5 is input into the third multi-scale feature fusion stage FPN-3 to obtain the F5 layer feature map with a size of 256×25×32; the F5 layer feature map is input into the fourth multi-scale feature fusion stage FPN-4 to obtain the F5 layer feature map. In the scale feature fusion stage FPN-4, the F6 layer feature map is obtained, with a size of 256×13×16; the F6 layer feature map is input into the fifth multi-scale feature fusion stage FPN-5 to obtain the F7 layer feature map, with a size of 256×7×8; the F7 layer feature map is sampled to the F6 layer feature map through the upsampling technology, and added to the F6 layer feature map to obtain the multi-scale feature map P6; the F6 layer feature map is sampled to the F5 layer feature map through the upsampling technology, and added to the F5 layer feature map to obtain the multi-scale feature map P5; the F5 layer feature map is sampled to the F4 layer feature map through the upsampling technology, and added to the F4 layer feature map to obtain the multi-scale feature map P4 layer; the F4 layer feature map is sampled to the F3 layer feature map through the upsampling technology, and added to the F3 layer feature map to obtain the multi-scale feature map P3.
[0019] Further, in step C), the detection head module includes five detection heads Head-1, Head-2, Head-3, Head-4 and Head-5; the 5-layer multi-scale feature maps P3-P7 obtained from the multi-scale feature fusion module have sizes of 256×100×128, 256×50×64, 256×25×32, 256×13×16 and 256×7×8 respectively; the multi-scale feature maps P3-P7 are respectively input into the corresponding detection heads; after P3 passes through the first detection head Head-1, the classification feature map Cls-1 and the positioning feature map Loc-1 are output, and the sizes are 100×128×80 and 100×128×4 respectively; after P4 passes through the second detection head After head Head-2, P5 outputs classification feature map Cls-2 and positioning feature map Loc-2, with sizes of 50×64×80 and 50×64×4 respectively; after P5 passes through the third detection head Head-3, it outputs classification feature map Cls-3 and positioning feature map Loc-3, with sizes of 25×32×80 and 25×32×4 respectively; after P6 passes through the fourth detection head Head-4, it outputs classification feature map Cls-4 and positioning feature map Loc-4, with sizes of 13×16×80 and 13×16×4 respectively; after P7 passes through the fifth detection head Head-5, it outputs classification feature map Cls-5 and positioning feature map Loc-5, with sizes of 7×8×80 and 7×8×4 respectively.
[0020] Furthermore, in step D), the working method of the mixed sample selection module is:
[0021] The classification feature maps and positioning feature maps of different scales obtained by the detection head module are used in turn. The classification and positioning losses are used to estimate the classification and positioning task quality of each instance i and each anchor point j on each feature map, and the task consistency of each anchor point is calculated as the selection metric of label assignment, which is expressed as:
[0022]
[0023] Among them, M i,j represents the task consistency between the i-th instance and the j-th anchor point; C i,j represents the spatial prior function. When the jth anchor point is inside the bounding box of the i-th instance, C i,j =1, otherwise, C i,j =0;L cls and L loc Represent classification and localization losses respectively; and Respectively represent the category and positioning box of the i-th instance in the picture; and They represent the category and prediction box of the jth anchor point predicted by the network respectively; θ represents the network model parameters; α and β represent the balance parameters;
[0024] First, task consistency is used as a one-to-many label assignment rule to select foreground samples. The specific metric is expressed as:
[0025]
[0026] Where Ψ(i) represents the index of the foreground sample matched by the i-th instance; TopK represents the selection function used to select the coordinates of the anchor points with the top k task consistency quality rankings; n represents the number of anchor points;
[0027] Secondly, a one-to-one label assignment rule is used to select a foreground sample with the highest task consistency score from the selected foreground samples, expressed as:
[0028]
[0029] Among them, Φ(i) represents the index of the foreground sample with the highest task consistency matched by the i-th instance; the argmax function is used to obtain the index of the maximum value.
[0030] Further, in step E), the working method of the stage-aware soft weight adjustment module is:
[0031] Using the two groups of foreground samples obtained in step D), one group is k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, and the other group is the only foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; by dynamically adjusting the soft weights of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, it dynamically adjusts the contribution of these foreground samples in different network training stages t to achieve a balance between representation learning and repeated prediction removal; the dynamic adjustment process is achieved through the classification loss L cls The implementation is shown as follows:
[0032]
[0033] Where BCE represents the binary cross entropy function, which is used to calculate the gap between the predicted probability and the true probability; Φ(i) represents the index of the unique foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; Ψ(i) represents the index of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule; represents the classification score predicted by the only foreground sample with the highest task consistency matched by the i-th instance; Represents the classification probability predicted by the jth foreground sample matched by the i-th instance; represents the soft weight; t represents the current training time; among them, the soft weight It is expressed as follows:
[0034]
[0035] in, represents the normalization parameter, which normalizes the weights of different samples to the same scale and multiplies them by the maximum value of the intersection over union (IoU) between the predicted box and the instance to highlight the instance levels of different samples; Δ t It is used to adjust the contribution of samples at different training times t, which is specifically expressed as follows:
[0036]
[0037] Among them, Δ t is a variable that changes with time t; at time t, all foreground samples are assigned the same contribution rate; δ1 and δ2 control the lower and upper limits of the contribution; η controls the linearity of the contribution decrease; N represents the total training time.
[0038] The present invention also provides an end-to-end target detection system based on dynamic hybrid label allocation, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.
[0039] Compared with the prior art, the present invention has the following beneficial effects: the present method and system effectively combine the advantages of one-to-one and one-to-many label assignment, not only learning robust feature representation capabilities during training, but also performing efficient end-to-end detection during reasoning. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is an architecture diagram of an end-to-end target detection network model based on dynamic label allocation in an embodiment of the present invention.
[0041] Figure 2 It is a working principle diagram of the hybrid sample selection module and the stage-aware soft weight adjustment module in an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0043] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0045] This embodiment provides an end-to-end target detection method based on dynamic hybrid label allocation, including the following steps:
[0046] 1) Obtain training data set and training set annotation files;
[0047] 2) constructing an end-to-end target detection network model based on dynamic hybrid label allocation, wherein the end-to-end target detection model includes a preliminary feature extraction module, a multi-scale feature fusion module, a detection head module, a hybrid sample selection module, and a stage-aware soft weight adjustment module; training the end-to-end target detection network model through a training data set and a training set annotation file to obtain generalizable model parameters;
[0048] 3) Use the trained end-to-end target detection network model to detect targets and achieve end-to-end target detection.
[0049] In this embodiment, a training data set and a corresponding training set annotation file are obtained from an official open source data set, and the training data set is a MSCOCO object detection data set.
[0050] Figure 2 is an architecture diagram of an end-to-end target detection network model based on dynamic hybrid label allocation in this embodiment. Figure 2 As shown, the implementation method for training the end-to-end target detection network model through the training data set and the training set annotation file is as follows.
[0051] A) The training data set and the training set annotation files are input into the preliminary feature extraction module in batches to extract feature maps, and then the extracted feature maps are simultaneously input into the multi-scale feature fusion module.
[0052] Specifically, the preliminary feature extraction module includes five preliminary feature extraction stages: Stage-1, Stage-2, Stage-3, Stage-4 and Stage-5; the input image undergoes feature extraction in five preliminary feature extraction stages; the sizes of feature maps C3, C4 and C5 extracted by Stage-3, Stage-4 and Stage-5 are 256×100×128, 512×50×64 and 1024×25×32 respectively. Here, the feature map is represented by C×W×H, where C, W and H represent the number of feature map channels, feature map width and feature map height respectively.
[0053] B) The multi-scale feature fusion module extracts multi-scale features from the feature map through multiple feature fusion stages of different scales, and then samples the low-scale features into high-scale features through upsampling layer by layer and fuses them with the high-scale features.
[0054] Specifically, the multi-scale feature fusion module includes five multi-scale feature fusion stages: FPN-1, FPN-2, FPN-3, FPN-4 and FPN-5; after the input image passes through the preliminary feature extraction module, the feature maps C3, C4 and C5 of the last three preliminary feature extraction stages are obtained; the feature map C3 is input into the first multi-scale feature fusion stage FPN-1 to obtain the F3 layer feature map with a size of 256×100×128; the feature map C4 is input into the second multi-scale feature fusion stage FPN-2 to obtain the F4 layer feature map with a size of 256×50×64; the feature map C5 is input into the third multi-scale feature fusion stage FPN-3 to obtain the F5 layer feature map with a size of 256×25×32; the F5 layer feature map is input into the fourth multi-scale feature In the fusion stage FPN-4, the F6 layer feature map is obtained, with a size of 256×13×16; the F6 layer feature map is input into the fifth multi-scale feature fusion stage FPN-5 to obtain the F7 layer feature map, with a size of 256×7×8; the F7 layer feature map is sampled to the F6 layer feature map through upsampling technology, and added to the F6 layer feature map to obtain the multi-scale feature map P6; the F6 layer feature map is sampled to the F5 layer feature map through upsampling technology, and added to the F5 layer feature map to obtain the multi-scale feature map P5; the F5 layer feature map is sampled to the F4 layer feature map through upsampling technology, and added to the F4 layer feature map to obtain the multi-scale feature map P4 layer; the F4 layer feature map is sampled to the F3 layer feature map through upsampling technology, and added to the F3 layer feature map to obtain the multi-scale feature map P3.
[0055] C) The multi-scale feature map obtained by the multi-scale feature fusion module is input into the detection head module, and the detection head module converts the high-dimensional feature map output by the multi-scale feature fusion module into a specific target detection result; the detection head divides the target detection task into a classification task and a positioning task; in the classification task, the detection head classifies each position in the feature map to determine whether the position belongs to the set target category or the background, and obtains a classification feature map; in the positioning task, the detection head regresses the bounding box coordinates of the target to obtain a positioning feature map.
[0056] Specifically, the detection head module includes five detection heads Head-1, Head-2, Head-3, Head-4 and Head-5; the 5-layer multi-scale feature maps P3-P7 obtained from the multi-scale feature fusion module have sizes of 256×100×128, 256×50×64, 256×25×32, 256×13×16 and 256×7×8 respectively; the multi-scale feature maps P3-P7 are respectively input into the corresponding detection heads; after P3 passes through the first detection head Head-1, the classification feature map Cls-1 and the positioning feature map Loc-1 are output, with sizes of 100×128×80 and 100×128×4 respectively; P4 passes through the second detection head Head-1 After d-2, the output is the classification feature map Cls-2 and the positioning feature map Loc-2, with sizes of 50×64×80 and 50×64×4 respectively; after P5 passes through the third detection head Head-3, the output is the classification feature map Cls-3 and the positioning feature map Loc-3, with sizes of 25×32×80 and 25×32×4 respectively; after P6 passes through the fourth detection head Head-4, the output is the classification feature map Cls-4 and the positioning feature map Loc-4, with sizes of 13×16×80 and 13×16×4 respectively; after P7 passes through the fifth detection head Head-5, the output is the classification feature map Cls-5 and the positioning feature map Loc-5, with sizes of 7×8×80 and 7×8×4 respectively.
[0057] D) The obtained classification feature map and positioning feature map are input into a hybrid sample selection module, which calculates the consistency of the classification and positioning tasks at each position on the feature map and uses it as a selection metric for label assignment; the hybrid sample selection module assigns two groups of foreground samples to each instance, one group is selected by a one-to-many label assignment rule, and multiple foreground samples are used to strengthen representation learning; the other group is selected by a one-to-one label assignment rule with a unique deduplication sample to achieve repeated prediction removal.
[0058] The working method of the mixed sample selection module is:
[0059] The classification feature maps and positioning feature maps of different scales obtained by the detection head module are used in turn. The classification and positioning losses are used to estimate the classification and positioning task quality of each instance i and each anchor point j (position point on the feature map) on each feature map, and the task consistency of each anchor point is calculated as the selection metric of label assignment, which is expressed as:
[0060]
[0061] Among them, M i,j represents the task consistency between the i-th instance and the j-th anchor point; C i,j represents the spatial prior function. When the jth anchor point is inside the bounding box of the i-th instance, C i,j =1, otherwise, C i,j =0;L cls and L loc Represent classification and localization losses respectively; and Respectively represent the category and positioning box of the i-th instance in the picture; and They represent the category and prediction box of the jth anchor point predicted by the network respectively; θ represents the network model parameters; α and β represent the balance parameters.
[0062] First, task consistency is used as a one-to-many label assignment rule to select foreground samples. The specific metric is expressed as:
[0063]
[0064] Among them, Ψ(i) represents the index of the foreground sample matched by the i-th instance; TopK represents the selection function used to select the coordinates of the anchor points with the top k task consistency quality rankings; the value of k is set to 9; and n represents the number of anchor points.
[0065] Secondly, a one-to-one label assignment rule is used to select a foreground sample with the highest task consistency score from the selected foreground samples, expressed as:
[0066]
[0067] Among them, Φ(i) represents the index of the foreground sample with the highest task consistency matched by the i-th instance; the argmax function is used to obtain the index of the maximum value.
[0068] E) The two groups of foreground samples obtained from the mixed sample selection module are input into a stage-aware soft weight adjustment module, which dynamically adjusts the contribution of the foreground samples in the network training during the network training stage, so as to fully perform representation learning in the early stage of training and learn to remove repeated predictions in the later stage of network training.
[0069] The working method of the stage-aware soft weight adjustment module is:
[0070] Using the two groups of foreground samples obtained in step D), one group is k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, and the other group is the only foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; by dynamically adjusting the soft weights of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, it dynamically adjusts the contribution of these foreground samples in different network training stages t to achieve a balance between representation learning and repeated prediction removal; the dynamic adjustment process is achieved through the classification loss L cls The implementation can be expressed as follows:
[0071]
[0072] Where BCE represents the binary cross entropy function, which is used to calculate the gap between the predicted probability and the true probability; Φ(i) represents the index of the unique foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; Ψ(i) represents the index of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule; represents the classification score predicted by the only foreground sample with the highest task consistency matched by the i-th instance; Represents the classification probability predicted by the jth foreground sample matched by the i-th instance; represents the soft weight; t represents the current training time; among them, the soft weight It is expressed as follows:
[0073]
[0074] in, represents the normalization parameter, which normalizes the weights of different samples to the same scale and multiplies them by the maximum value of the intersection over union (IoU) between the predicted box and the instance to highlight the instance levels of different samples; Δ t It is used to adjust the contribution of samples at different training times t, which is specifically expressed as follows:
[0075]
[0076] Among them, Δ t is a variable that changes with time t; at time t, all foreground samples are assigned the same contribution rate; δ1 and δ2 control the lower and upper limits of the contribution; η controls the linearity of the contribution decrease; N represents the total training time.
[0077] This embodiment also provides an end-to-end target detection system based on dynamic hybrid label allocation, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.
[0078] In this embodiment, the MSCOCO dataset is used to use ResNet-50 as the backbone network for comparative verification. Table 1 shows the comparison results of the method proposed in the present invention with other non-end-to-end and end-to-end methods on the MSCOCO dataset. It can be seen from Table 1 that the method proposed in the present invention has higher accuracy and robustness than other non-end-to-end and end-to-end methods, which is specifically reflected in mAP, AP S ,AP M and AP L In the field of target detection, mAP stands for Average Precision and is used as the main evaluation indicator. S ,AP M and AP L They represent the average accuracy of detecting objects of different areas, specifically, AP S (area<32 2 ), AP M (area<64 2 ) and AP S (area<128 2 )).
[0079] Table 1
[0080]
[0081]
[0082] In Table 1, RetinaNet corresponds to the method proposed by Y.Lin et al. (T.-Y.Lin, P.Goyal, R.Girshick, K.He, and P.Dollar, "Focal loss for dense object detection," in Proc.IEEE Int.Conf.Comp.Vis., 2017, pp.2980–2988.)
[0083] FCOS corresponds to the method proposed by Tian et al. (Z.Tian, C.Shen, H.Chen, and T.He, "FCOS: Fully convolutional onestage object detection," in Proc.IEEE Int.Conf. Comp.Vis., 2019, pp.9627–9636.)
[0084] FreeAnchor corresponds to the method proposed by Zhang et al. (X. Zhang, F. Wan, C. Liu, R. Ji, and Q. Ye, "FreeAnchor: Learning to match anchors for visual object detection," Advances in Neural Inf. Process. Syst., vol. 32, 2019.)
[0085] FSAF corresponds to the method proposed by Zhu et al. (C. Zhu, Y. He, and M. Savvides, "Feature selective anchor-free module for single-shot object detection," in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019, pp. 840–849.)
[0086] SAPD corresponds to the method proposed by Zhu et al. (C. Zhu, F. Chen, Z. Shen, and M. Savvides, "Softanchor-point object detection," in Proc. Eur. Conf. Comp. Vis., 2020, pp. 91–107.)
[0087] ATSS corresponds to the method proposed by Zhang et al. (S. Zhang, C. Chi, Y. Yao, Z. Lei, and S. Z Li, "Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection," in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020, pp. 9759–9768.)
[0088] OneNet corresponds to the method proposed by Sun et al. (P.Sun et al., "What makes for end-to-end object detection?" in Proc.Int.Conf.Mach.Learn., 2021, pp.9934–9944.)
[0089] DeFCN corresponds to the method proposed by Wang et al. (J. Wang, L. Song, Z. Li, H. Sun, J. Sun, and N. Zheng, "End-to-end object detection with fully convolutional network," in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021, pp. 15849–15858.)
[0090] O2F corresponds to the method proposed by Li et al. (S.Li, M.Li, R.Li, C.He, and L.Zhang, "One-to-few label assignment for end-to-end dense detection," in Proc.IEEEConf.Comp.Vis.Patt.Recogn., 2023, pp.7350–7359.)
[0091] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0092] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0093] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0095] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. An end-to-end target detection method based on dynamic hybrid label allocation, characterized in that: The following steps are involved: 1) Obtain training data set and training set annotation files; 2) constructing an end-to-end target detection network model based on dynamic hybrid label allocation, wherein the end-to-end target detection model includes a preliminary feature extraction module, a multi-scale feature fusion module, a detection head module, a hybrid sample selection module, and a stage-aware soft weight adjustment module; training the end-to-end target detection network model through a training data set and a training set annotation file to obtain generalizable model parameters; 3) Use the trained end-to-end target detection network model to detect targets and achieve end-to-end target detection.
2. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 1, characterized in that: In step 1), a training data set and a corresponding training set annotation file are obtained from an official open source data set, wherein the training data set is a MSCOCO object detection data set.
3. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 1, characterized in that: In step 2), the implementation method of training the end-to-end target detection network model through the training data set and the training set annotation file is: A) Inputting the training data set and the training set annotation files into the preliminary feature extraction module in batches to extract feature maps, and then inputting the extracted feature maps into the multi-scale feature fusion module at the same time; B) the multi-scale feature fusion module extracts multi-scale features from the feature map through multiple feature fusion stages of different scales, and then samples the low-scale features into high-scale features through upsampling layer by layer and fuses them with the high-scale features; C) inputting the multi-scale feature map obtained by the multi-scale feature fusion module into the detection head module, and the detection head module converts the high-dimensional feature map output by the multi-scale feature fusion module into a specific target detection result; the detection head divides the target detection task into a classification task and a positioning task; in the classification task, the detection head classifies each position in the feature map to determine whether the position belongs to the set target category or the background, and obtains a classification feature map; In the localization task, the detection head regresses the bounding box coordinates of the target and obtains the localization feature map; D) inputting the obtained classification feature map and positioning feature map into a hybrid sample selection module, wherein the hybrid sample selection module calculates the consistency of the classification and positioning tasks at each position on the feature map and uses it as a selection metric for label assignment; the hybrid sample selection module assigns two groups of foreground samples to each instance, one group is selected by a one-to-many label assignment rule, and multiple foreground samples are used to strengthen representation learning; and the other group is selected by a one-to-one label assignment rule with a unique deduplication sample to achieve repeated prediction removal; E) The two groups of foreground samples obtained from the mixed sample selection module are input into a stage-aware soft weight adjustment module, which dynamically adjusts the contribution of the foreground samples in the network training during the network training stage, so as to fully perform representation learning in the early stage of training and learn to remove repeated predictions in the later stage of network training.
4. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 3 is characterized in that: In step A), the preliminary feature extraction module includes five preliminary feature extraction stages, namely Stage-1, Stage-2, Stage-3, Stage-4 and Stage-5; the input image undergoes feature extraction in five preliminary feature extraction stages respectively; the sizes of feature maps C3, C4 and C5 extracted by Stage-3, Stage-4 and Stage-5 are 256×100×128, 512×50×64 and 1024×25×32 respectively.
5. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 3 is characterized in that: In step B), the multi-scale feature fusion module includes five multi-scale feature fusion stages: FPN-1, FPN-2, FPN-3, FPN-4 and FPN-5; after the input image passes through the preliminary feature extraction module, the feature maps C3, C4 and C5 of the last three preliminary feature extraction stages are obtained; the feature map C3 is input into the first multi-scale feature fusion stage FPN-1 to obtain the F3 layer feature map with a size of 256×100×128; the feature map C4 is input into the second multi-scale feature fusion stage FPN-2 to obtain the F4 layer feature map with a size of 256×50×64; the feature map C5 is input into the third multi-scale feature fusion The F5 layer feature map is input into the fourth multi-scale feature fusion stage FPN-4 to obtain the F6 layer feature map with a size of 256×13×16; the F6 layer feature map is input into the fifth multi-scale feature fusion stage FPN-5 to obtain the F7 layer feature map with a size of 256×7×8; the F7 layer feature map is sampled to the F6 layer feature map through the upsampling technology, and added to the F6 layer feature map to obtain the multi-scale feature map P6; the F6 layer feature map is sampled to the F5 layer feature map through the upsampling technology, and added to the F5 layer feature map to obtain the multi-scale feature map P5; The F5 layer feature map is sampled to the F4 layer feature map through upsampling technology, and added to the F4 layer feature map to obtain the multi-scale feature map P4 layer; the F4 layer feature map is sampled to the F3 layer feature map through upsampling technology, and added to the F3 layer feature map to obtain the multi-scale feature map P3.
6. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 3, characterized in that: In step C), the detection head module includes five detection heads Head-1, Head-2, Head-3, Head-4 and Head-5; the 5-layer multi-scale feature maps P3-P7 obtained from the multi-scale feature fusion module have sizes of 256×100×128, 256×50×64, 256×25×32, 256×13×16 and 256×7×8 respectively; the multi-scale feature maps P3-P7 are respectively input into the corresponding detection heads; after P3 passes through the first detection head Head-1, the classification feature map Cls-1 and the positioning feature map Loc-1 are output, with sizes of 100×128×80 and 100×128×4 respectively; P4 passes through the second detection head He After ad-2, P5 outputs classification feature map Cls-2 and positioning feature map Loc-2, with sizes of 50×64×80 and 50×64×4 respectively; after P5 passes through the third detection head Head-3, it outputs classification feature map Cls-3 and positioning feature map Loc-3, with sizes of 25×32×80 and 25×32×4 respectively; after P6 passes through the fourth detection head Head-4, it outputs classification feature map Cls-4 and positioning feature map Loc-4, with sizes of 13×16×80 and 13×16×4 respectively; after P7 passes through the fifth detection head Head-5, it outputs classification feature map Cls-5 and positioning feature map Loc-5, with sizes of 7×8×80 and 7×8×4 respectively.
7. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 3 is characterized in that: In step D), the working method of the mixed sample selection module is: The classification feature maps and positioning feature maps of different scales obtained by the detection head module are used in turn. The classification and positioning losses are used to estimate the classification and positioning task quality of each instance i and each anchor point j on each feature map, and the task consistency of each anchor point is calculated as the selection metric of label assignment, which is expressed as: Among them, M i,j represents the task consistency between the i-th instance and the j-th anchor point; C i,j represents the spatial prior function. When the jth anchor point is inside the bounding box of the i-th instance, C i,j =1, otherwise, C i,j =0;L cls and L loc Represent classification and localization losses respectively; and Respectively represent the category and positioning box of the i-th instance in the picture; and They represent the category and prediction box of the jth anchor point predicted by the network respectively; θ represents the network model parameters; α and β represent the balance parameters; First, task consistency is used as a one-to-many label assignment rule to select foreground samples. The specific metric is expressed as: Where Ψ(i) represents the index of the foreground sample matched by the i-th instance; TopK represents the selection function used to select the coordinates of the anchor points with the top k task consistency quality rankings; n represents the number of anchor points; Secondly, a one-to-one label assignment rule is used to select a foreground sample with the highest task consistency score from the selected foreground samples, expressed as: Among them, Φ(i) represents the index of the foreground sample with the highest task consistency matched by the i-th instance; the argmax function is used to obtain the index of the maximum value.
8. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 7, characterized in that: In step E), the working method of the stage-aware soft weight adjustment module is: Using the two groups of foreground samples obtained in step D), one group is k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, and the other group is the only foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; By dynamically adjusting the soft weights of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, it dynamically adjusts the contribution of these foreground samples in different network training stages t to achieve a balance between representation learning and repeated prediction removal; the dynamic adjustment process is achieved through the classification loss L cls The implementation is shown as follows: Where BCE represents the binary cross entropy function, which is used to calculate the gap between the predicted probability and the true probability; φ(i) represents the index of the unique foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; Ψ(i) represents the index of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule; represents the classification score predicted by the only foreground sample with the highest task consistency matched by the i-th instance; Represents the classification probability predicted by the jth foreground sample matched by the i-th instance; represents the soft weight; t represents the current training time; among them, the soft weight It is expressed as follows: in, represents the normalization parameter, which normalizes the weights of different samples to the same scale and multiplies them by the maximum value of the intersection over union (IoU) between the predicted box and the instance to highlight the instance levels of different samples; Δ t It is used to adjust the contribution of samples at different training times t, which is specifically expressed as follows: Among them, Δ t is a variable that changes with time t; at time t, all foreground samples are assigned the same contribution rate; δ1 and δ2 control the lower and upper limits of the contribution; η controls the linearity of the contribution decrease; N represents the total training time.
9. An end-to-end target detection system based on dynamic hybrid label allocation, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 8 can be implemented.
Citation Information
Patent Citations
Small target detection method based on weighted score label distribution
CN117115412A
Method for training student network and method for recognizing image
US20230046088A1
Semi-supervised iterative keypoint and viewpoint invariant feature learning for visual recognition
WO2019010147A1
