End-to-end object detection method and system based on dynamic hybrid label assignment

Abstract

The application relates to an end-to-end target detection method and system based on dynamic mixed label allocation, which comprises the following steps: 1) obtaining a training data set and a training set annotation file; 2) constructing an end-to-end target detection network model based on dynamic mixed label allocation, wherein the end-to-end target detection model comprises a preliminary feature extraction module, a multi-scale feature fusion module, a detection head module, a mixed sample selection module and a stage perception soft weight adjustment module; the end-to-end target detection network model is trained through the training data set and the training set annotation file to obtain model parameters that can be generalized; and 3) applying the trained end-to-end target detection network model to detect targets to realize end-to-end target detection. The method can not only learn stable feature representation, but also perform efficient end-to-end detection in the inference process.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an end-to-end target detection method and system based on dynamic hybrid label allocation. Background Technology

[0002] Object detection is a long-standing task in computer vision, aiming to detect the category and location of predetermined objects in images. It has broad application prospects in various fields such as autonomous driving, video surveillance, face recognition, and medical image analysis. Currently, mainstream detectors predict objects with different scales, aspect ratios, and categories at each feature location using a regular, dense sampling method. These dense detectors rely on a manually defined component, non-maximum suppression (NMS), to eliminate duplicate predictions caused by dense detection in post-processing steps. However, this process makes these detectors sensitive to the hyperparameters of NMS and may lead to suboptimal solutions. Furthermore, when faced with complex, densely overlapping objects, detectors easily miss targets, resulting in false negatives. These problems have prompted researchers to remove NMS to achieve end-to-end detection.

[0003] In earlier research, some recurrent neural network (RNN) frameworks with long short-term memory (LSTM) units decoded image content into a set of variable-length prediction boxes. While they achieved end-to-end detection, they only demonstrated effectiveness on limited, small datasets. CenterNet (X. Zhou, D. Wang, and P. Krahenbuhl, “Objects as points,” arXiv preprint arXiv:1904.07850, 2019.) and its predecessor (H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in Proc. Eur. Conf. Comp. Vis., 2018, pp. 734–750.) achieved differentiable end-to-end training by modeling ground reality as a heatmap and transforming object detection into a keypoint estimation problem. However, it requires significantly more training time compared to baseline detectors. Recently, Hu et al. (H.Hu, J.Gu, Z.Zhang, J.Dai, and Y.Wei, “Relationnetworks for object detection,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018, pp. 3588–3597.) introduced an attention mechanism to model pairwise relationships between different predictions and used binary matching to address the problem of duplicate predictions. Inspired by the above methods, DETR (N.Carion, F.Massa, G.Synnaeve, N.Usunier, A.Kirrillov, and S.Zagoruyko, “End-to-end object detection with transformers,” in Proc. Eur. Conf. Comp. Vis., 2020, pp. 213–229.) utilizes a set of learnable query objects to interact with the global context. By employing binary matching rules and the Hungarian algorithm, DETR can directly output sparse results during inference without NMS. However, DETR lacks a multi-scale feature fusion mechanism, performs poorly on small objects, and requires a longer training time than mainstream detectors.

[0004] Recently, Sun et al. (P. Sun et al., “What makes for end-to-end object detection?” in Proc. Int. Conf. Mach. Learn., 2021, pp. 9934–9944.) pointed out that the one-to-many label assignment rule used by mainstream detectors is the main reason for duplicate predictions during inference. Label assignment refers to the selection of anchor points as foreground or background samples for training. It is considered a key factor affecting detector performance. Typically, during training, the one-to-many label assignment rule used by mainstream detectors assigns multiple predictions to each ground-truth as a foreground sample to obtain robust feature representation capabilities. However, this rule inevitably leads to redundant detection requiring NMS to remove duplicate predictions, making end-to-end detection impossible. To address this issue, researchers attempted to use a one-to-one label assignment rule to eliminate the dependence on NMS. Unlike the one-to-many label assignment rule, the one-to-one assignment rule matches only one foreground sample with a single ground-truth during training. Wang et al. (J.Wang, L.Song, Z.Li, H.Sun, J.Sun, and N.Zheng, “End-to-end object detection with fully convolutional network,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021, pp. 15849–15858.) achieved competitive detection performance by directly outputting detection results without NMS using this rule. However, one-to-one label assignment provides fewer foreground samples for supervision, exacerbating the foreground-background class imbalance and resulting in poor feature representation and slow network convergence. Furthermore, under the same conditions, the detector using the one-to-one assignment rule performs worse than the original detector trained using the one-to-many assignment rule. Summary of the Invention

[0005] The purpose of this invention is to provide an end-to-end target detection method and system based on dynamic hybrid label allocation. This method can not only learn robust feature representations, but also perform efficient end-to-end detection during inference.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: an end-to-end target detection method based on dynamic hybrid label allocation, comprising the following steps:

[0007] 1) Obtain the training dataset and training set annotation files;

[0008] 2) Construct an end-to-end object detection network model based on dynamic hybrid label assignment. The end-to-end object detection model includes a preliminary feature extraction module, a multi-scale feature fusion module, a detection head module, a hybrid sample selection module, and a stage-aware soft weight adjustment module. Train the end-to-end object detection network model using the training dataset and training set annotation files to obtain generalizable model parameters.

[0009] 3) Use the trained end-to-end target detection network model to detect targets and achieve end-to-end target detection.

[0010] Further, in step 1), the training dataset and the corresponding training set annotation file are obtained from the official open-source dataset, wherein the training dataset is the MSCOCO object detection dataset.

[0011] Furthermore, in step 2), the method for training the end-to-end object detection network model using the training dataset and the training set annotation file is as follows:

[0012] A) Input the training dataset and training set annotation file into the preliminary feature extraction module in batches to extract feature maps, and then input the extracted feature maps into the multi-scale feature fusion module at the same time;

[0013] B) The multi-scale feature fusion module extracts multi-scale features from the feature map through multiple feature fusion stages of different scales, and then upsamples the low-scale features to high-scale features layer by layer and fuses them with the high-scale features.

[0014] C) The multi-scale feature map obtained by the multi-scale feature fusion module is input into the detection head module. The detection head module converts the high-dimensional feature map output by the multi-scale feature fusion module into a specific target detection result. The detection head divides the target detection task into a classification task and a localization task. In the classification task, the detection head classifies each position in the feature map to determine whether the position belongs to the set target category or to the background, thus obtaining a classification feature map. In the localization task, the detection head regresses the bounding box coordinates of the target, thus obtaining a localization feature map.

[0015] D) Input the obtained classification feature map and localization feature map into the mixed sample selection module. The mixed sample selection module calculates the consistency between the classification and localization tasks at each location on the feature map and uses it as the selection metric for label assignment. The mixed sample selection module assigns two sets of foreground samples to each instance. One set is selected by a one-to-many label assignment rule, and multiple foreground samples are used to enhance representation learning. The other set is selected by a one-to-one label assignment rule to select a unique deduplicated sample to achieve duplicate prediction removal.

[0016] E) The two sets of foreground samples obtained from the mixed sample selection module are input into the stage-aware soft weight adjustment module. The stage-aware soft weight adjustment module dynamically adjusts the contribution of the foreground samples in the network training stage, so as to fully perform representation learning in the early stage of training and learn to remove repeated predictions in the later stage of network training.

[0017] Further, in step A), the preliminary feature extraction module includes five preliminary feature extraction stages: Stage-1, Stage-2, Stage-3, Stage-4, and Stage-5. The input image undergoes feature extraction through each of the five preliminary feature extraction stages. The feature maps C3, C4, and C5 extracted by Stage-3, Stage-4, and Stage-5 are 256×100×128, 512×50×64, and 1024×25×32, respectively.

[0018] Further, in step B), the multi-scale feature fusion module includes five multi-scale feature fusion stages: FPN-1, FPN-2, FPN-3, FPN-4, and FPN-5. After the input image passes through the preliminary feature extraction module, feature maps C3, C4, and C5 from the last three preliminary feature extraction stages are obtained. Feature map C3 is input into the first multi-scale feature fusion stage FPN-1 to obtain the F3 layer feature map, with a size of 256×100×128. Feature map C4 is input into the second multi-scale feature fusion stage FPN-2 to obtain the F4 layer feature map, with a size of 256×50×64. Feature map C5 is input into the third multi-scale feature fusion stage FPN-3 to obtain the F5 layer feature map, with a size of 256×25×32. The F5 layer feature map is input into the fourth multi-scale feature fusion stage FPN-5. In the scale feature fusion stage FPN-4, a feature map of layer F6 with a size of 256×13×16 is obtained. The feature map of layer F6 is input into the fifth multi-scale feature fusion stage FPN-5 to obtain a feature map of layer F7 with a size of 256×7×8. The feature map of layer F7 is upsampled to the feature map of layer F6 and added to the feature map of layer F6 to obtain a multi-scale feature map P6. The feature map of layer F6 is upsampled to the feature map of layer F5 and added to the feature map of layer F5 to obtain a multi-scale feature map P5. The feature map of layer F5 is upsampled to the feature map of layer F4 and added to the feature map of layer F4 to obtain a multi-scale feature map layer P4. The feature map of layer F4 is upsampled to the feature map of layer F3 and added to the feature map of layer F3 to obtain a multi-scale feature map P3.

[0019] Further, in step C), the detection head module includes five detection heads: Head-1, Head-2, Head-3, Head-4, and Head-5; five multi-scale feature maps P3-P7 obtained from the multi-scale feature fusion module have sizes of 256×100×128, 256×50×64, 256×25×32, 256×13×16, and 256×7×8, respectively; the multi-scale feature maps P3-P7 are input into the corresponding detection heads; after P3 passes through the first detection head Head-1, it outputs a classification feature map Cls-1 and a localization feature map Loc-1, with sizes of 100×128×80 and 100×128×4, respectively; P4 passes through the second detection head... After Head-2, P5 outputs classification feature map Cls-2 and localization feature map Loc-2, with sizes of 50×64×80 and 50×64×4, respectively; after passing through the third detection head Head-3, P5 outputs classification feature map Cls-3 and localization feature map Loc-3, with sizes of 25×32×80 and 25×32×4, respectively; after passing through the fourth detection head Head-4, P6 outputs classification feature map Cls-4 and localization feature map Loc-4, with sizes of 13×16×80 and 13×16×4, respectively; after passing through the fifth detection head Head-5, P7 outputs classification feature map Cls-5 and localization feature map Loc-5, with sizes of 7×8×80 and 7×8×4, respectively.

[0020] Further, in step D), the working method of the mixed sample selection module is as follows:

[0021] For the classification and localization feature maps of different scales obtained from the detection head module, the classification and localization task quality of each instance i and each anchor point j on each feature map is estimated using classification and localization losses. The task consistency of each anchor point is calculated as a selection metric for label assignment, expressed as:

[0022]

[0023] Among them, M i,j This indicates the task consistency between the i-th instance and the j-th anchor point; C i,j C represents the spatial prior function, where C is the value of the anchor point j within the bounding box of the instance i. i,j =1, except for this, C i,j =0; L cls and L loc These represent the classification loss and the localization loss, respectively. and Let i and j represent the category and bounding box of the i-th instance in the image, respectively. and represents the category and predicted bounding box of the j-th anchor point predicted by the network, respectively; θ represents the network model parameters; α and β represent the balancing parameters;

[0024] First, task consistency is used as a one-to-many label assignment rule to select foreground samples, and the specific metric is expressed as follows:

[0025]

[0026] Where Ψ(i) represents the index of the foreground sample matched by the i-th instance; TopK represents the selection function used to select the coordinates of the top k anchor points in terms of task consistency quality; and n represents the number of anchor points.

[0027] Secondly, using a one-to-one label assignment rule, a foreground sample with the highest task consistency score is selected from the chosen foreground samples, denoted as:

[0028]

[0029] Where Φ(i) represents the index of the foreground sample with the highest task consistency matched by the i-th instance; the argmax function is used to obtain the index of the maximum value.

[0030] Further, in step E), the working method of the stage-aware soft weight adjustment module is as follows:

[0031] Using the two sets of foreground samples obtained in step D), one set consists of k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, and the other set consists of the unique foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule. By dynamically adjusting the soft weights of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, the contributions of these foreground samples in different network training stages t are dynamically adjusted to achieve a balance between representation learning and duplicate prediction removal. The dynamic adjustment process is achieved through the classification loss L. cls The implementation is represented as follows:

[0032]

[0033] Where BCE represents the binary cross-entropy function, used to calculate the difference between the predicted probability and the true probability; Φ(i) represents the index of the unique foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; Ψ(i) represents the index of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule. The classification score predicted by the unique foreground sample with the highest task consistency matched by the i-th instance represents the classification score. This represents the predicted classification probability of the j-th foreground sample matched with the i-th instance; The soft weights represent the training time; t represents the current training time. It is expressed as follows:

[0034]

[0035] in, Δ represents the normalization parameter, which normalizes the weights of different samples to the same scale, while multiplying by the maximum Intersection over Union (IoU) between the predicted bounding box and the instance to highlight the different instance levels; t This is used to adjust the contribution of samples at different training times t, and is specifically represented as follows:

[0036]

[0037] Where, Δ t η is a variable that changes with time t; at time t, all foreground samples are assigned the same contribution rate; δ1 and δ2 control the lower and upper bounds of the contribution; η controls the linearity of the contribution decrease; N represents the total training time.

[0038] The present invention also provides an end-to-end target detection system based on dynamic hybrid tag allocation, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0039] Compared with the prior art, the present invention has the following advantages: the method and system effectively combine the advantages of one-to-one and one-to-many label assignment, not only learning robust feature representation capabilities during training, but also performing efficient end-to-end detection during inference. Attached Figure Description

[0040] Figure 1 This is an architecture diagram of an end-to-end target detection network model based on dynamic label allocation in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram illustrating the working principle of the hybrid sample selection module and the stage-aware soft weight adjustment module in this embodiment of the invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] This embodiment provides an end-to-end target detection method based on dynamic hybrid label allocation, including the following steps:

[0046] 1) Obtain the training dataset and training set annotation files;

[0047] 2) Construct an end-to-end object detection network model based on dynamic hybrid label assignment. The end-to-end object detection model includes a preliminary feature extraction module, a multi-scale feature fusion module, a detection head module, a hybrid sample selection module, and a stage-aware soft weight adjustment module. Train the end-to-end object detection network model using the training dataset and training set annotation files to obtain generalizable model parameters.

[0048] 3) Use the trained end-to-end target detection network model to detect targets and achieve end-to-end target detection.

[0049] In this embodiment, the training dataset and corresponding training set annotation files are obtained from the official open-source dataset. The training dataset is the MSCOCO object detection dataset.

[0050] Figure 2 This is an architecture diagram of the end-to-end object detection network model based on dynamic hybrid label allocation in this embodiment. Figure 2 As shown, the implementation method for training an end-to-end object detection network model using a training dataset and a training set annotation file is as follows.

[0051] A) Input the training dataset and training set annotation file into the preliminary feature extraction module in batches to extract feature maps, and then input the extracted feature maps into the multi-scale feature fusion module at the same time.

[0052] Specifically, the preliminary feature extraction module includes five preliminary feature extraction stages: Stage-1, Stage-2, Stage-3, Stage-4, and Stage-5. The input image undergoes feature extraction through each of these five stages. The feature maps C3, C4, and C5 extracted by Stage-3, Stage-4, and Stage-5 have sizes of 256×100×128, 512×50×64, and 1024×25×32, respectively. Here, the feature map is represented by C×W×H, where C, W, and H represent the number of feature map channels, the feature map width, and the feature map height, respectively.

[0053] B) The multi-scale feature fusion module extracts multi-scale features from the feature map through multiple feature fusion stages of different scales, and then upsamples the low-scale features to high-scale features layer by layer and fuses them with the high-scale features.

[0054] Specifically, the multi-scale feature fusion module includes five multi-scale feature fusion stages: FPN-1, FPN-2, FPN-3, FPN-4, and FPN-5. After the input image passes through the preliminary feature extraction module, feature maps C3, C4, and C5 from the last three preliminary feature extraction stages are obtained. Feature map C3 is input into the first multi-scale feature fusion stage FPN-1 to obtain the F3 layer feature map, with a size of 256×100×128. Feature map C4 is input into the second multi-scale feature fusion stage FPN-2 to obtain the F4 layer feature map, with a size of 256×50×64. Feature map C5 is input into the third multi-scale feature fusion stage FPN-3 to obtain the F5 layer feature map, with a size of 256×25×32. The F5 layer feature map is then input into the fourth multi-scale feature fusion stage. In the fusion stage FPN-4, the F6 layer feature map is obtained, with a size of 256×13×16. The F6 layer feature map is input into the fifth multi-scale feature fusion stage FPN-5 to obtain the F7 layer feature map, with a size of 256×7×8. The F7 layer feature map is upsampled to the F6 layer feature map and added to the F6 layer feature map to obtain the multi-scale feature map P6. The F6 layer feature map is upsampled to the F5 layer feature map and added to the F5 layer feature map to obtain the multi-scale feature map P5. The F5 layer feature map is upsampled to the F4 layer feature map and added to the F4 layer feature map to obtain the multi-scale feature map P4. The F4 layer feature map is upsampled to the F3 layer feature map and added to the F3 layer feature map to obtain the multi-scale feature map P3.

[0055] C) The multi-scale feature map obtained by the multi-scale feature fusion module is input into the detection head module. The detection head module converts the high-dimensional feature map output by the multi-scale feature fusion module into a specific target detection result. The detection head divides the target detection task into a classification task and a localization task. In the classification task, the detection head classifies each position in the feature map to determine whether the position belongs to the set target category or to the background, thus obtaining a classification feature map. In the localization task, the detection head regresses the bounding box coordinates of the target, thus obtaining a localization feature map.

[0056] Specifically, the detection head module includes five detection heads: Head-1, Head-2, Head-3, Head-4, and Head-5; five multi-scale feature maps P3-P7 obtained from the multi-scale feature fusion module, with sizes of 256×100×128, 256×50×64, 256×25×32, 256×13×16, and 256×7×8, respectively; the multi-scale feature maps P3-P7 are input into the corresponding detection heads; after passing through the first detection head Head-1, P3 outputs a classification feature map Cls-1 and a localization feature map Loc-1, with sizes of 100×128×80 and 100×128×4, respectively; P4 passes through the second detection head Head-5... After d-2, the output class feature map Cls-2 and localization feature map Loc-2 are 50×64×80 and 50×64×4 respectively; after P5 passes through the third detection head Head-3, the output class feature map Cls-3 and localization feature map Loc-3 are 25×32×80 and 25×32×4 respectively; after P6 passes through the fourth detection head Head-4, the output class feature map Cls-4 and localization feature map Loc-4 are 13×16×80 and 13×16×4 respectively; after P7 passes through the fifth detection head Head-5, the output class feature map Cls-5 and localization feature map Loc-5 are 7×8×80 and 7×8×4 respectively.

[0057] D) Input the obtained classification feature map and localization feature map into the mixed sample selection module. The mixed sample selection module calculates the consistency of the classification and localization tasks at each position on the feature map and uses it as the selection metric for label assignment. The mixed sample selection module assigns two sets of foreground samples to each instance. One set is selected by a one-to-many label assignment rule, and multiple foreground samples are used to enhance representation learning. The other set is selected by a one-to-one label assignment rule to select a unique deduplicated sample to achieve duplicate prediction removal.

[0058] The working method of the hybrid sample selection module is as follows:

[0059] For the classification and localization feature maps of different scales obtained from the detection head module, the classification and localization task quality of each instance i and each anchor point j (the location point on the feature map) on each feature map is estimated using classification and localization losses. The task consistency of each anchor point is calculated as a selection metric for label assignment, expressed as:

[0060]

[0061] Among them, M i,j This indicates the task consistency between the i-th instance and the j-th anchor point; C i,j C represents the spatial prior function, where C is the value of the anchor point j within the bounding box of the instance i. i,j =1, except for this, C i,j =0; L cls and L loc These represent the classification loss and the localization loss, respectively. and Let i and j represent the category and bounding box of the i-th instance in the image, respectively. and θ represents the category and prediction box of the j-th anchor point predicted by the network, respectively; θ represents the network model parameters; α and β represent the balancing parameters.

[0062] First, task consistency is used as a one-to-many label assignment rule to select foreground samples, and the specific metric is expressed as follows:

[0063]

[0064] Where Ψ(i) represents the index of the foreground sample matched by the i-th instance; TopK represents the selection function used to select the coordinates of the top k anchor points ranked by task consistency quality; the value of k is set to 9; and n represents the number of anchor points.

[0065] Secondly, using a one-to-one label assignment rule, a foreground sample with the highest task consistency score is selected from the chosen foreground samples, denoted as:

[0066]

[0067] Where Φ(i) represents the index of the foreground sample with the highest task consistency matched by the i-th instance; the argmax function is used to obtain the index of the maximum value.

[0068] E) The two sets of foreground samples obtained from the mixed sample selection module are input into the stage-aware soft weight adjustment module. The stage-aware soft weight adjustment module dynamically adjusts the contribution of the foreground samples in the network training stage, so as to fully perform representation learning in the early stage of training and learn to remove repeated predictions in the later stage of network training.

[0069] The working method of the stage-aware soft weight adjustment module is as follows:

[0070] Using the two sets of foreground samples obtained in step D), one set consists of k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, and the other set consists of the unique foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule. By dynamically adjusting the soft weights of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule, the contributions of these foreground samples in different network training stages t are dynamically adjusted to achieve a balance between representation learning and duplicate prediction removal. The dynamic adjustment process is achieved through the classification loss L. cls The implementation can be represented as follows:

[0071]

[0072] Where BCE represents the binary cross-entropy function, used to calculate the difference between the predicted probability and the true probability; Φ(i) represents the index of the unique foreground sample with the highest task consistency matched by the i-th instance selected by the one-to-one label assignment rule; Ψ(i) represents the index of the k foreground samples matched by the i-th instance selected by the one-to-many label assignment rule. The classification score predicted by the unique foreground sample with the highest task consistency matched by the i-th instance represents the classification score. This represents the predicted classification probability of the j-th foreground sample matched with the i-th instance; The soft weights represent the training time; t represents the current training time. It is expressed as follows:

[0073]

[0074] in, Δ represents the normalization parameter, which normalizes the weights of different samples to the same scale, while multiplying by the maximum Intersection over Union (IoU) between the predicted bounding box and the instance to highlight the different instance levels; t This is used to adjust the contribution of samples at different training times t, and is specifically represented as follows:

[0075]

[0076] Where, Δ t η is a variable that changes with time t; at time t, all foreground samples are assigned the same contribution rate; δ1 and δ2 control the lower and upper bounds of the contribution; η controls the linearity of the contribution decrease; N represents the total training time.

[0077] This embodiment also provides an end-to-end target detection system based on dynamic hybrid tag allocation, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0078] In this embodiment, the MSCOCO dataset is used with ResNet-50 as the backbone network for comparative validation. Table 1 shows the comparison results of the proposed method with other non-end-to-end and end-to-end methods on the MSCOCO dataset. As can be seen from Table 1, the proposed method has higher accuracy and robustness compared to other non-end-to-end and end-to-end methods, specifically reflected in mAP and AP. S AP M and AP L In the field of target detection, mAP stands for Average Precision and is used as the primary evaluation metric. S AP M and AP L These represent the average accuracy for detecting objects of different areas, specifically, AP. S (area<32 2 AP M (area<64 2 ) and AP S (area<128 2 )).

[0079] Table 1

[0080]

[0081]

[0082] In Table 1, RetinaNet corresponds to the method proposed by Y. Lin et al. (T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” in Proc. IEEE Int. Conf. Comp. Vis., 2017, pp. 2980–2988.)

[0083] FCOS corresponds to the method proposed by Tian et al. (Z. Tian, ​​C. Shen, H. Chen, and T. He, “FCOS: Fully convolutional onestage object detection,” in Proc. IEEE Int. Conf. Comp. Vis., 2019, pp. 9627–9636.)

[0084] FreeAnchor corresponds to the method proposed by Zhang et al. (X. Zhang, F. Wan, C. Liu, R. Ji, and Q. Ye, “FreeAnchor: Learning to match anchors for visual object detection,” Advances in Neural Inf. Process. Syst., vol. 32, 2019.)

[0085] FSAF corresponds to the method proposed by Zhu et al. (C. Zhu, Y. He, and M. Savvides, “Feature-selective anchor-free module for single-shot object detection,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019, pp. 840–849.)

[0086] SAPD corresponds to the method proposed by Zhu et al. (C. Zhu, F. Chen, Z. Shen, and M. Savvides, “Soft anchor-point object detection,” in Proc. Eur. Conf. Comp. Vis., 2020, pp. 91–107.)

[0087] ATSS corresponds to the method proposed by Zhang et al. (S. Zhang, C. Chi, Y. Yao, Z. Lei, and SZLi, “Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020, pp. 9759–9768.)

[0088] OneNet corresponds to the method proposed by Sun et al. (P. Sun et al., “What makes for end-to-endobject detection?” in Proc. Int. Conf. Mach. Learn., 2021, pp. 9934–9944.)

[0089] DeFCN corresponds to the method proposed by Wang et al. (J.Wang, L.Song, Z.Li, H.Sun, J.Sun, and N.Zheng, “End-to-end object detection with fully convolutional network,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021, pp. 15849–15858.)

[0090] O2F corresponds to the method proposed by Li et al. (S.Li, M.Li, R.Li, C.He, and L.Zhang, “One-to-fewlabel assignment for end-to-end dense detection,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2023, pp. 7350–7359.)

[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An end-to-end target detection method based on dynamic hybrid label allocation, characterized in that, Includes the following steps: 1) Obtain the training image dataset and training set annotation files; 2) Construct an end-to-end object detection network model based on dynamic hybrid label assignment. The end-to-end object detection model includes a preliminary feature extraction module, a multi-scale feature fusion module, a detection head module, a hybrid sample selection module, and a stage-aware soft weight adjustment module. Train the end-to-end object detection network model using a training image dataset and a training set annotation file to obtain generalizable model parameters. 3) Use the trained end-to-end object detection network model to detect objects, thus achieving end-to-end object detection; In step 2), the method for training the end-to-end object detection network model using the training image dataset and the training set annotation file is as follows: A) Input the training image dataset and training set annotation file into the preliminary feature extraction module in batches to extract feature maps, and then input the extracted feature maps into the multi-scale feature fusion module at the same time; B) The multi-scale feature fusion module extracts multi-scale features from the feature map through multiple feature fusion stages of different scales, and then upsamples the low-scale features to high-scale features layer by layer and fuses them with the high-scale features. C) The multi-scale feature map obtained by the multi-scale feature fusion module is input into the detection head module. The detection head module converts the high-dimensional feature map output by the multi-scale feature fusion module into a specific target detection result. The detection head divides the target detection task into a classification task and a localization task. In the classification task, the detection head classifies each position in the feature map to determine whether the position belongs to the set target category or to the background, and obtains the classification feature map. In the localization task, the detection head regresses the bounding box coordinates of the target to obtain the localization feature map; D) Input the obtained classification feature map and localization feature map into the mixed sample selection module. The mixed sample selection module calculates the consistency between the classification and localization tasks at each location on the feature map and uses it as the selection metric for label assignment. The mixed sample selection module assigns two sets of foreground samples to each instance. One set is selected by a one-to-many label assignment rule, and multiple foreground samples are used to enhance representation learning. The other set is selected by a one-to-one label assignment rule to select a unique deduplicated sample to achieve duplicate prediction removal. E) The two sets of foreground samples obtained from the mixed sample selection module are input into the stage-aware soft weight adjustment module. The stage-aware soft weight adjustment module dynamically adjusts the contribution of the foreground samples in the network training stage, so as to fully perform representation learning in the early stage of training and learn to remove repeated predictions in the later stage of network training.

2. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 1, characterized in that, In step 1), the training image dataset and the corresponding training set annotation file are obtained from the official open source dataset. The training image dataset is the MSCOCO object detection dataset.

3. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 1, characterized in that, In step A), the preliminary feature extraction module includes five preliminary feature extraction stages: Stage-1, Stage-2, Stage-3, Stage-4, and Stage-5. The input image undergoes feature extraction through each of these five preliminary feature extraction stages. The feature maps C3, C4, and C5 extracted by Stage-3, Stage-4, and Stage-5 have sizes of [missing information]. , and .

4. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 1, characterized in that, In step B), the multi-scale feature fusion module includes five multi-scale feature fusion stages: FPN-1, FPN-2, FPN-3, FPN-4, and FPN-5. After the input image passes through the preliminary feature extraction module, feature maps C3, C4, and C5 from the last three preliminary feature extraction stages are obtained. Feature map C3 is input into the first multi-scale feature fusion stage, FPN-1, to obtain the F3 layer feature map, with a size of [missing information]. The feature map C4 is input into the second multi-scale feature fusion stage FPN-2 to obtain the F4 layer feature map, which has a size of [missing information]. The feature map C5 is input into the third multi-scale feature fusion stage FPN-3 to obtain the F5 layer feature map, which has a size of [missing information]. The F5 layer feature map is input into the fourth multi-scale feature fusion stage FPN-4 to obtain the F6 layer feature map, which has a size of [missing information]. The F6 layer feature map is input into the fifth multi-scale feature fusion stage FPN-5 to obtain the F7 layer feature map, which has a size of [missing information]. The feature map of layer F7 is upsampled to the feature map of layer F6 and then added to the feature map of layer F6 to obtain the multi-scale feature map P6; the feature map of layer F6 is upsampled to the feature map of layer F5 and then added to the feature map of layer F5 to obtain the multi-scale feature map P5. The feature map of layer F5 is upsampled to the feature map of layer F4 and then added to the feature map of layer F4 to obtain the multi-scale feature map layer P4; the feature map of layer F4 is upsampled to the feature map of layer F3 and then added to the feature map of layer F3 to obtain the multi-scale feature map P3.

5. The end-to-end object detection method based on dynamic hybrid tag assignment of claim 1, wherein, In step C), the detection head module includes five detection heads: Head-1, Head-2, Head-3, Head-4, and Head-5; and five multi-scale feature maps P3-P7 obtained from the multi-scale feature fusion module, with sizes of... , , , and The multi-scale feature maps P3-P7 are input into the corresponding detection heads respectively; after passing through the first detection head Head-1, P3 outputs the classification feature map Cls-1 and the localization feature map Loc-1, with sizes of respectively and After passing through the second detection head Head-2, P4 outputs a classification feature map Cls-2 and a localization feature map Loc-2, with sizes of [sizes to be filled in]. and After passing through the third detection head, Head-3, P5 outputs a classification feature map Cls-3 and a localization feature map Loc-3, with sizes of [sizes to be filled in]. and After passing through the fourth detection head, Head-4, P6 outputs a classification feature map Cls-4 and a localization feature map Loc-4, with sizes of [sizes to be filled in]. and After passing through the fifth detection head, Head-5, P7 outputs a classification feature map Cls-5 and a localization feature map Loc-5, with sizes of [sizes to be filled in]. and .

6. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 1, characterized in that, In step D), the working method of the mixed sample selection module is as follows: For the classification and localization feature maps of different scales obtained from the detection head module, the classification and localization task quality of each instance i and each anchor point j on each feature map is estimated using classification and localization losses. The task consistency of each anchor point is calculated as a selection metric for label assignment, expressed as: in, Indicates the first The first instance and the first Task consistency at each anchor point; Represents the spatial prior function, when the j-th anchor point is at the th... When inside the bounding box of an instance, In addition to this, ; and These represent the classification and localization losses, respectively. and They represent the first and second images in the picture. The category and location box of each instance; and These represent the category and prediction box of the j-th anchor point predicted by the network, respectively; and Indicates the balance parameters; First, task consistency is used as a one-to-many label assignment rule to select foreground samples, and the specific metric is expressed as follows: in, Indicates the first The index of the foreground sample matched by each instance; This represents a selection function used to prioritize tasks based on their consistency and quality. The anchor point coordinates of each; Indicates the number of anchor points; Secondly, using a one-to-one label assignment rule, a foreground sample with the highest task consistency score is selected from the chosen foreground samples, denoted as: in, Indicates the first The index of the foreground sample that matches the highest task consistency among the instances; The function is used to get the index of the maximum value.

7. The end-to-end target detection method based on dynamic hybrid label allocation according to claim 6, characterized in that, In step E), the working method of the stage-aware soft weight adjustment module is as follows: Using the two sets of foreground samples obtained in step D), one set is the first set selected by the one-to-many label assignment rule. Matching instances The foreground samples are selected by a one-to-one label assignment rule. The unique foreground sample with the highest task consistency is matched against the nth instance; the nth instance selected by the one-to-many label assignment rule is dynamically adjusted. Matching instances The soft weights of each foreground sample are dynamically adjusted at different stages of network training. The contribution of the model is used to achieve a balance between representation learning and duplicate prediction removal; the dynamic adjustment process is achieved through classification loss. The implementation is represented as follows: Wherein, BCE represents the binary cross-entropy function, which is used to calculate the difference between the predicted probability and the true probability; This indicates the first label selected by the one-to-one label assignment rule. The index of the unique foreground sample that matches each instance and has the highest task consistency; This indicates the first label selected by the one-to-many label assignment rule. Matching instances An index of foreground samples; Indicates the first The classification score predicted by the unique foreground sample with the highest task consistency among the matched instances; Indicates the first The classification probability predicted by the j-th foreground sample matched with each instance; Indicates soft weight; This indicates the current training time; where soft weights are... It is expressed as follows: in, This represents the normalization parameter, which normalizes the weights of different samples to the same scale, while multiplying by the maximum Intersection over Union (IoU) between the predicted box and the instance to highlight the different sample instance levels; Used at different training times The contribution of the adjusted sample is expressed as follows: in, It is a time The changing variable; in the first At any given time, all foreground samples are assigned the same contribution rate; and Control the lower and upper limits of contributions; Control the linearity of the decrease in contribution; N represents the total training time.

8. An end-to-end target detection system based on dynamic hybrid label allocation, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-7.

Images