Grouping-based hungarian matching and iou-aware target detection method

Abstract

The application provides a target detection method based on group Hungarian matching and loU perception, and relates to the technical field of image detection. The target detection method based on group Hungarian matching and loU perception adopts multi-scale Hungarian matching and multi-scale query selection, avoids the situation that final prediction targets are all inclined to large targets, and improves small target precision; meanwhile, an IoU branch is added in the training stage, and IoU confidence and classification confidence are combined as a comprehensive confidence evaluation index, in the Topk stage, higher-quality initial anchor frames can be selected, the model can be fitted faster, and therefore higher precision improvement can be achieved at 12 epochs.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, specifically to a target detection method based on grouped Hungarian matching and IoU perception. Background Technology

[0002] Object detection is one of the fundamental tasks in computer vision, involving the identification and localization of objects in images. Modern object detection algorithms mainly fall into two categories: CNN-based algorithms and transformer-based algorithms. However, CNN-based algorithms suffer from drawbacks such as the need for post-processing (NMS) to manually set thresholds and long training epochs. Transformer-based object detection algorithms address these issues by optimizing the model structure. The DETR series represents the transformer architecture paradigm, but currently, the DETR series exhibits low accuracy in detecting small objects.

[0003] The study found the following two problems: 1. Most of the receptive fields of DETR come from deep feature maps, which leads to the final predicted targets being biased towards large targets, resulting in low detection accuracy for small targets; 2. The initial anchor box quality of DETR is low (the classification confidence and IoU confidence of the anchor box are inconsistent), which means that more training epochs are needed to make the model more accurate.

[0004] To address this, we developed a novel target detection method based on grouped Hungarian matching and IoU awareness. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a target detection method based on grouped Hungarian matching and IoU perception, which solves the problem of low accuracy in small target detection that exists in the current DETR series of technologies.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, this invention provides the following technical solution: a target detection method based on grouped Hungarian matching and IoU awareness, wherein the algorithm model is based on the DINO model and follows the DETR paradigm, and includes the following specific steps:

[0009] The image undergoes four layers of feature extraction from the backbone, and the positional encoding is fused and input into the encoder to perform simultaneous intra-scale and cross-scale feature fusion.

[0010] We obtain tokens for predicting objects of different sizes at different scales by using different proportions of the tokens. Specifically, we group the tokens at different scales according to proportions and select the tokens with the highest confidence at each scale. The tokens are then passed through three branch prediction heads, including the bounding box branch prediction head, the category branch prediction head, and the IoU branch prediction head. The IoU branch prediction value and the category confidence are combined together as a new evaluation criterion.

[0011] Then, the multi-scale query selection module is executed. The class prediction head and IoU prediction head are used to generate class scores and IoU scores respectively. The obtained class scores and IoU scores are combined as a new confidence evaluation index. Then, the top K is selected within different groups to find the token with the highest confidence. The token is sent to the bounding box prediction head to generate the initial prediction box. Then, the prediction box corresponding to the token is used as the initial box and the input data of the decoder. The prediction box is sent to the decoder part and fused with the position encoding to form the query. Memory-related features are further extracted, and then multi-scale label allocation training is performed.

[0012] Furthermore, an algorithm is provided for Hungarian grouping matching of ground truth boxes of different scales and tokens of different scales:

[0013] The value ranges from 0 to 1.

[0014] (1)

[0015] (2)

[0016] (3)

[0017] (4)

[0018] (5)

[0019] (6)

[0020] Where n is the number of ground truth (GT) boxes and m is the number of groups.

[0021] Furthermore, an algorithm for a multi-scale selection module is provided, including:

[0022] It is The flattened form in the spatial direction, used as input data for the module, is shown in the following formula:

[0023] (7)

[0024] This is the i-th layer feature map, with a scale of (hi, wi), where m is the layer number of the feature map, and PredClassHead is the class prediction head. Ti is flattened and concatenated as the overall input, which is then processed by PredClassHead to obtain the class score for each token. As shown in the formula below:

[0025] (8)

[0026] It is the confidence level of C taking the maximum value in the last dimension, representing the maximum probability of the final predicted class of the token;

[0027] A token predicts the probability that an object belongs to different categories. The maximum value is taken as the confidence score, which is the probability of the highest predicted category. The formula is as follows:

[0028] (9)

[0029] Where dim represents the dimension of C, DINO concatenates feature maps from different scales, then predicts the confidence levels at all scales through a category prediction branch. The top N tokens are selected based on their confidence levels, and the predicted location information (indices) of these tokens is used as the initial location information (x, y, w, h) for the downstream query. This refers to the indices of the top N confidence levels in D, obtained through the top K algorithm. These indices also represent the token serial numbers, as shown in the following formula:

[0030] (10)

[0031] N is the number of tokens selected from the top K. In the model, the tokens are first divided according to the proportion of the number of tokens in different feature maps to the total number of tokens in all feature maps. Finally, the top-ranked tokens are selected. As shown in the formula below:

[0032] (11)

[0033] (hi, wi) represents the size of the i-th layer feature map. There are a total of m layers of feature maps. Similar to DINO, a flattening operation is performed to flatten the feature maps of different layers separately. The following formulas represent the flattened forms of different feature maps:

[0034] (12)

[0035] The m represents the feature map of the i-th layer, and m represents the scale layer. Each scale is flattened and independently fed into the class prediction head for class confidence prediction. The following is a public notice:

[0036] (13)

[0037] m: the number of layers in the feature map; Ti: the unfolded form of the feature map at different layers; obtain the maximum value of the class confidence corresponding to the token at each layer at each scale. As shown in the formula below:

[0038] (14)

[0039] Where dim: is the number of categories. These are the predicted class probabilities for each of the above tokens. A top-K ranking is then performed at each scale to obtain the top-ranked tokens for each class. The location information corresponding to the token: As shown in the formula below:

[0040] (15)

[0041] Finally, the location index information at all scales is concatenated together to obtain... As shown in the formula below:

[0042] (16).

[0043] Furthermore, an algorithm for predicting IoU branch values ​​is provided, including:

[0044] The predicted IoU value is used as the ground truth value, and L2 loss is applied during training, as shown in the following formula:

[0045] (17)

[0046] in For feature points, in the topK stage, a comprehensive evaluation metric combining classification and IoU values ​​is used for filtering. In the inference stage, this combination is used as a threshold metric for filtering bounding boxes, as shown in the following formula:

[0047] (18)

[0048] Where s is the predicted classification score and u is the predicted IoU score.

[0049] (III) Beneficial Effects

[0050] This invention provides a target detection method based on grouped Hungarian matching and IoU awareness, which has the following beneficial effects:

[0051] 1. This target detection method based on grouped Hungarian matching and IoU awareness adopts multi-scale Hungarian matching and multi-scale query selection, which avoids the situation where the final predicted targets are all biased towards large targets and improves the accuracy of small targets.

[0052] 2. This object detection method based on grouped Hungarian matching and IoU awareness adds an IoU branch during the training phase and combines IoU confidence and classification confidence as a comprehensive confidence evaluation index. In the Topk phase, it can select higher quality initial anchor boxes, allowing the model to fit faster, thereby achieving a higher accuracy improvement after 12 epochs. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall model detection method of the present invention;

[0054] Figure 2 This is a structural diagram of the model of the present invention;

[0055] Figure 3 This is a diagram of the grouped Hungarian matching method of the present invention;

[0056] Figure 4 This is a diagram of the multi-scale selection module of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example

[0059] like Figure 1 - Figure 4 As shown, this embodiment of the invention provides a target detection method based on grouped Hungarian matching and IoU awareness. The algorithm model is based on the DINO model and follows the DETR paradigm, and includes the following specific steps:

[0060] Step 1: The image undergoes four layers of feature extraction from the backbone, and the positional encoding is fused and input into the encoder to perform simultaneous intra-scale and cross-scale feature fusion.

[0061] Step 2: Obtain tokens for predicting objects of different sizes at different scales by using different proportions of the tokens. That is, group the tokens proportionally at different scales and filter the top K tokens with the highest confidence at each scale. Then, pass the tokens through three branch prediction heads, including the bounding box branch prediction head, the class branch prediction head, and the IoU branch prediction head. Combine the IoU branch prediction value and the class confidence as a new evaluation criterion.

[0062] In this approach, considering that different queries should be responsible for finding objects of different sizes, a grouping approach is adopted. The ground truth bounding boxes of different scales and the tokens of different scales are matched using Hungarian grouping. Subsequently, the ground truth bounding boxes of different scales and the queries of different groups can be matched using Hungarian grouping, thereby guiding the tokens of different scales and the queries of different groups to focus on objects of different sizes.

[0063] Specifically, an algorithm is provided for Hungarian grouping matching of ground truth boxes of different scales and tokens of different scales:

[0064] This set of parameters is a threshold division interval that needs to be determined manually. It is a relative threshold with a value range of 0 to 1. Next, the relative threshold is converted into an absolute threshold as shown in the following formula (1):

[0065] (1)

[0066] To determine which group the ground truth (GT) boxes should be assigned to, where n is the number of GT boxes, we obtain the absolute area of ​​the GT boxes as shown in formula (2):

[0067] (2)

[0068] if Then the GT box is assigned to the i-th group, and the anchor box in the i-th group is processed by position encoding and MLP to obtain the position embedding information as shown in the following formula (3):

[0069] (3)

[0070] Then, initialize a learnable content information, concatenate the position information and content information to obtain the initial query corresponding to the i-th group as shown in formula (4):

[0071] (4)

[0072] Perform Hungarian matching on the query generated in the i-th group and the corresponding ground truth bounding box to obtain the position indices of the positive and negative samples in the i-th group as shown in formula (5):

[0073] (5)

[0074] Where m is the number of groups and n is the number of true bounding boxes. Finally, the indices of the positive and negative samples from all groups are concatenated to form the final positive and negative samples, as shown in formula (6) below.

[0075] (6);

[0076] Step 3: Then execute the multi-scale query selection module. Use the class prediction head and IoU prediction head to generate class scores and IoU scores respectively. Combine the obtained class scores and IoU scores as a new confidence evaluation index. Then, perform topK filtering within different groups to select the token with the highest confidence. Feed the token into the box prediction head to generate the initial prediction box. Then, use the prediction box corresponding to the token as the initial box and the input data of the decoder. Feed the prediction box into the decoder part and fuse it with the position encoding to form a query. Further extract memory-related features and then perform multi-scale label allocation training.

[0077] Specifically, an algorithm for a multi-scale selection module is provided, including:

[0078] It is The flattened form in the spatial direction, used as input data for the module, is shown in the following formula:

[0079] (7)

[0080] This is the i-th layer feature map, with a scale of (hi, wi), where m is the layer number of the feature map, and PredClassHead is the class prediction head. Ti is flattened and concatenated as the overall input, which is then processed by PredClassHead to obtain the class score for each token. As shown in the formula below:

[0081] (8)

[0082] It is the confidence level of C taking the maximum value in the last dimension, representing the maximum probability of the final predicted class of the token;

[0083] A token predicts the probability that an object belongs to different categories. The maximum value is taken as the confidence score, which is the probability of the highest predicted category. The formula is as follows:

[0084] (9)

[0085] Where dim represents the dimension of C, DINO concatenates feature maps from different scales, then predicts the confidence levels at all scales through a category prediction branch. The top N tokens are selected based on their confidence levels, and the predicted location information (indices) of these tokens is used as the initial location information (x, y, w, h) for the downstream query. This refers to the indices of the top N confidence levels in D, obtained through the top K algorithm. These indices also represent the token serial numbers, as shown in the following formula:

[0086] (10)

[0087] N is the number of tokens selected from the top K. In the model, the tokens are first divided according to the proportion of the number of tokens in different feature maps to the total number of tokens in all feature maps. Finally, the top-ranked tokens are selected. As shown in the formula below:

[0088] (11)

[0089] (hi, wi) represents the size of the i-th layer feature map. There are a total of m layers of feature maps. Similar to DINO, a flattening operation is performed to flatten the feature maps of different layers separately. The following formulas represent the flattened forms of different feature maps:

[0090] (12)

[0091] The m represents the feature map of the i-th layer, and m represents the scale layer. Each scale is flattened and independently fed into the class prediction head for class confidence prediction. The following is a public notice:

[0092] (13)

[0093] m: the number of feature map layers; Ti: the format of the displayed features at different layers; and the maximum value of the category confidence for each layer's token is obtained at each scale. As shown in the formula below:

[0094] (14)

[0095] Where dim: is the number of categories. These are the predicted class probabilities for each of the above tokens. A top-K ranking is then performed at each scale to obtain the top-ranked tokens for each class. The location information corresponding to the token: As shown in the formula below:

[0096] (15)

[0097] Finally, the location index information at all scales is concatenated together to obtain... As shown in the formula below:

[0098] (16);

[0099] In DINO, top-k selects the predicted location information of 900 tokens as the query location information. However, it can be found that most of the selected tokens have relatively large receptive fields. This causes most of the initialized anchor boxes to float around large objects. This means that the anchor boxes responsible for predicting small objects may be around large objects or in the background, making it difficult to optimize for small objects. Therefore, we use different proportions to obtain tokens for predicting objects of different sizes at different scales.

[0100] Based on these characteristics, the tokens are predicted through three branches. Furthermore, it was observed that there are significant differences between the classification confidence and IoU confidence of anchor boxes in the topK stage. That is, the classification confidence and IoU confidence of a certain anchor box are not consistent. There are some poor-quality boxes with high category scores but low IoU scores, while other poor-quality boxes have low category scores but high IoU scores. High-quality anchor boxes have both high category scores and high IoU scores, meaning that the predicted category of the box is accurate and the localization of the box is also more accurate.

[0101] After analyzing the class confidence and IoU confidence of the predicted bounding boxes generated from the top K tokens in DINO, it was found that most of the blue dots were distributed in the upper left and lower right corners of the image. If the dots were closer to the central line, the quality of the bounding box was better. To address this issue, an independent fully connected branch will be used to perceive the IoU value of each predicted bounding box, providing an algorithm for IoU branch prediction, including:

[0102] To obtain the predicted IoU value, we use the IoU value between the predicted box and the ground truth box as the ground truth information and train it using L2 loss, as shown in the following formula (11):

[0103] (17)

[0104] in For feature points, in the topK stage, the classification score is no longer simply used as the standard for ranking. Instead, a comprehensive evaluation index combining the classification and IoU values ​​is used for screening. In the inference stage, this combination method will also be used as the threshold evaluation index to screen the bounding boxes. As shown in the following formula (18).

[0105] (18)

[0106] s is the predicted classification score, and u is the predicted IoU score.

[0107] ablation experiment

[0108]

[0109] Table 1: Ablation experiments for each component, epoch=12, where SLM: scalelabelmatch module, SQS: scalequeryselection module, and IoU: IoUbranch module.

[0110] The ablation studies of the designed components were conducted over 12 epochs. Both components are indispensable. The SLM part hierarchically separates and distinguishes tokens with different receptive fields and provides grouping ratios for subsequent query grouping. This ensures that the initial distribution of subsequent queries falls precisely around the predictions of their respective tokens, thereby further improving the model's positional convergence speed. Different groups of queries, through cross-attention adaptive acquisition of feature information from objects of different scales, can predict objects of different scales, thus reducing competition between queries predicting different scales. Similarly, between queries predicting the same scale within the same group, competition is also reduced due to the significantly reduced number of queries in the same group. The combination of SLM and SQS significantly improved the model's performance on small and medium-sized targets, increasing AP by 1.8% for small targets and 0.2% for medium targets. We can see that simply adding the SLM part has a significant impact on the model because grouping only tokens without grouping queries in the same way affects the query's determination of the target scale, which is detrimental to model accuracy.

[0111] By adding the IoU branch, we can clearly see that the accuracy of AP75 is further improved by 0.2%, and the accuracy of medium and large targets is also improved by 0.8% and 0.5%, respectively. We speculate that this is because medium and large targets have sufficient semantic and target information, so the query responsible for predicting medium and large targets converges faster. However, interestingly, adding only the IoU branch causes the performance of AP50 to drop below the original value of DINO. This may be because when the IoU threshold is low, the shape difference of the bounding boxes is large, which will affect the suppression and convergence of downstream boxes. However, our previous method can effectively alleviate this problem.

[0112]

[0113] Table 2: Ablation epochs=12 for various grouping ratios in the scalelabelmatch module

[0114] We scaled the grouping of ground truth (GT) boxes in the SLM module to ensure that GT boxes of different sizes are matched one-to-one with the queries responsible for predicting that group, thus correctly classifying positive and negative samples. We used four scale feature layers, so we needed to divide the queries into four groups. When the area of ​​a GT box falls within a certain threshold range, it is labeled with the queries within that group. Therefore, the threshold division has a sensitive impact on model performance. We tried different combinations, some of which led to a performance decrease. After multiple ablation studies, we finally adopted the parameter combination [0.2, 0.6, 0.8, 1]. Ultimately, we used the square of this combination, which is [0.04, 0.36, 0.64, 1], as the threshold combination. Under the condition of 12 epochs, the accuracy reached 49.4% AP. We also found that this combination had the best compatibility with large, medium, and small targets.

[0115] Experimental results

[0116] The results were compared with those of other models and the model in this application using DINO as a baseline. The model in this application, like DINO, uses 900 queries. The models shown with * are all built on the mmdetection framework, and the model data were all trained on a single 3090 GPU.

[0117] We used DINO as a baseline for our experiments. This baseline used deformable-transformer as the backbone and employed methods such as CDN and lookforward-twice to improve performance. We followed DINO's approach and adopted its techniques.

[0118]

[0119] Table 3: Main Results

[0120] For the backbone, we used a ResNet50 backbone network with four scales as input. The results are shown in Table 2. Under the condition of epochs=12, our method improved the accuracy of small targets by 2.3% and medium targets by 1.2%, but found that the performance of large targets decreased by 0.5%. Based on our experience, it is possible that the convergence speed of large targets is slowed down by the parameter setting of the partition ratio in our proposed model. When our model is used with epochs=24 and epochs=36, the accuracy of large targets also increases. It can be seen that our model pays more attention to the small target region and improves the convergence speed of small targets.

[0121] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A target detection method based on packet Hungarian matching and IoU awareness, the algorithm model is based on DINO model and is in DETR paradigm, characterized in that, Comprise the following specific steps: Pictures from backbone four layers of feature extraction, location coding fusion input to the encoder, scale and cross-scale features are fused at the same time; Grouping topK according to the proportion on different scales, the highest confidence token on each scale is selected, and the token is predicted through three branch prediction heads, including bbox branch prediction head, category branch prediction head and IoU branch prediction head, the IoU branch prediction value and the category confidence are combined together as a new evaluation standard; The multi-scale query selection module is executed, the category prediction head and the IoU prediction head are used to form the category score and the IoU score respectively, the obtained category score and the IoU score are combined together as a new confidence evaluation index, then topK is selected in different groups to select the token with the highest confidence, the token is sent into the frame prediction head to generate the initialized prediction frame, then the prediction frame corresponding to the token is taken as the input data of the decoder, the prediction frame is sent into the decoder part and the position coding fusion to form the query, further extract the memory related features, and then perform multi-scale label assignment training; An algorithm for determining GT boxes of different scales and token of different scales to perform hungarian grouping matching is provided: , the value range is between 0~1, this set of parameters is a threshold division interval, which needs to be determined manually, and is a relative threshold; The relative threshold is converted into an absolute threshold as shown in the following formula (1): (1) In order to determine which group the GT box is divided into, n is the number of GT boxes, the absolute area of the GT box is obtained as shown in the following formula (2): (2) If Then the GT box is assigned to the i-th group, and the anchor box of the i-th group is encoded by the position and MLP to obtain the position embedding information as shown in equation (3): (3) Then the position information and the content information are spliced to obtain the initialization query corresponding to the i-th group as shown in the following formula (4): (4) Hungarian matching is performed on the query generated by the i-th group and the corresponding GT box to obtain the position index of the positive and negative samples corresponding to the i-th group as shown in the following formula (5): (5) Wherein, m is the number of groups, n is the number of real boxes, a plurality of queries of the i-th group and a plurality of GT boxes divided into the i-th group are hungarian matched to form the result of hungarian matching of the i-th group, the number of positive samples in this group is equal to the total number of GT, and the number of negative samples is equal to the total number of queries minus the number of positive samples; (6) The final result is the combined set of positive samples of m groups and the combined set of negative samples of m groups to form positive and negative samples respectively.

2. The method of claim 1, wherein, An algorithm of a multi-scale selection module is provided, comprising: is to be After flattening in the spatial direction, the form is taken as the input data of the module as shown in the following equation: (7) is the i-th layer feature map, the scale size is (hi, wi), m is the number of layers of the feature map, and PredClassHead is a class prediction head. The Ti is spliced after being flattened as a total input to obtain the class score of each token through the PredClassHead as shown in the following formula: (8) is the confidence that C takes the maximum value in the last dimension, representing the maximum probability of the final predicted class of the token; A token can predict the probability of the object C being different categories, and the maximum value is the probability of the maximum category prediction, which is the confidence as shown in the following formula: （9） where dim is the dimension of C, DINO is to splice the feature maps on different scales together, and then predict the confidence of all scales through the category prediction branch, sort according to the confidence, select the top N tokens, and take the position information indices predicted by the token as the position information (x, y, w, h) of the downstream query initialization, It is referred to as the subscript of the top N confidence in D through topK, which also represents the token sequence number as shown in the following formula: (10) N is the number of topK selected, in the model, first according to the proportion of the token number of different feature maps in the total token number of all feature maps, the corresponding number is divided, and finally the number of top-ranked tokens to be selected is As shown in the following formula: (11) (hi,wi) represents the size of the i-th layer feature map, and there are m layer feature maps in total, and the flattening operation is performed as in DINO, and different layer feature maps are flattened respectively, The flattened form of different feature maps is represented as follows: (12) representing the i-th layer feature map, m represents the scale layer number, and a single scale is flattened into a class prediction head for class confidence prediction as shown in the following formula: (13) m: the number of layers of the feature map, Ti: the form of the feature map after unfolding, and the maximum value of the class confidence of each layer token is obtained at each scale As shown in the following formula: (14) where dim: is the number of categories, is the category probability predicted by each token above, and topK is performed on each scale to obtain the position information of the top category token on each scale: , as shown in the following formula: (15) Finally, the position index information in all scales is spliced together to obtain as shown in the following formula: (16）。 3. The method of claim 2, wherein, An algorithm of an IoU branch prediction value is provided, comprising: For the predicted IoU value, the IoU value corresponding to the predicted box and the real box is taken as the real information, and the L2 loss is used for training, as shown in the following formula: (17) wherein is a feature point, in the topK stage, the classification and IoU value are combined together as a comprehensive evaluation index to filter, in the inference stage, the combination is used as a threshold evaluation index to filter the box, as shown in the following formula: (18) Wherein, s is the predicted classification score, and u is the predicted IoU score.

Images