Target association method based on normalized intersection-union ratio

Abstract

The invention relates to a target association method based on a normalized intersection-union ratio. The method comprises the following steps: obtaining a source target set; performing target detection on the to-be-processed image to obtain a plurality of candidate bounding boxes and confidence thereof; extracting the candidate bounding boxes with the confidence higher than a set confidence threshold to form a candidate target set; constructing a cost matrix by taking a normalized intersection-to-union ratio of a bounding box corresponding to each target to be associated in a source target set and each selected bounding box as an element, and solving the selected bounding box which is most matched with the target to be associated by the normalized intersection-to-union ratio # imgabs0 # based on the cost matrix by utilizing a Hungary algorithm, obtaining a plurality of optimal matching pairs; and determining whether the targets are associated or not according to the normalized intersection-to-union ratio of each group of optimal matching pairs. According to the invention, the accuracy of the target tracking association task can be obviously improved.

Description

Technical Field

[0001] The invention relates to a target association tracking technology, and in particular to a target association method based on normalized intersection-over-union ratio. Background Art

[0002] The target association method based on Intersection over Union (IoU) is a commonly used technology in fields such as target detection and target tracking. It is used to evaluate the similarity between the predicted bounding box and the true bounding box, and to determine the correspondence between the predicted target and the true target in multi-target tracking or data association problems.

[0003] The target association method based on IoU has been widely used in the field of target detection and tracking due to its simplicity and effectiveness, but it also has some shortcomings. IoU cannot accurately reflect the overlap of two boxes. In some cases, even if the IoU value is the same, the actual overlap of the two boxes may be very different, which limits the effectiveness of IoU as a target association. In addition, IoU is insensitive to the scale of the object, and in the target association task, the IoU optimization of two target boxes or compressed masks is not completely equivalent to whether they are the same target, which may cause problems in the association process. For example, when there is a containment relationship, if the sizes of the two target boxes are very different, their IoU values ​​may not accurately reflect the association relationship between the two targets, and may even be misjudged as two different targets. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a target association method based on normalized intersection-over-union ratio, which can significantly improve the accuracy of target tracking association tasks.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a target association method based on normalized intersection-over-union ratio, comprising the following steps:

[0006] Acquire a source target set, wherein the source target set includes a target to be associated and a bounding box thereof;

[0007] Perform target detection on the image to be processed and obtain several candidate bounding boxes and their confidence scores;

[0008] Extracting the candidate bounding boxes whose confidence is higher than a set confidence threshold to form a candidate target set;

[0009] The normalized intersection-over-union (IoU) of the bounding boxes corresponding to the targets to be associated in the source target set and the selected bounding boxes in the candidate target set is calculated. norm Construct a cost matrix for the element;

[0010] Based on the cost matrix, the Hungarian algorithm is used to solve the selected bounding boxes that best match each of the to-be-associated targets in the source target set to obtain multiple best matching pairs;

[0011] According to the normalized intersection-over-union ratio IoU of each group of the best matching pairs norm Determines whether the target is relevant.

[0012] Furthermore, the normalized intersection-over-union ratio IoU norm Calculated by the following formula:

[0013]

[0014] Among them, A is the bounding box corresponding to the associated target in the source target set, and B is the selected bounding box in the segmentation target set.

[0015] Furthermore, the normalized intersection-over-union (IoU) of each group of the best matching pairs is norm Determine if the target is relevant, including:

[0016] If the normalized intersection-over-union ratio IoU of the current best matching pair norm If the value is greater than the set intersection-over-union ratio threshold, it is considered that the corresponding target to be associated is associated with the candidate bounding box.

[0017] Furthermore, the normalized intersection-over-union (IoU) of the bounding boxes corresponding to the targets to be associated in the source target set and the selected bounding boxes in the candidate target set is calculated as follows: norm Construct a cost matrix for the elements, including:

[0018] The bounding boxes corresponding to the targets to be associated in the source target set are rows, and the selected bounding boxes in the candidate target set are columns. The normalized intersection-over-union (IoU) of each row and each column is calculated. norm , and obtain the cost matrix.

[0019] Furthermore, the method further includes the steps of initializing the selected bounding boxes in the candidate target set that do not match any target to be associated as new targets to be associated, and adding the new targets to be associated to the source target set.

[0020] The present invention also provides a target association method based on normalized intersection-over-union ratio, comprising the following steps:

[0021] Acquire a source target set, wherein the source target set includes a target to be associated and a mask thereof;

[0022] Perform target segmentation on the image to be processed and obtain several candidate masks and their confidences;

[0023] Extracting the candidate masks whose confidence is higher than a set confidence threshold to form a candidate target set;

[0024] The normalized intersection-over-union (IoU) of the masks corresponding to the targets to be associated in the source target set and the selected masks in the candidate target set is calculated. norm Construct a cost matrix for the element;

[0025] Based on the cost matrix, the Hungarian algorithm is used to solve the selected mask that best matches each of the to-be-associated targets in the source target set, to obtain a plurality of best matching pairs;

[0026] According to the normalized intersection-over-union ratio IoU of each group of the best matching pairs norm Determines whether the target is relevant.

[0027] Furthermore, the normalized intersection-over-union ratio IoU norm Calculated by the following formula:

[0028]

[0029] Among them, A is the mask corresponding to the associated target in the source target set, and B is the selected mask in the segmentation target set.

[0030] Furthermore, the normalized intersection-over-union (IoU) of each group of the best matching pairs is norm Determine if the target is relevant, including:

[0031] If the normalized intersection-over-union ratio IoU of the current best matching pair norm If the value is greater than the set intersection-over-union ratio threshold, it is considered that the corresponding target to be associated is associated with the candidate mask.

[0032] Furthermore, the normalized intersection-over-union (IoU) of the masks corresponding to the targets to be associated in the source target set and the selected masks in the candidate target set is calculated as follows: norm Construct a cost matrix for the elements, including:

[0033] The masks corresponding to the targets to be associated in the source target set are taken as rows, and the selected masks in the candidate target set are taken as columns. The normalized intersection-over-union ratio IoU of each row and each column is calculated. norm , and obtain the cost matrix.

[0034] Furthermore, the method further includes the steps of initializing the selected mask in the candidate target set that does not match any target to be associated as a new target to be associated, and adding the new target to be associated to the source target set.

[0035] Beneficial Effects

[0036] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: The present invention aims at the irrationality and ineffectiveness of target association based on IoU under the condition of large size difference. The present invention analyzes the relationship between IoU value and target frame overlap under different target size difference conditions, and designs a new normalized intersection-over-union ratio IoU. norm , a target association method based on the normalized intersection-over-union ratio was proposed, which achieved significant improvement in target tracking association tasks during scene mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of target frame overlap in the first embodiment of the present invention;

[0038] Figure 2 : is a relationship diagram between IoU and target overlap (β) under different target size difference conditions (different values ​​of α) of the first embodiment of the present invention;

[0039] Figure 3 is a flow chart of a first embodiment of the present invention;

[0040] Figure 4 is a flow chart of a second embodiment of the present invention;

[0041] Figure 5 It is a visual comparison diagram of scene reconstruction segmentation and recognition quality of different association methods according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0043] The first embodiment of the present invention relates to a target association method based on the normalized intersection-over-union ratio, comprising the following steps:

[0044] S0 obtains the source target set;

[0045] S1 performs target detection on the image to be processed and obtains several candidate bounding boxes and their confidences;

[0046] S2 extracts candidate bounding boxes whose confidence is higher than the set confidence threshold to form a candidate target set;

[0047] S3 is the normalized intersection-over-union ratio (IoU) of the bounding boxes corresponding to each target to be associated in the source target set and each selected bounding box in the candidate target set. normConstruct a cost matrix for the element; normalize the intersection-over-union ratio IoU norm Calculated by the following formula:

[0048]

[0049] Among them, A is the bounding box corresponding to the associated target in the source target set, and B is the selected bounding box in the segmentation target set;

[0050] S4 uses the Hungarian algorithm to solve the selected bounding boxes that best match each target to be associated in the source target set based on the cost matrix, and obtains multiple best matching pairs;

[0051] S5 is based on the normalized intersection-over-union ratio IoU of each group of best matching pairs norm Determines whether the target is relevant.

[0052] Next, we combine the normalized intersection-over-union ratio IoU norm The present embodiment is further described with reference to the principle of FIG.

[0053] When tracking target association, the IoU value may not accurately reflect the association between two targets, and may even be misjudged as two different targets. Figure 1 As shown in , in cases 1 and 3, IoU can basically reflect the degree of overlap of two target frames, and it is reasonable and effective to judge whether they are the same target based on this. However, in cases 2 and 4, it is very unreasonable to reflect the degree of overlap of two target frames based on IoU and judge whether they are the same associated target, because in cases 2 and 4, the values ​​of IoU are relatively small, and it is difficult to judge them as the same target. However, in actual detection or tracking applications, due to local perspective or detection errors, they are very likely to belong to the same associated target. Therefore, many target association methods based on IoU in practical applications have theoretical defects and need to be improved.

[0054] Assume that the bounding boxes of two targets are A and B, and their bounding box sizes have the following relationship:

[0055] |A|=α|B|

[0056] Formula 1

[0057] Where 0<α≤1, that is, the area of ​​A is smaller than that of B. At this time, the intersection of A and B (reflecting the overlap of the target box) can be expressed as:

[0058] A∩B=β|A|=αβ|B|

[0059] Formula 2

[0060] It must satisfy 0<β≤1. Then the IoU calculation formula of A and B is as follows:

[0061]

[0062] When α is taken at different values, the relationship between IoU and the overlap between two target boxes (β) will change greatly. Figure 2 As shown in . Only when α=1, that is, the bounding box sizes of targets a and b are equal or similar, the relationship between IoU and target overlap (β) is approximately linearly positively correlated. At this time, it is reasonable and effective to judge whether the two are associated based on IoU. However, when α takes values ​​of 0.5, 0.2, 0.1, and 0.05, even if the overlap of the two target boxes (β) is close to 1, the corresponding IoU value is also capped at α. In other words, when the size difference between the two targets is large, their IoU must not exceed the size multiple between them (less than 1). The larger the size difference, the smaller the IoU.

[0063] To this end, the present invention designs a new normalized intersection-over-union (IoU) norm , its definition and calculation method are as follows:

[0064]

[0065] At this time, based on the assumption of the size relationship between the two target boxes, the calculation formula of the normalized intersection-over-union ratio is as follows:

[0066]

[0067] It can be found that the final normalized IoU is only related to the overlap of the two target frames (β) through derivation, and when α=1 in Formula 3, the results of IoU and normalized IoU are the same. That is, no matter how the size difference between the two target frames to be matched is, the relationship between the normalized IoU designed by the present invention and the overlap of the two target frames (β) is as shown in the attached figure. Figure 2 As shown by the purple line in the middle, it presents an approximately linear positive correlation trend.

[0068] Based on the above design, this implementation method realizes a target association method that is insensitive to scale differences, which is called a target association method based on normalized intersection-over-union ratio. The specific implementation process is as follows: Figure 3 As shown, including:

[0069] Step 1: Complete the target detection of the current frame and generate a series of candidate edge bounding boxes; set a threshold (such as confidence σ) to filter out the detection results with low confidence and obtain the input detection or segmentation target set TO;

[0070] Step 2: For each target in the existing source target set SO, calculate the normalized intersection-over-union ratio IoU between it and all detected bounding boxes in the current frame according to Formula 4: norm, where the initial value of the source target set SO is the candidate target set TO detected and extracted in the first frame of the tracking or mapping task;

[0071] Step 3: Construct a normalized intersection over union (IoU) norm Matrix, where each element represents the normalized intersection-over-union (IoU) between a target box in the source target set SO and a target box in the target set TO norm ;

[0072] Step 4: Using the Hungarian Algorithm, calculate the best match between each target in the source target set and the target in the current frame;

[0073] Step 5: Determine the normalized intersection-over-union (IoU) of each group of best matches norm Is it greater than or equal to a preset threshold (e.g., threshold θ), if yes, the group of targets is successfully associated, otherwise, the association fails;

[0074] Step 6: For those detected targets that do not match any target in the source target set in the current frame, initialize them as new targets to be associated and add them to the source target set SO;

[0075] Step 7: Repeat the above steps in a new frame to maintain continuous correlation tracking of the target.

[0076] A second embodiment of the present invention relates to a target association method based on the normalized intersection-over-union ratio, comprising the following steps:

[0077] S0 obtains the source target set;

[0078] S1 performs target segmentation on the image to be processed and obtains several candidate masks and their confidences;

[0079] S2 extracts candidate masks whose confidence is higher than the set confidence threshold to form a candidate target set;

[0080] S3 is the normalized intersection-over-union ratio (IoU) of the masks corresponding to each target to be associated in the source target set and each selected mask in the candidate target set. norm Construct a cost matrix for the element; normalize the intersection-over-union ratio IoU norm Calculated by the following formula:

[0081]

[0082] Among them, A is the mask corresponding to the associated target in the source target set, and B is the selected mask in the segmentation target set;

[0083] S4 uses the Hungarian algorithm to solve the selected mask that best matches each target to be associated in the source target set based on the cost matrix, and obtains multiple best matching pairs;

[0084] S5 is based on the normalized intersection-over-union ratio IoU of each group of best matching pairs norm Determines whether the target is relevant.

[0085] The present implementation is further described below in conjunction with the incremental mapping task in a mobile robot application.

[0086] The specific implementation process of this embodiment is as follows: Figure 4 As shown, including:

[0087] Step 1: Complete the target segmentation of the current frame and generate a series of candidate masks; set a threshold (such as confidence σ) to filter out the detection results with lower confidence and obtain the input detection or segmentation target set TO.

[0088] Step 2: For each target in the existing source target set SO, calculate the normalized intersection-over-union ratio IoU between it and all detected segmentation masks in the current frame according to Formula 4: norm , where the initial value of the source target set SO is the candidate target set TO detected and extracted in the first frame of the tracking or mapping task.

[0089] Step 3: Construct a normalized intersection over union (IoU) norm Matrix, where each element represents the normalized intersection-over-union (IoU) between a mask in the source target set SO and a mask in the target set TO norm .

[0090] Step 4: Using the Hungarian Algorithm, calculate the best match between each target in the source target set and the target in the current frame.

[0091] Step 5: Determine the normalized intersection-over-union (IoU) of each group of best matches norm Is it greater than or equal to a preset threshold (such as threshold θ), if yes, the group of targets is successfully associated, otherwise the association fails.

[0092] Step 6: For those detected targets that do not match any target in the source target set in the current frame, initialize them as new targets to be associated and add them to the source target set SO.

[0093] Step 7: Repeat the above steps in a new frame to maintain continuous correlation tracking of the target.

[0094] For incremental mapping tasks in applications such as mobile robots, as the perspective continues to change, the scene map is incrementally built, and in the process from local to global, many targets need to be modeled through continuous association and integration of targets from parts to the whole. In the association process, due to the limitation of the perspective, there are a large number of associations between local fragments and relatively complete targets. The association method based on intersection over union (IoU) performs poorly when there is a huge size difference between the fragments to be associated and the target. As shown in the table below, the target segmentation quality and recognition quality of the target association method based on IoU in the instance map reconstruction task of the ScanNetv2 dataset are poor. However, by using the target association method based on normalized intersection over union (IoU) proposed in the present invention, the segmentation and recognition quality of the instance targets in the reconstructed map are greatly improved. See the visual comparison. Figure 5 .

[0095] Using the target association method Reconstructing the quality of map object segmentation Reconstructing the quality of map object recognition Object association based on IoU 53.0 24.5 <![CDATA[Object association based on IoU norm > 62.9 64.7

Claims

1. A target association method based on normalized intersection-over-union ratio, characterized in that: The following steps are involved: Acquire a source target set, wherein the source target set includes a target to be associated and a bounding box thereof; Perform target detection on the image to be processed and obtain several candidate bounding boxes and their confidence scores; Extracting the candidate bounding boxes whose confidence is higher than a set confidence threshold to form a candidate target set; The normalized intersection-over-union (IoU) of the bounding boxes corresponding to the targets to be associated in the source target set and the selected bounding boxes in the candidate target set is calculated. norm Construct a cost matrix for the element; Based on the cost matrix, the Hungarian algorithm is used to solve the selected bounding boxes that best match each of the to-be-associated targets in the source target set to obtain multiple best matching pairs; According to the normalized intersection-over-union ratio IoU of each group of the best matching pairs norm Determines whether the target is relevant.

2. The method according to claim 1, characterized in that The normalized intersection-over-union ratio IoU norm Calculated by the following formula: Among them, A is the bounding box corresponding to the associated target in the source target set, and B is the selected bounding box in the segmentation target set.

3. The method according to claim 1, characterized in that The normalized intersection-over-union ratio IoU of each group of the best matching pairs norm Determine if the target is relevant, including: If the normalized intersection-over-union ratio IoU of the current best matching pair norm If the value is greater than the set intersection-over-union ratio threshold, it is considered that the corresponding target to be associated is associated with the candidate bounding box.

4. The method according to claim 1, characterized in that: The normalized intersection-over-union ratio IoU of the bounding boxes corresponding to the targets to be associated in the source target set and the selected bounding boxes in the candidate target set is norm Construct a cost matrix for the elements, including: The bounding boxes corresponding to the targets to be associated in the source target set are rows, and the selected bounding boxes in the candidate target set are columns. The normalized intersection-over-union (IoU) of each row and each column is calculated. norm , and obtain the cost matrix.

5. The method according to claim 1, characterized in that: The method also includes the steps of initializing the selected bounding boxes in the candidate target set that do not match any target to be associated as new targets to be associated, and adding the new targets to be associated to the source target set.

6. A target association method based on normalized intersection-over-union ratio, characterized in that: The following steps are involved: Acquire a source target set, wherein the source target set includes a target to be associated and a mask thereof; Perform target segmentation on the image to be processed and obtain several candidate masks and their confidences; Extracting the candidate masks whose confidence is higher than a set confidence threshold to form a candidate target set; The normalized intersection-over-union (IoU) of the masks corresponding to the targets to be associated in the source target set and the selected masks in the candidate target set is calculated. norm Construct a cost matrix for the element; Based on the cost matrix, the Hungarian algorithm is used to solve the selected mask that best matches each of the to-be-associated targets in the source target set, to obtain a plurality of best matching pairs; According to the normalized intersection-over-union ratio IoU of each group of the best matching pairs norm Determines whether the target is relevant.

7. The method according to claim 6, characterized in that The normalized intersection-over-union ratio IoU norm Calculated by the following formula: Among them, A is the mask corresponding to the associated target in the source target set, and B is the selected mask in the segmentation target set.

8. The method according to claim 6, characterized in that The normalized intersection-over-union ratio IoU of each group of the best matching pairs norm Determine if the target is relevant, including: If the normalized intersection-over-union ratio IoU of the current best matching pair norm If the value is greater than the set intersection-over-union ratio threshold, it is considered that the corresponding target to be associated is associated with the candidate mask.

9. The method according to claim 6, characterized in that The normalized intersection-over-union ratio IoU of the mask corresponding to each of the to-be-associated targets in the source target set and each of the selected masks in the candidate target set is norm Construct a cost matrix for the elements, including: The masks corresponding to the targets to be associated in the source target set are taken as rows, and the selected masks in the candidate target set are taken as columns. The normalized intersection-over-union ratio IoU of each row and each column is calculated. norm , and obtain the cost matrix.

10. The method according to claim 6, characterized in that The method also includes the steps of initializing the selected masks in the candidate target set that do not match any target to be associated as new targets to be associated, and adding the new targets to be associated to the source target set.

Images