Method and system for detecting change in occlusion rate of tracked object

By calculating the intersecting and comparison of continuous image frame pairs in the video sequence, the occlusion rate changes are detected, and the complexity of occlusion rate detection in the prior art is solved, the accuracy and efficiency of object tracking are improved, and unnecessary re-identification is avoided .

CN119068382APending Publication Date: 2024-12-03AXIS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410653176.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2024-05-24
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is complex and not effective enough to detect changes in the occlusion rate of the tracked object in a video sequence, making it difficult to accurately judge the occlusion situation of the object between continuous image frames, affecting the accuracy of object tracking and re-identification.

Method used

By calculating the intersection ratio of IoU between consecutive image frame pairs, the occlusion rate changes are detected, the occlusion rate changes are determined using the difference in IoU exceeding the threshold, and re-identification is avoided when necessary to improve the accuracy of tracking objects.

Benefits of technology

It realizes straightforward and uncomplexing detection of occlusion rate changes in video sequences, improves the accuracy and efficiency of object tracking, avoids unnecessary re-identification, and reduces complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068382A_ABST
    Figure CN119068382A_ABST
Patent Text Reader

Abstract

Methods and systems for detecting a change in occlusion rate of a tracked object. For each of a plurality of image frames of the video sequence, a bounding box or mask of the tracked object is determined, and for each of a plurality of consecutive image frame pairs of the plurality of image frames, a bounding box or mask of the tracked object is determined. An intersection-to-sum ratio IoU of a first bounding box or first mask in a first image frame of the consecutive image frame pair and a second bounding box or second mask in a second image frame of the consecutive image frame pair is calculated. Similarly, for another consecutive image frame pair following a plurality of consecutive image frame pairs of the plurality of image frames, another intersection-to-union ratio IoU of a first bounding box or first mask in a first image frame of the another consecutive image frame pair to a second bounding box or second mask in a second image frame of the another consecutive image frame pair is calculated. Under the condition that the other IoU differs from the calculated IoU of the plurality of consecutive pairs of image frames by more than a threshold amount, it is detected that the occlusion rate of the tracked object has changed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to detecting changes in video sequences related to tracked objects and processing of tracked objects, and in particular, to detecting changes in occlusion rate of tracked objects in video sequences and / or deciding to avoid using re-identification when tracking objects. Background Art

[0002] When tracking an object in a video sequence, it may be useful to detect when a portion of a tracked object changes from being visible in an image frame to being occluded in consecutive image frames of the video sequence or when a portion of a tracked object changes from being occluded in an image frame to being visible in consecutive image frames of the video sequence, i.e., detecting a change in the occlusion rate of a tracked object between two consecutive image frames (e.g., when a tracked object moves behind or out from behind something in a scene captured in the video sequence). Such detection is useful, for example, when deciding whether to use a function based on the appearance of a detected object (e.g., re-identification) in object tracking when identifying which object detection in an image frame corresponds to an object detection in a previous image frame. Such detection has been proposed in the prior art. However, the proposed method has disadvantages such as complexity. Therefore, there is still a need for improvement in detecting changes in the occlusion rate of a tracked object between two consecutive image frames. Summary of the invention

[0003] An object of the present invention is to overcome or at least alleviate the problems of the prior art.

[0004] According to a first aspect, a computer-implemented method for detecting changes in the occlusion rate of a tracked object in a video sequence is provided. For each of a plurality of image frames of a video sequence, a bounding box or a mask of a tracked object is determined. For each of a plurality of consecutive image frame pairs in the plurality of image frames, an intersection over union (IoU) of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair is calculated, or an intersection over union (IoU) of a first mask in a first image frame of the consecutive image frame pair and a second mask in a second image frame of the consecutive image frame pair is calculated. Similarly, for another consecutive image frame pair after the plurality of consecutive image frame pairs in the plurality of image frames, another intersection over union (IoU) of a first bounding box in a first image frame of the another consecutive image frame pair and a second bounding box in a second image frame of the another consecutive image frame pair is calculated, or another intersection over union (IoU) of a first mask in a first image frame of the another consecutive image frame pair and a second mask in a second image frame of the another consecutive image frame pair is calculated. Under the condition that another IoU differs from the calculated IoU of multiple consecutive image frame pairs by more than a threshold amount, it is detected that the occlusion rate of the tracked object has changed.

[0005] The present invention takes advantage of the realization that when an object is moving, the intersection over union (IoU) between bounding boxes or masks in a first image frame and a second image frame of a video sequence will be different when a portion of an object that was visible in a first image frame becomes occluded in the second image frame, or when a portion of an object that was occluded in the first image frame becomes visible in the second image frame, compared to the IoU between bounding boxes or masks in a first image frame and a second image frame of the video sequence where the occlusion ratio of the object does not change between the first image and the second image.

[0006] The calculation of another IoU for the other consecutive image frame pair and the calculation of the IoU for the multiple consecutive image frame pairs and the comparison for determining whether the other IoU differs from the calculated IoU for the multiple consecutive image frame pairs by more than a threshold amount are straightforward and uncomplicated. Therefore, the method according to the first aspect can detect that the occlusion rate of the tracked object has changed straightforwardly and uncomplicatedly.

[0007] In addition, when the occlusion rate of the tracked object changes significantly between the other consecutive image frame pairs compared to the occlusion rate of the tracked object in the multiple consecutive image frame pairs, the other IoU will be significantly different from the calculated IoU of the multiple consecutive image frame pairs. Therefore, the method according to the first aspect can reliably detect that the occlusion rate of the tracked object has changed.

[0008] By "bounding box of a tracked object" is meant a box that surrounds the tracked object. The term "bounding box" is used as commonly understood in video analysis related to object detection. Typically, the bounding box of a tracked object is the smallest box that surrounds the tracked object. Methods for detecting objects and determining bounding boxes are well known in the art.

[0009] By "mask of a tracked object" is meant a mask that surrounds the tracked object. The term "mask" is used as commonly understood in video analysis related to object detection. Typically, the mask of a tracked object coincides with or is close to the outline of the tracked object. Methods for detecting objects and determining masks (e.g., by segmentation) are well known in the art.

[0010] The so-called "intersection over union" IoU refers to the division or ratio of the area of ​​the intersection between a first region (e.g., a first bounding box or mask) and a second region (e.g., a second bounding box or mask) to the area of ​​the union between the first region and the second region.

[0011] It should be noted that both the plurality of consecutive image frame pairs and the further consecutive image frame pair belong to the plurality of image frames. Therefore, it is implicit that the plurality of consecutive image frame pairs do not constitute the entirety of the plurality of image frames.

[0012] The method according to the first aspect may further include calculating an average or median of the IoUs of a plurality of consecutive image frame pairs. Subsequently, the detection action includes: detecting that the occlusion rate of the tracked object has changed under the condition that another IoU differs from the determined average or median of the IoUs of the plurality of consecutive image frame pairs by more than a threshold amount.

[0013] The average may be an unweighted average, or may be a weighted average. For example, a weighted average may be used, where the weight of the IoU increases as it is further back in a plurality of consecutive image frame pairs, i.e., the closer to another consecutive image frame pair, the more there are consecutive image frame pairs for which the IoU is calculated. For example, the weight may increase exponentially.

[0014] By using a weighted average, the difference between the other IoU and the calculated IoU of the multiple consecutive image frame pairs can be made to depend more on the IoU of the consecutive image frame pairs in the multiple image frames (i.e., in the video sequence) that are closer to the other consecutive image frame pair.

[0015] In addition to calculating the average or median, the method according to the first aspect may further include calculating the variance of the IoU of multiple consecutive image frame pairs, and using the calculated variance to set the threshold.

[0016] By using the calculated variance to set the threshold, the threshold can be set so that a change in the occlusion rate of the tracked object is detected only when the difference between the other IoU and the calculated IoU of the multiple consecutive image frame pairs is greater than the normal variation within the multiple consecutive image frame pairs (as indicated by the variance).

[0017] According to a second aspect, a computer-implemented method for deciding to avoid using re-identification when tracking an object in an image frame of a video sequence is provided. For each of a plurality of image frames of the video sequence, a bounding box or a mask of a tracked object is determined. For each of a plurality of consecutive image frame pairs in the plurality of image frames, an intersection-over-union (IoU) of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair is calculated, or an intersection-over-union (IoU) of a first mask in the first image frame of the consecutive image frame pair and a second mask in the second image frame of the consecutive image frame pair is calculated. For another consecutive image frame pair after the plurality of consecutive image frame pairs in the plurality of image frames, another intersection-over-union (IoU) of a first bounding box in the first image frame of the another consecutive image frame pair and a second bounding box in the second image frame of the another consecutive image frame pair is calculated, or another intersection-over-union (IoU) of a first mask in the first image frame of the another consecutive image frame pair and a second mask in the second image frame of the another consecutive image frame pair is calculated. Under the condition that the another IoU differs from the calculated IoU of the plurality of consecutive image frame pairs by more than a threshold amount, it is determined to avoid using re-identification when tracking the object in the another consecutive image frame pair.

[0018] The above optional additional features of the method according to the first aspect are also applicable to the method according to the second aspect, where applicable. In order to avoid unnecessary repetitions, reference is made to the above.

[0019] According to a third aspect, a non-transitory computer-readable storage medium is provided, which has instructions stored thereon, and when the instructions are executed in a device having a processor, the method according to the first aspect or the method according to the second aspect is implemented.

[0020] The above optional additional features of the method according to the first aspect also apply, where applicable, to the non-transitory computer-readable storage medium according to the third aspect. In order to avoid unnecessary repetitions, reference is made to the above.

[0021] According to a fourth aspect, a device is provided. The device includes a circuit configured to perform a determination function, a first calculation function, a second calculation function, and a detection function. The determination function is configured to determine a bounding box or a mask of a tracked object for each of a plurality of image frames of a video sequence. The first calculation function is configured to calculate, for each of a plurality of consecutive image frame pairs in a plurality of image frames, an intersection-of-units (IoU) ratio of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair, or to calculate an intersection-of-units (IoU) ratio of a first mask in a first image frame of the consecutive image frame pair and a second mask in a second image frame of the consecutive image frame pair. The second calculation function is configured to calculate, for another continuous image frame pair after multiple continuous image frame pairs in the multiple image frames, another intersection-of-union (IoU) ratio of a first bounding box in a first image frame of the another continuous image frame pair and a second bounding box in a second image frame of the another continuous image frame pair, or another intersection-of-union (IoU) ratio of a first mask in a first image frame of the another continuous image frame pair and a second mask in a second image frame of the another continuous image frame pair. The detection function is configured to detect that the occlusion rate of the tracked object has changed under the condition that the difference between the other IoU and the calculated IoU of the multiple continuous image frame pairs exceeds a threshold amount.

[0022] The above optional additional features of the method according to the first aspect also apply to the device according to the fourth aspect, where applicable. In order to avoid unnecessary repetitions, reference is made to the above.

[0023] According to a fifth aspect, a device is provided. The device includes a circuit configured to perform a determination function, a first calculation function, a second calculation function, and a decision function. The determination function is configured to determine a bounding box or mask of a tracked object for each of a plurality of image frames of a video sequence. The first calculation function is configured to calculate, for each of a plurality of consecutive image frame pairs in a plurality of image frames, an intersection-of-units (IoU) ratio of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair, or to calculate an intersection-of-units (IoU) ratio of a first mask in a first image frame of the consecutive image frame pair and a second mask in a second image frame of the consecutive image frame pair. The second calculation function is configured to calculate, for another continuous image frame pair after a plurality of continuous image frame pairs in the plurality of image frames, another intersection-of-union (IoU) ratio of a first bounding box in a first image frame of the another continuous image frame pair and a second bounding box in a second image frame of the another continuous image frame pair, or another intersection-of-union (IoU) ratio of a first mask in a first image frame of the another continuous image frame pair and a second mask in a second image frame of the another continuous image frame pair. The decision function is configured to decide to avoid using re-identification when tracking an object in the another continuous image frame pair, if the other IoU differs from the calculated IoU of the plurality of continuous image frame pairs by more than a threshold amount.

[0024] The above optional additional features of the method according to the first aspect also apply, where applicable, to the device according to the fifth aspect. In order to avoid unnecessary repetitions, reference is made to the above.

[0025] According to the detailed description given below, the further applicable scope of the present invention will become apparent. However, it should be understood that while indicating the preferred embodiments of the present invention, the detailed description and specific examples are given only by way of illustration, because for those skilled in the art, various changes and modifications within the scope of the present invention will become apparent from the detailed description.

[0026] Therefore, it should be understood that the present invention is not limited to the specific component parts of the described equipment or the actions of the described methods, because such equipment and methods can be changed. It should also be understood that the terms used herein are only used for the purpose of describing specific embodiments and are not intended to be limiting. It must be noted that, as used in this specification and the appended claims, the words "one", "an", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly stipulates otherwise. Therefore, for example, references to "a unit" or "the unit" can include several devices, etc. In addition, the words "comprise", "include" and "contain" and similar words do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other aspects of the present invention will now be described in more detail with reference to the accompanying drawings which are not to be considered limiting but rather are for explanation and understanding.

[0028] Figure 1 A flow chart is shown relating to an embodiment of a method of detecting a change in an occlusion rate of a tracked object in a video sequence.

[0029] Figure 2a to Figure 2f A diagram showing bounding boxes of a tracked object in consecutive image frame pairs and IOUs between bounding boxes in consecutive image frame pairs.

[0030] Figures 3a to 3f A diagram showing masks of a tracked object in consecutive image frame pairs and IoU between masks in consecutive image frame pairs.

[0031] Figure 4 A flow chart is shown relating to an embodiment of a method for determining to avoid using re-identification when tracking an object in a video sequence.

[0032] Figure 5 Schematic diagrams related to embodiments of the apparatus are shown.

[0033] Figure 6 A schematic diagram related to an embodiment of another apparatus is shown. DETAILED DESCRIPTION

[0034] The invention will now be described hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the invention are illustrated. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0035] The present invention is applicable to the context of tracking an object in a video sequence. In such a context, it may be beneficial to detect when the occlusion rate of the tracked object changes. For example, in some embodiments, so-called frame-by-frame re-identification (or simply re-identification) can be used to assist in object tracking, for example, initially based only on the Kalman filter state, while considering the possibility that the candidate object in the image frame is the same object as the object detected by the object tracker in the previous frame. An example of such re-identification is DeepSORT described in https: / / arxiv.org / pdf / 1703.07402.pdf. In such an embodiment, weighting can be used, indicating to what extent the detection should be based on re-identification and Kalman filter state, respectively. In the case where the occlusion rate of the tracked object changes between consecutive image frame pairs of a video sequence for the tracked object, for example because the part of the object visible in one image frame of the pair becomes occluded in the next image frame of the pair or because the occluded part of the object in one image frame of the pair becomes visible in the next image frame of the pair, re-identification may not be helpful for assisting tracking. Therefore, in this case, when assisting tracking with respect to consecutive image frame pairs in which a change in the occlusion rate of the tracked object is detected, it is best to avoid using re-identification, or at least reduce the weight of the results from the re-identification. Other methods that utilize comparison of pixel data between an object and a candidate object are also affected by such changes in the occlusion rate of the tracked object in the video sequence, and therefore, detecting a change in the occlusion rate of the tracked object can be a trigger for avoiding the use of such methods or reducing the weight of the results of such methods with respect to consecutive image frame pairs in which a change in the occlusion rate of the tracked object is detected.

[0036] According to an embodiment of the present invention, the change in occlusion rate is detected by calculating the IoU between the bounding boxes or masks of the tracked objects in multiple consecutive image frame pairs, and detecting the difference of the IoU in another consecutive image frame relative to the IoU of multiple consecutive image frame pairs that exceeds a threshold. In addition, this difference exceeding the threshold may also be caused by other reasons other than the change in occlusion rate. Therefore, when a difference of the IoU in another consecutive image frame pair relative to the IoU of multiple consecutive image frame pairs that exceeds a threshold is detected, it is generally beneficial to avoid using re-identification or other methods that use comparison of pixel data between the object and the candidate object when assisting in tracking the object, or at least reduce their weight.

[0037] Now combine Figure 1 The flowchart and Figure 2a to Figure 2f An embodiment of a method 100 for detecting changes in the occlusion rate of a tracked object in a video sequence is described with reference to a diagram of bounding boxes and IoU in FIG.

[0038] In method 100, for each of a plurality of image frames of a video sequence, a bounding box of a tracked object is determined (S110). The bounding box may be determined using any known method for detecting an object and providing a bounding box that defines an extension of the detected object. Typically, the bounding box of a tracked object is a minimum box or a nearly minimum box that encloses the tracked object. Known algorithms for detecting objects and providing bounding boxes are YOLO, MobileNet, and R-CNN. A mask may also be provided from R-CNN.

[0039] Now go to Figure 2a , Figure 2b , Figure 2d and Figure 2e , Figure 2a The middle figure shows a dashed bounding box 210 around the tracked object 200 in the first image frame of the pair of consecutive image frames, and Figure 2b The middle figure shows a bounding box 220 with a solid line around the tracked object 200 in the second image frame of the consecutive image frame pair. Similarly, Figure 2a The middle figure shows a dotted bounding box 250 around the tracked object 200 in the first image frame of another consecutive image frame pair, and Figure 2e The middle figure shows a bounding box 260 with a solid line around the tracked object 200 in the second image frame of another continuous image frame pair. Figure 2e , the object 200 becomes partially occluded in the second image frame of another consecutive image frame pair, and therefore, the bounding box 260 has the same Figure 2a , Figure 2b and Figure 2d The other bounding boxes 210, 220, 250 in FIG. 2 have different shapes.

[0040] When selecting a bounding box in a second image frame of a continuous image frame pair from a plurality of candidate bounding boxes, the bounding box that is closest to the bounding box of the first image frame of the continuous image frame pair is generally selected. Similarly, when selecting a bounding box in a second image frame of another continuous image frame pair from a plurality of candidate bounding boxes, the bounding box that is closest to the bounding box of the first image frame of the other continuous image frame pair is generally selected.

[0041] For each of a plurality of consecutive image frame pairs in a plurality of image frames, an IoU of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair is calculated (S120). Typically, the plurality of consecutive image frame pairs are overlapping, so that each image frame is included in a pair with a previous image frame and in a pair with a subsequent image frame. The result will be that if an IoU is calculated for each image frame, the IoU is calculated for a pair consisting of the image frame and the previous image frame in the video sequence. However, the plurality of consecutive image frame pairs may also be non-overlapping, so that each image frame is included in only one pair, but so that there is no image frame in the sequence that is not included in any pair. The result will be that if an IoU is calculated for each second image frame, the IoU is calculated for a pair consisting of the image frame and the previous image frame in the video sequence. Preferably, all pairs are overlapping, or all pairs are non-overlapping. However, combinations are also possible.

[0042] Now go to Figure 2a to Figure 2c ,in Figure 2c The figure shows Figure 2a The bounding box 210 of the tracked object 200 in the first image frame of the pair of consecutive image frames shown in FIG. Figure 2b , which illustrates the IoU between the bounding boxes 220 of the tracked object 200 in the second image frame of the consecutive image frame pair. The intersection 230 in the numerator is the area illustrated with vertical stripes, and the union 240 in the denominator is illustrated with horizontal stripes. From the difference in position between the bounding box 210 in the first image frame of the consecutive image frame pair and the bounding box 220 in the second image frame of the consecutive image frame pair, it can be seen that the object has moved between the first image frame and the second image frame of the consecutive image frame pair. In addition, the size and shape of the bounding box 210 in the first image frame of the consecutive image frame pair are the same as the size and shape of the bounding box 220 in the second image frame of the consecutive image frame pair. Under the condition that the object 200 moves at the same or similar speed and the occlusion rate is the same or within an acceptable range on each image frame of the multiple consecutive image frame pairs, the IoU of the multiple pairs will be in the range including Figure 2c The example IoU shown in the figure varies within a range.

[0043] For another continuous image frame pair after a plurality of continuous image frame pairs in a plurality of image frames, the intersection over union (IoU) of a first bounding box in a first image frame of the another continuous image frame pair and a second bounding box in a second image frame of the another continuous image frame pair is calculated (S130). The another continuous image frame pair is usually composed of a last image frame of a last continuous image frame pair of a plurality of image frames and a subsequent image frame of a plurality of image frames. Alternatively, the another continuous image frame pair is composed of a first image frame and a second image frame after the last image frame of the last continuous image frame pair of a plurality of image frames. The another continuous image frame pair may also include an even later continuous image frame pair of a plurality of image frames. However, preferably, the another continuous image frame pair is close to the last image frame of the last continuous image frame pair of a plurality of image frames.

[0044] Now go to Figure 2d to Figure 2f ,in Figure 2f The figure shows Figure 2d The bounding box 250 of the tracked object 200 in the first image frame of another consecutive image frame pair shown in FIG. Figure 2e Another IoU between the bounding box 260 of the tracked object 200 in the second image frame of another consecutive image frame pair is illustrated in . The intersection 270 in the numerator is the area illustrated with vertical stripes, and the union 280 in the denominator is illustrated with vertical stripes. From the difference in position between the bounding box 250 in the first image frame of another consecutive image frame pair and the bounding box 260 in the second image frame of another consecutive image frame pair, it can be seen that the object has moved between the first image frame and the second image frame of the consecutive image frame pair. In addition, the size and shape of the bounding box 250 in the first image frame of another consecutive image frame pair are substantially different from the size and shape of the bounding box 260 in the second image frame of another consecutive image frame pair. This is because the object 200 has been partially occluded in the second image frame of another consecutive image frame pair.

[0045] Under the condition C140 that another IoU differs from the calculated IoU of multiple consecutive image frame pairs by more than a threshold amount, it is detected (S150) that the occlusion rate of the tracked object has changed. It should be noted that the absolute value of the difference is meaningful, because a change to a higher occlusion rate will have a difference of the opposite value to a change to a lower occlusion rate.

[0046] Go to Figure 2c and Figure 2f, it can be seen that the size of the intersection 270 between the bounding box 250 of the tracked object 200 in the first image frame of another consecutive image frame pair and the bounding box 260 of the tracked object 200 in the second image frame of another consecutive image frame pair is substantially different from the size of the intersection 230 between the bounding box 210 of the tracked object 200 in the first image frame of the consecutive image frame pair and the bounding box 220 of the tracked object 200 in the second image frame of the consecutive image frame pair. On the other hand, the difference between the sizes of the union 280 between the bounding box 250 of the tracked object 200 in the first image frame of another consecutive image frame pair and the bounding box 260 of the tracked object 200 in the second image frame of another consecutive image frame pair and the union 240 between the bounding box 210 of the tracked object 200 in the first image frame of the consecutive image frame pair and the bounding box 220 of the tracked object 200 in the second image frame of the consecutive image frame pair is small. Therefore, another IoU between the bounding box 250 of the tracked object 200 in the first image frame of another consecutive image frame pair and the bounding box 260 of the tracked object 200 in the second image frame of another consecutive image frame pair is substantially different from the IoU between the bounding box 210 of the tracked object 200 in the first image frame of the consecutive image frame pair and the bounding box 220 of the tracked object 200 in the second image frame of the consecutive image frame pair. Assuming that the IoUs of multiple frame pairs are generally similar to Figure 2c If the IoU illustrated in FIG. 1 is greater than the IoU illustrated in FIG. 1 , the other IoU will differ from the calculated IoU of the plurality of consecutive image frame pairs by more than a threshold amount, and it will be detected ( S150 ) that the occlusion rate of the tracked object has changed.

[0047] The method 100 may further include calculating an average or median of the IoUs of the plurality of consecutive image frame pairs. Subsequently, the act of detecting (S150) includes: detecting that the occlusion rate of the tracked object has changed under the condition that another IoU differs from the average or median of the determined IoUs of the plurality of consecutive image frame pairs by more than a threshold amount.

[0048] The average may be an unweighted average, or may be a weighted average. For example, a weighted average may be used, where the weight of the IoU increases as it is further back in the consecutive image frame pair, i.e., the closer to another consecutive image frame pair, the more there are consecutive image frame pairs for which the IoU is calculated. For example, the weight may increase exponentially.

[0049] In addition to calculating the mean or median, method 100 may further include calculating the variance of the IoU of multiple consecutive image frame pairs and using the calculated variance to set a threshold. By using the calculated variance to set the threshold, the threshold can be set so that a change in the occlusion rate of the tracked object is detected only when the difference between another IoU and the calculated IoU of multiple consecutive image frame pairs is greater than the normal variation within the multiple consecutive image frame pairs (as indicated by the variance). When setting the threshold, other measures for indicating the typical variation of the IoU of the consecutive image frame pairs in the multiple consecutive image frame pairs are of course applicable.

[0050] The method 100 has been described with respect to a specific example of detecting a change in the occlusion rate of a tracked object in a video sequence, wherein it is to be determined whether the change in the occlusion rate occurs in another consecutive image frame pair, and wherein the IoU is first determined for a plurality of consecutive image frames occurring before the other consecutive image frame pair in the video sequence. Typically, the detection is performed for each consecutive image frame pair in the video sequence such that for each consecutive image frame pair in the video sequence, the IoU calculated for the consecutive image frame pair is compared with the IoU calculated for the previous consecutive image frame pair in the video sequence in order to detect any change in the occlusion rate. The IoUs of a predetermined number of previous consecutive image frame pairs may be compared, i.e., the plurality of consecutive image frame pairs is a predetermined number of consecutive image frame pairs constituting a rolling window of previous consecutive image frame pairs whose IoUs are used in the comparison.

[0051] The multiple consecutive image frame pairs should preferably be such that no change in occlusion rate is detected for any consecutive image frame pair among the multiple consecutive image frame pairs, that is, the IoU of each consecutive image frame pair in the multiple image frames is no different from the IoU of the multiple consecutive image frame pairs before the consecutive image frame pair.

[0052] When a change in occlusion rate is detected for (another) continuous image frame pair, the IoU will be different from the IoU of multiple previous image frames, and therefore, for the next continuous image frame pair after the other continuous image frame pair, the change in occlusion rate should preferably be detected by comparison with the continuous image frame pair in which the change in occlusion rate is detected. Therefore, when a change in occlusion rate is detected for another continuous image frame pair, multiple continuous image frame pairs in the method should be selected starting from the continuous image frame pair in which the change in occlusion rate is detected. Because there is a previous continuous image frame pair in which a change in occlusion rate is detected, or if no change in occlusion rate is detected since the first continuous image frame pair of the tracked object, multiple continuous image frame pairs can therefore be selected as all or a subset of the continuous image frame pairs.

[0053] As an alternative to determining a bounding box and calculating the IoU in method 100 based on the determined bounding box, a mask may be determined and the IoU may be calculated based on the determined mask. The mask may be determined using any known method for detecting an object and providing a mask for defining the extension of the detected object. Typically, the mask of the tracked object coincides with or is close to the outline of the tracked object. Figures 3a to 3f A diagram showing masks (310; 320; 350; 360) in a pair of consecutive image frames of a tracked object 300 and the IoU between masks (310; 320; 350; 360) in a pair of consecutive image frames is shown. Figure 3c The figure shows Figure 3a The mask 310 of the tracked object 300 in the first image frame of the pair of consecutive image frames shown in FIG. Figure 3b The IoU between the masks 320 of the tracked object 300 in the second image frame of the consecutive image frame pair illustrated in . The intersection 330 in the numerator is an area illustrated with a vertical stripe diagram, and the union 240 in the denominator is illustrated with a vertical stripe diagram. It can be seen from the difference in position between the mask 310 in the first image frame of the consecutive image frame pair and the mask 320 in the second image frame of the consecutive image frame pair that the object has moved between the first image frame and the second image frame of the consecutive image frame pair. In addition, the size and shape of the mask 310 in the first image frame of the consecutive image frame pair are the same as the size and shape of the mask 320 in the second image frame of the consecutive image frame pair. Specifically, the mask (310; 320) is circular and coincides with the outline of the tracked object 300. Under the condition that the object 300 moves at the same or similar speed and the occlusion rate is the same or within an acceptable range on each image frame of multiple consecutive image frame pairs, the IoU of multiple pairs will be in the range including Figure 3c The example IoU shown in the figure varies within the range. Now go to Figure 3d to Figure 3f ,in Figure 3f The figure shows Figure 3d The mask 350 of the tracked object 300 in the first image frame of another consecutive image frame pair shown in FIG. Figure 3eAnother IoU between the mask 360 of the tracked object 300 in the second image frame of another consecutive image frame pair is illustrated in . The intersection 370 in the numerator is the area illustrated with vertical stripes, and the union 380 in the denominator is illustrated with horizontal stripes. From the difference in position between the mask 350 in the first image frame of another consecutive image frame pair and the mask 360 in the second image frame of another consecutive image frame pair, it can be seen that the object has moved between the first image frame and the second image frame of the consecutive image frame pair. In addition, the size and shape of the mask 350 in the first image frame of another consecutive image frame pair are substantially different from the size and shape of the mask 360 in the second image frame of another consecutive image frame pair. This is because the object 300 has been partially occluded in the second image frame of another consecutive image frame pair. Compare Figure 3c and Figure 3f , it can be seen that the size of the intersection 370 between the mask 350 of the tracked object 300 in the first image frame of another consecutive image frame pair and the mask 360 of the tracked object 300 in the second image frame of another consecutive image frame pair is substantially different from the size of the intersection 330 between the mask 310 of the tracked object 300 in the first image frame of the consecutive image frame pair and the mask 320 of the tracked object 300 in the second image frame of the consecutive image frame pair. On the other hand, the difference between the sizes of the union 380 between the mask 350 of the tracked object 300 in the first image frame of another consecutive image frame pair and the mask 360 of the tracked object 300 in the second image frame of another consecutive image frame pair and the union 340 between the mask 310 of the tracked object 300 in the first image frame of the consecutive image frame pair and the mask 320 of the tracked object 300 in the second image frame of the consecutive image frame pair is small. Therefore, another IoU between the mask 350 of the tracked object 300 in the first image frame of another consecutive image frame pair and the mask 360 of the tracked object 300 in the second image frame of another consecutive image frame pair is substantially different from the IoU between the mask 310 of the tracked object 300 in the first image frame of the consecutive image frame pair and the mask 320 of the tracked object 300 in the second image frame of the consecutive image frame pair.

[0054] Now combine Figure 4 An embodiment of a method 400 for determining to avoid using re-identification when tracking an object in an image frame of a video sequence is described with reference to the flowchart in FIG.

[0055] Method 400 includes combining Figure 1 The steps of determining (S110), calculating (S120) and further calculating (S130) are performed in the same manner as the method 100 described above. Figure 1 The detailed description of these actions also applies to method 400.

[0056] Method 400 differs from method 100 in that the action of detecting (S150) is replaced by the action of deciding (S450). Specifically, under the condition C140 that the other IoU differs from the calculated IoU of the plurality of consecutive image frame pairs by more than a threshold amount, it is decided (S450) to avoid using re-identification when tracking the object in the other consecutive image frame pairs. To this end, the condition C140 that the other IoU differs from the calculated IoU of the plurality of consecutive image frame pairs by more than a threshold amount is different from the condition C140 regarding Figure 1 The conditions described for C140 are the same and therefore, reference is made to Figure 1 Description of this in method 100. However, method 400 does not necessarily need to include explicit detection of changes in occlusion rate as in method 100. Instead, condition C140 in method 400 is used to perform the action of deciding (S450) whether condition C140 is satisfied. Specifically, the action of deciding (S450) includes deciding (S450) to avoid using re-identification when tracking the object in another pair of consecutive image frames.

[0057] Although changes in the occlusion rate of the tracked object are the most common reason for the change in IoU relative to the previous IoU exceeding the threshold, there are other reasons for such changes. Specifically, anything that causes a significant change in the size, shape, or position of the bounding box or mask of the tracked object between consecutive image frames will generally cause such a change in IoU relative to the previous IoU exceeding the threshold. Such other reasons are, for example, a sudden change in the posture of the tracked object, a sudden change in the shape of the tracked object, etc.

[0058] Note that even if there are other reasons for the change in IoU relative to the previous IoU that exceeds the threshold besides the change in the occlusion rate of the tracked object, the other reasons only constitute a small fraction of the total number of such changes in IoU relative to the previous IoU that exceed the threshold. Therefore, in method 100, a small proportion of false detections of changes in the occlusion rate of the tracked object is generally acceptable.

[0059] In another embodiment, under the condition that another IoU differs from the IoU of multiple consecutive image frame pairs by more than a threshold amount, alternatively or additionally, for example with respect to auxiliary tracked objects, in addition to deciding to avoid using re-identification, it can also be decided to avoid using one or more other methods for utilizing pixel data comparison between the object and the candidate object or to reduce the weight of these methods.

[0060] about Figure 1 and Figure 2a to Figure 2f The optional additional features of the method 100 described herein also apply, where applicable, to the Figure 4 Method 400 is described.

[0061] Figure 5 A schematic diagram related to an embodiment of a device 500 is shown. The device 500 is configured to detect that an occlusion rate of a tracked object has changed. The device 500 may be, be included in, or be connected to a camera, for example.

[0062] Device 500 includes circuit 510. Circuit 510 is configured to perform the functions of device 500. Circuit 510 may include processor 512, such as a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a microcontroller or a microprocessor. Processor 512 is configured to execute program code. The program code may, for example, be configured to perform the functions of device 500.

[0063] The device 500 may further include a memory 520. The memory 520 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile storage, a random access memory (RAM), or other suitable devices. In a typical arrangement, the memory 520 may include a non-volatile memory for long-term data storage and a volatile memory used as a device memory for the circuit 510. The memory 520 may exchange data with the circuit 510 via a data bus. There may also be accompanying control lines and an address bus between the memory 520 and the circuit 510.

[0064] The functionality of the device 500 may be implemented in the form of an executable logic routine (e.g., a line of code, a software program, etc.) that is stored on a non-transitory computer-readable medium (e.g., memory 520) of the device 500 and executed by the circuit 510 (e.g., using the processor 512). In addition, the functionality of the device 500 may be a stand-alone software application, or form part of a software application for performing additional tasks associated with the device 500. The described functionality may be considered as a method that a processing unit (e.g., a processor 512 of the circuit 510) is configured to perform. In addition, although the described functionality may be implemented in software, such functionality may also be implemented via dedicated hardware or firmware, or some combination of hardware, firmware, and / or software.

[0065] The circuit 510 is configured to perform a determination function 521 , a first calculation function 522 , a second calculation function 523 , and a detection function 524 .

[0066] The determination function 521 is configured to determine, for each of a plurality of image frames of a video sequence, a bounding box or mask of the tracked object.

[0067] The first calculation function 522 is configured to calculate, for each consecutive image frame pair in a plurality of consecutive image frame pairs in a plurality of image frames, an intersection-over-union (IoU) ratio of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair, or to calculate an intersection-over-union (IoU) ratio of a first mask in the first image frame of the consecutive image frame pair and a second mask in the second image frame of the consecutive image frame pair.

[0068] The second calculation function 523 is configured to calculate, for another consecutive image frame pair after multiple consecutive image frame pairs in the multiple image frames, another intersection-and-union (IoU) of a first bounding box in a first image frame of the another consecutive image frame pair and a second bounding box in a second image frame of the another consecutive image frame pair, or to calculate another intersection-and-union (IoU) of a first mask in the first image frame of the another consecutive image frame pair and a second mask in the second image frame of the another consecutive image frame pair.

[0069] The detection function 524 is configured to detect that the occlusion rate of the tracked object has changed if another IoU differs from the calculated IoU of multiple consecutive image frame pairs by more than a threshold amount.

[0070] Above about Figure 1 and Figure 2a to Figure 2f The detailed description of the actions of the described method 100 also applies to the corresponding functions of the device 500. Figure 1 and Figure 2a to Figure 2f The optional additional features described for method 100 are also applicable to apparatus 500 , where applicable.

[0071] Figure 6 A block diagram is shown relating to an embodiment of another device 600. The device 600 is configured to decide to avoid using re-identification when tracking an object in a video sequence. The device 600 may be, be included in, or be connected to a camera, for example.

[0072] Device 600 is similar to Figure 5 Therefore, for a general description of the device 600, reference is made to the description of the device 500. Figure 5 A general description of the device 500 is provided.

[0073] The device 600 includes a circuit 510 configured to perform the Figure 5 The determination function 521, the first calculation function 522 and the second calculation function 523 described above. Figure 5 With the detection function 524 described, the circuitry 510 of the device 600 is configured to perform the decision function 624 .

[0074] The decision function 624 is configured to decide to avoid using re-identification when tracking the object in another pair of consecutive image frames, conditional on the other IoU differing from the calculated IoU of the plurality of consecutive image frame pairs by more than a threshold amount.

[0075] The above about Figure 1 and Figure 2a to Figure 2f The detailed description of the actions of the method 100 described above and the detailed description of the actions of the method 300 described with respect to FIG. 2 also apply to the corresponding functions of the device 600. Figure 1 and Figure 2a to Figure 2f The optional additional features of the described method 100 and the optional additional features of the method 300 described with respect to FIG. 3 are also applicable to the further apparatus 600 , where applicable.

[0076] Even though it is disclosed herein that the IoU is calculated for each of a plurality of consecutive image frames and the IoU of another consecutive image frame pair is compared with the IoU of a plurality of consecutive image frame pairs, it is also feasible to determine the IoU of only one consecutive image frame pair and then compare the IoU of another consecutive image frame pair with the IoU of the one consecutive image frame pair. However, the method and apparatus using only one consecutive image frame pair for comparison will be less accurate. Furthermore, even though it is disclosed herein that the image frame pair is a consecutive image frame pair, an image frame pair that is not consecutive but is separated by one or more intermediate image frames in a plurality of image frames may also be used. However, the farther the distance between the image frames in each image frame pair, the less accurate the method and apparatus will be.

[0077] Those skilled in the art will appreciate that the present invention is not limited to the embodiments described above. On the contrary, within the scope of the appended claims, many modifications and variations are feasible. By studying the accompanying drawings, the present disclosure and the appended claims, the technicians practicing the claimed invention can understand and implement such modifications and variations.

Claims

1. A computer-implemented method for detecting a change in an occlusion rate of a tracked object in a video sequence, the method comprising: determining a bounding box or a mask of the tracked object for each of a plurality of image frames of the video sequence; For each of a plurality of consecutive image frame pairs in the plurality of image frames, calculating an intersection-over-union (IoU) ratio of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair, or calculating an intersection-over-union (IoU) ratio of a first mask in the first image frame of the consecutive image frame pair and a second mask in the second image frame of the consecutive image frame pair; For another continuous image frame pair after the multiple continuous image frame pairs in the multiple image frames, calculating another intersection-over-union (IoU) of a first bounding box in a first image frame of the another continuous image frame pair and a second bounding box in a second image frame of the another continuous image frame pair, or calculating another intersection-over-union (IoU) of a first mask in the first image frame of the another continuous image frame pair and a second mask in the second image frame of the another continuous image frame pair; as well as Under the condition that the another IoU differs from the calculated IoUs of the plurality of consecutive image frame pairs by more than a threshold amount, it is detected that the occlusion rate of the tracked object has changed.

2. The method according to claim 1, further comprising: Calculate the average or median of the IoU of the multiple consecutive image frame pairs, The detection actions include: Under the condition that the another IoU differs from an average or median of the determined IoUs of the plurality of consecutive image frame pairs by more than a threshold amount, it is detected that the occlusion rate of the tracked object has changed.

3. The method according to claim 2, further comprising: Calculating the variance of the IoU of the plurality of consecutive image frame pairs; as well as The calculated variance is used to set the threshold.

4. A computer-implemented method for determining to avoid using re-identification when tracking an object in an image frame of a video sequence, the method comprising: determining a bounding box or a mask of the tracked object for each of a plurality of image frames of the video sequence; For each of a plurality of consecutive image frame pairs in the plurality of image frames, calculating an intersection-over-union (IoU) ratio of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair, or calculating an intersection-over-union (IoU) ratio of a first mask in the first image frame of the consecutive image frame pair and a second mask in the second image frame of the consecutive image frame pair; For another continuous image frame pair after the multiple continuous image frame pairs in the multiple image frames, calculating another intersection-over-union (IoU) of a first bounding box in a first image frame of the another continuous image frame pair and a second bounding box in a second image frame of the another continuous image frame pair, or calculating another intersection-over-union (IoU) of a first mask in the first image frame of the another continuous image frame pair and a second mask in the second image frame of the another continuous image frame pair; as well as Under the condition that the another IoU differs from the calculated IoUs of the plurality of consecutive image frame pairs by more than a threshold amount, it is determined to avoid using re-identification when tracking the object in the another consecutive image frame pair.

5. The method according to claim 4, further comprising: Calculate the average or median of the IoU of the multiple consecutive image frame pairs, The detection actions include: Under the condition that the another IoU differs from an average or a median of the determined IoUs of the plurality of consecutive image frame pairs by more than a threshold amount, it is decided to avoid using re-identification when tracking the object in the another consecutive image frame pair.

6. The method according to claim 5, further comprising: Calculating the variance of the IoU of the plurality of consecutive image frame pairs; as well as The calculated variance is used to set the threshold.

7. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed in a device having a processor, implement the method according to any one of claims 1 and 4.

8. An apparatus comprising a circuit configured to perform: a determination function configured to determine, for each of a plurality of image frames of a video sequence, a bounding box or mask of a tracked object; A first calculation function is configured to calculate, for each of a plurality of consecutive image frame pairs in the plurality of image frames, an intersection-over-union (IoU) ratio of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair, or calculate an intersection-over-union (IoU) ratio of a first mask in the first image frame of the consecutive image frame pair and a second mask in the second image frame of the consecutive image frame pair; A second calculation function is configured to calculate, for another continuous image frame pair after the multiple continuous image frame pairs in the multiple image frames, another intersection-over-union (IoU) of a first bounding box in a first image frame of the another continuous image frame pair and a second bounding box in a second image frame of the another continuous image frame pair, or calculate another intersection-over-union (IoU) of a first mask in the first image frame of the another continuous image frame pair and a second mask in the second image frame of the another continuous image frame pair; as well as A detection function is configured to detect that the occlusion rate of the tracked object has changed under the condition that the other IoU differs from the calculated IoU of the plurality of consecutive image frame pairs by more than a threshold amount.

9. The device according to claim 8, wherein: The circuit is further configured to perform: A third calculation function is configured to calculate the average or median of the IoU of the plurality of consecutive image frame pairs, The detection function is configured to detect that the occlusion rate of the tracked object has changed under the condition that the difference between the other IoU and the average or median of the determined IoUs of the multiple consecutive image frame pairs exceeds a threshold amount.

10. The device according to claim 9, wherein: The circuit is further configured to perform: a fourth calculation function configured to calculate the variance of the IoU of the plurality of consecutive image frame pairs; and A setting function is configured to use the calculated variance to set the threshold.

11. An apparatus comprising a circuit configured to perform: a determination function configured to determine, for each of a plurality of image frames of a video sequence, a bounding box or mask of a tracked object; A first calculation function is configured to calculate, for each of a plurality of consecutive image frame pairs in the plurality of image frames, an intersection-over-union (IoU) ratio of a first bounding box in a first image frame of the consecutive image frame pair and a second bounding box in a second image frame of the consecutive image frame pair, or calculate an intersection-over-union (IoU) ratio of a first mask in the first image frame of the consecutive image frame pair and a second mask in the second image frame of the consecutive image frame pair; A second calculation function is configured to calculate, for another continuous image frame pair after the multiple continuous image frame pairs in the multiple image frames, another intersection-over-union (IoU) of a first bounding box in a first image frame of the another continuous image frame pair and a second bounding box in a second image frame of the another continuous image frame pair, or calculate another intersection-over-union (IoU) of a first mask in the first image frame of the another continuous image frame pair and a second mask in the second image frame of the another continuous image frame pair; as well as A decision function is configured to decide to avoid using re-identification when tracking the object in the another consecutive image frame pair if the another IoU differs from the calculated IoU of the plurality of consecutive image frame pairs by more than a threshold amount.

12. The device according to claim 11, wherein The circuit is further configured to perform: A third calculation function is configured to calculate the average or median of the IoU of the plurality of consecutive image frame pairs, The detection function is configured to decide to avoid using re-identification when tracking the object in the other consecutive image frame pair if the other IoU differs from the average or median of the determined IoUs of the multiple consecutive image frame pairs by more than a threshold amount.

13. The device according to claim 12, wherein: The circuit is further configured to perform: a fourth calculation function configured to calculate the variance of the IoU of the plurality of consecutive image frame pairs; and A setting function is configured to use the calculated variance to set the threshold.