Method, system and computer readable medium for prioritizing objects
By determining the appropriate area of interest for feature extraction based on the priority sorting method of object detection and motion data in object tracking applications, the problem of feature extraction difficulties caused by object transformation is solved, the accuracy and efficiency of object tracking is improved, and the accuracy and efficiency of object tracking is reduced. Error tracking risks.
Patent Information
- Application Number
- CN202410588504.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-05-13
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-05-13
AI Technical Summary
In object tracking applications, during object re-identification, the feature extraction and matching is difficult due to transformations such as the appearance, orientation, proportion and occlusion of the object. The existing technology relies on unverified object motion prediction, which easily leads to error tracking and feature vector leakage. , reducing tracking accuracy.
By receiving image frames and motion data, determine the area of interest (ROI), sort the objects based on object detection data and motion data prioritize them, avoid feature extraction when objects overlap exceed the threshold, reduce the risk of feature vector leakage, and select appropriate objects for feature Extract, reduce dependence on future motion prediction.
It improves the accuracy and flexibility of object tracking, reduces the risks of error tracking and feature vector leakage, realizes low complexity and efficient feature extraction, and enhances the performance of object re-identification.
Smart Images

Figure CN118968009A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of object re-identification technology, and in particular, to a method, system, and non-transitory computer-readable medium for prioritizing objects for feature extraction for the purpose of object re-identification in object tracking applications. Background Art
[0002] In object tracking applications, re-identification refers to the process of identifying an object in a video sequence as the same object that was previously detected in one or more previous frames of the video sequence or detected in a frame of another video sequence. This is an important task in object tracking because it allows the system to keep track of the movement of an object across image frames of a video sequence.
[0003] The process of object re-identification can be challenging because objects in a video sequence may undergo various transformations over time (e.g., changes in appearance, orientation, scale, and occlusion). These changes can make it difficult to track an object continuously from one image frame to the next.
[0004] To address these challenges, object tracking systems typically use a combination of object detection, feature extraction, and matching techniques to identify and track objects in video sequences.
[0005] Feature extraction includes extracting a set of descriptive features (feature vectors) from the objects detected in each frame. These features may include color, texture, shape, or other characteristics that can be used to distinguish one object from another object. Feature extraction may be sensitive to object occlusion. For example, if two objects partially overlap, the feature extraction process may extract a feature vector representing the combination of the two objects, rather than extracting two feature vectors for each object separately. This may cause confusion during the matching process because the extracted features may not match well with the corresponding features in the previous frame.
[0006] Furthermore, it may not be possible or appropriate to determine a feature vector for each object detected in each image frame of a video sequence, as this may increase the computational resources required for both the feature extraction process and the matching process.
[0007] Therefore, improvements are needed in this context. Summary of the invention
[0008] In view of the above, it would be beneficial to address or at least reduce one or more of the above discussed disadvantages, as set out in the accompanying independent patent claims.
[0009] According to a first aspect of the present invention, there is provided a method for prioritizing objects for feature extraction for the purpose of object re-identification in an object tracking application, comprising the following steps: receiving an image frame depicting a scene including a plurality of objects; receiving object detection data, the object detection data comprising positioning data indicating, for each of the plurality of objects, the position and spatial extent of the object in the image frame; receiving motion data indicating one or more motion regions in the image frame, each motion region corresponding to a region in the scene in which motion has been detected; for each motion region, determining a region of interest (ROI) in the image frame for object feature extraction, the ROI for object feature extraction overlapping with the motion region; for each ROI for object feature extraction, using the object detection data to determine a list of objects that at least partially overlap with the ROI for object feature extraction, and associating the ROI for object feature extraction with the list of objects.
[0010] The method further includes the following steps: for each ROI for object feature extraction associated with two or more objects, determining a list of prioritized candidate objects for feature extraction in the following manner: for each object of the two or more objects, when it is determined that the object does not overlap with any other object of the multiple objects by more than a threshold amount, adding the object to the list of prioritized candidate objects for feature extraction.
[0011] The method further comprises the steps of selecting at least one object from the prioritized list of candidate objects, and determining, for each selected object, a feature vector of the object based on image data of the image frame according to positioning data of the selected object.
[0012] Advantageously, the present method can provide low complexity and efficient selection criteria against which objects can be analyzed to determine feature vectors for object re-identification. For example, the method does not rely on future object motion predictions for identifying object tracks that may be at risk of being lost or falsely associated. Instead, motion data indicative of one or more motion regions in the (currently analyzed) image frame is used. The motion data can be obtained by motion detection techniques such as frame differencing that do not require prediction. By avoiding reliance on future object motion predictions, the present method reduces the risk of associating erroneous object tracks with detected objects.
[0013] In addition, the method of the present invention does not rely on existing object tracking or the objects it is associated with, thereby increasing the flexibility of the method. The selection of the object for feature vector determination is separated from the existing object tracking and its association, which can provide greater freedom to select the most appropriate object for feature vector determination.
[0014] In addition, the current method can reduce the risk of feature vector leakage between objects, thereby reducing the possibility of associating an erroneous object track with an object detected in an image frame. By reducing the risk of feature vector leakage, the current method improves the accuracy of object tracking and reduces the incidence of incorrect association between object tracks and objects in image frames.
[0015] Furthermore, by prioritizing motion regions where two or more objects in an image are located, objects in subsequent image frames that may be occluded or cause occlusions are prioritized for feature extraction. This approach prioritizes objects in subsequent image frames that an object tracker may lose track of.
[0016] In some embodiments, the step of determining the ROI for object feature extraction includes extending the motion region by a predetermined range in each direction. Advantageously, a low-complexity approach that takes into account motion in a scene as captured by an image frame can be implemented. The ROI can be determined so that the ROI includes a motion region that is extended by an increased margin (predetermined range) around it. The motion region can be extended to identify objects that may potentially occlude each other in the near future or have been occluded not long ago, and these motion regions are selected for feature vector determination to facilitate object re-identification. By utilizing this method, an effective and efficient means for analyzing scene motion to identify relevant objects for feature vector determination and subsequent object re-identification can be implemented.
[0017] In some embodiments, the motion data further includes an indication of the speed of the motion detected in the corresponding area of the scene for each motion region in the image frame, wherein the step of determining the ROI for object feature extraction includes expanding the motion region based on the speed. Therefore, the ROI for object feature extraction is determined by merging the motion region with an additional margin around it, the additional margin being based on the characteristics of the detected motion. This approach can achieve flexible and accurate analysis of scene motion to identify suitable objects for feature vector determination and subsequent object re-identification. By utilizing this approach, the current embodiment can achieve a targeted and adaptive means of selecting an ROI for feature extraction, thereby improving object tracking and re-identification performance.
[0018] In some embodiments, the ROI for object feature extraction is determined by extending the motion region to a greater extent in a direction corresponding to the direction of the velocity than in a direction not corresponding to the direction of the velocity. Advantageously, when determining the ROI for object feature extraction, possible future motion in the scene captured by the first image can be taken into account, which can improve the identification of suitable objects for feature vector determination and subsequent object re-identification.
[0019] In the example, the shape of the ROI for object feature extraction is one of a pixel mask, a circle, an ellipse and a rectangle. Advantageously, a flexible method for determining the ROI for object feature extraction can be implemented. For example, a motion region can be used to determine a pixel mask shaped ROI, which can be subjected to a morphological operation (e.g., dilation) to expand it. Alternatively, a circular shaped ROI can be obtained by determining the minimum circle enclosing the motion region, which is optionally dilated in addition. An ellipse can be determined similarly to a circle, which is optionally expanded according to the direction of the speed. When determining a rectangular shaped ROI, the same strategy can be utilized.
[0020] In some examples, the threshold amount is 0. This means that no overlap between objects detected in the ROI used for object feature extraction is allowed, which in turn can reduce the risk of feature vector leakage between objects, thereby reducing the possibility of erroneous object tracking association with objects detected in the image frame. In the context of the present disclosure, feature vector leakage refers to the pixel data of overlapping objects affecting or making the feature vector calculation of the overlapped objects inferior. Pixel leakage between bounding boxes can corrupt feature vectors and, for example, make feature vectors more similar than they should be due to the fact that bounding boxes share pixels. Therefore, feature vector leakage can lead to inaccuracies in feature representation and inaccuracies in subsequent analysis or tasks (e.g., re-identification) performed on the object.
[0021] In other examples, a portion of overlap is allowed, such as making the threshold amount a predetermined percentage of the spatial extent of one of the overlapping objects. For example, assume that a first object of size X in an image frame overlaps with a second object that is 100 times larger (size 100X) in the image frame. The second object overlaps the first object by only 1%, while the first object overlaps the second object by 100%. In the case where the threshold amount is 1% (including 1%), the second object will be added to the list of prioritized candidate objects for feature extraction, while the first object will not be added. Therefore, the size of the object is taken into account, which may play a role in the effect of feature vector leakage between objects.
[0022] In some embodiments, a maximum of N objects from the prioritized list of candidate objects are selected, where N is a predetermined number. Advantageously, N may be selected taking into account the computational resources available for object re-identification.
[0023] In an example, when the list of prioritized candidate objects consists of more than N objects, the step of selecting includes: comparing each prioritized candidate object with one or more selection criteria and selecting (up to) N objects that satisfy at least one of the one or more selection criteria. The selection criteria may be related to object size, object detection probability, object motion, object type, object shape, object orientation, etc. As a result, the current example may provide a flexible method for identifying appropriate objects for feature vector determination and object re-identification taking into account any constraints imposed by limited computing resources. This method may allow for efficient use of available resources and improve the performance of object tracking and re-identification tasks.
[0024] In some examples, when the list of prioritized candidate objects consists of a plurality of N objects, the step of selecting includes: sorting each prioritized candidate object according to one or more sorting criteria and selecting the highest ranked N objects. For example, the N objects may be sorted according to object size, object detection score, object motion, object type, object shape, object orientation, etc. As a result, the current example may provide a flexible method for identifying appropriate objects for feature vector determination and object re-identification taking into account any constraints imposed by limited computing resources. This method may allow for efficient use of available resources and improve the performance of object tracking and re-identification tasks.
[0025] In some examples, the method further comprises the step of associating, for each selected object, the determined feature vector with at least part of the localization data of the selected object. When performing object re-identification, the localization data may be used to determine the relevance of the feature vector for a particular track.
[0026] According to some embodiments, the method further comprises the step of associating, for each selected object, the determined feature vector with a timestamp indicating the capture time of the image frame. During the object re-identification process, the timestamp may be utilized to evaluate the suitability of the feature vector for a particular object track. For example, the maximum allowed distance between the position of the selected object and the track may be adjusted based on the time difference between the latest object instance in the track and the timestamp of the selected object. This approach may enable more accurate and relevant object re-identification taking into account the time-dependent dynamics of the tracked object.
[0027] According to a second aspect of the present invention, the above-mentioned purpose is achieved by a system for prioritizing feature extraction for object re-identification in an object tracking application, the system comprising: one or more processors; and one or more non-temporary computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the system to perform actions comprising the following items: receiving an image frame depicting a scene including multiple objects; receiving object detection data, the object detection data comprising positioning data indicating the position and spatial range of each of the multiple objects in the image frame; receiving motion data indicating one or more motion regions in the image frame, each motion region corresponding to an area in the scene where motion has been detected; for each motion region, determining a region of interest ROI in the image frame for object feature extraction, the region of interest ROI for object feature extraction being used for the object feature extraction. The ROI overlaps with the motion area; for each ROI for object feature extraction, using object detection data to determine a list of objects that at least partially overlap with the ROI for object feature extraction, and associating the ROI for object feature extraction with the list of objects; for each ROI for object feature extraction associated with two or more objects, determining a list of prioritized candidate objects for feature extraction in the following manner: for each of the two or more objects, when it is determined that the object does not overlap with any object in the other objects of the plurality of objects by more than a threshold amount, adding the object to the list of prioritized candidate objects for feature extraction; selecting at least one object from the list of prioritized candidate objects, and for each selected object, determining a feature vector of the object based on image data of the image frame according to positioning data of the selected object.
[0028] According to a third aspect of the present invention, the above-mentioned purpose is achieved by one or more non-temporary computer-readable media storing instructions that can be executed by one or more processors, wherein the above-mentioned instructions, when executed, cause the above-mentioned one or more processors to perform operations including the following items: receiving an image frame depicting a scene including multiple objects; receiving object detection data, the object detection data including positioning data indicating the position and spatial range of each object in the image frame for each of the multiple objects; receiving motion data indicating one or more motion regions in the image frame, each motion region corresponding to an area in the scene where motion has been detected; for each motion region, determining a region of interest ROI in the image frame for object feature extraction, the ROI for object feature extraction overlapping with the motion region; for each motion region for object feature extraction, determining a region of interest ROI in the image frame for object feature extraction ROIs, using object detection data to determine a list of objects that at least partially overlap with the ROI for object feature extraction, and associating the ROI for object feature extraction with the list of objects; for each ROI for object feature extraction associated with two or more objects, determining a list of prioritized candidate objects for feature extraction in the following manner: for each of the two or more objects, upon determining that the object does not overlap with any of the other objects in the plurality of objects by more than a threshold amount, adding the object to the list of prioritized candidate objects for feature extraction; selecting at least one object from the list of prioritized candidate objects, and for each selected object, determining a feature vector for the object based on image data of the image frame according to positioning data of the selected object.
[0029] The second and third aspects may generally have the same features and advantages as the first aspect.It should further be noted that the present disclosure relates to all possible combinations of features, unless explicitly stated otherwise. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The above and additional objects, features and advantages of the present invention will be better understood through the following illustrative and non-limiting detailed description of embodiments of the present disclosure with reference to the accompanying drawings, in which like reference numerals will be used for similar elements, wherein:
[0031] Figure 1 shows a scenario of prioritizing feature extraction for object re-identification according to an embodiment;
[0032] Figures 2 to 4 A schematic example of determining a region of interest for object feature extraction according to an embodiment is shown;
[0033] Figure 5 illustrating overlapping bounding boxes of objects according to an embodiment;
[0034] Figure 6 A flow chart showing a method for prioritizing feature extraction for object re-identification in an object tracking application according to an embodiment is shown;
[0035] Figure 7 A flowchart showing a method of determining a prioritized list of candidate objects for feature extraction from one or more regions of interest for object feature extraction according to an embodiment;
[0036] Figure 8 The implementation according to the embodiment is shown Figure 6 to Figure 7 system of methods. DETAILED DESCRIPTION
[0037] Object tracking involves the process of detecting and monitoring the movement of objects in a scene captured through a series of image frames. In the event that the tracking algorithm fails or loses track of the object due to any reason, re-identification helps to regain the lost track. Re-identification can be computationally complex due to several factors. For example, in order to perform re-identification, it may be necessary to extract features from the object that can effectively capture its appearance, pose, and other unique attributes. When matching objects, the high dimensional feature space obtained may lead to increased computational requirements. In addition, the need for real-time processing and the use of computationally intensive deep learning models for extracting features from objects may lead to a situation where insufficient available computing resources are available, especially when a large number of objects are detected in the scene.
[0038] One reason for the need for re-identification in object tracking applications is occlusion. Objects may be partially or completely occluded by other objects, making it difficult for the tracking system to maintain continuous tracking. Re-identification (ReID) helps to identify the object once it reappears from behind the object that applies the occlusion. However, occlusion may complicate the extraction of feature vectors (appearance vectors, ReID vectors, etc.) of the object that applies the occlusion or the occluded object. For example, when people in a crowd partially occlude each other, the pixel data related to the first person who partially occludes the second person (i.e., the pixel data from the bounding box of the first person) may affect the feature vector calculation for the second person or make the calculation inferior. This scenario may complicate the process of accurately distinguishing and identifying each individual.
[0039] The present disclosure aims to solve the above two problems by providing a method, system, and non-transitory computer-readable medium for prioritizing feature extraction for object re-identification in object tracking applications.
[0040] Figure 1A scene captured by five image frames 100a to 100e is shown. In this scene, three objects 102, 104, 112 appear. Figure 1 In the scene, all the objects are people, but it should be noted that the techniques described herein can be applied to any appropriate object type (e.g., cars, bicycles, animals, etc.). The scene can be any type of scene (e.g., indoor scene (room, corridor, library, convenience store, shopping mall, building entrance, etc.) or outdoor scene (road, park, suburb, beach, etc.)). The techniques discussed here can provide greater advantages in scenarios with a larger number of objects. However, for simplicity of explanation, Figure 1 The case where there are only three objects 102, 104, and 112 is demonstrated. Figure 1 , the positions and spatial extents of the objects 102, 104, 112 are shown as dashed (long dashed) rectangles 106, 108, and 114. Thus, in this example, the positions and spatial extents are shown as bounding boxes. However, depending on the object detection algorithm applied thereto, the form of the positions and spatial extents of the detected objects 102, 104, 112 may be different and may include segmentation masks, bounding polygons, etc.
[0041] In addition, Figure 1 In FIG. 1 , the regions of interest ROI 110, 116 for object feature extraction in the image frame are shown as dashed (short dashed) rectangles. Each ROI 110, 116 is determined based on the motion region (the region in the scene where motion has been detected), which will be combined with the following Figures 2 to 4 Further discussion. It is important to note that Figure 1 The ROIs 110 and 116 depicted in FIG. 1 are included for illustration purposes and may not be accurately represented as they would be identified in an actual occurring scenario.
[0042] Each ROI 110, 116 overlaps with a corresponding moving object 104, 112 in the scene. The method for prioritizing feature extraction in object re-identification discussed herein focuses on limiting feature extraction to ROIs that are at least partially overlapped by at least two objects, which can limit feature extraction to objects that are at risk of being occluded or that may have been occluded recently. Figure 1 In the context of , it means that at least two bounding boxes 106 , 108 , 114 intersect with the ROIs 110 , 116 .
[0043] In a first image frame 100a, two persons 102, 104 are shown. The first person 102 is stationary, while the second person 104 is moving. In this first frame, the two persons are not sufficiently close to allow feature extraction for re-identification purposes as described herein to work. However, it should be noted that according to some embodiments, feature extraction may be performed on the two persons 102 and 104 even if they are not determined to be prioritized objects for feature extraction as described herein, provided sufficient computing power is available for feature extraction and re-identification.
[0044] In the second image frame 100b, a moving third person 112 enters the scene. In addition, the first person 102 and the second person 104 are closer to each other. More specifically, the two people 102, 104 at least partially overlap with the ROI 110 (i.e., the bounding boxes 106, 108 of the people 102, 104 at least partially overlap with the ROI 110). Therefore, re-identification may be required in the future to maintain continuous tracking of the people 102, 104. However, in order to avoid the risk that the feature vector calculated for the first person 102 based on its visual appearance in the image frame 100b may "leak" into the feature vector calculated for the second person 104 and vice versa, the overlap between the people 102, 104 (and also the person 112) is checked. In the image frame 100b, the people do not overlap, that is, the bounding boxes 106, 108 of the people 102, 104 do not overlap. Therefore, the risk of "leakage" is determined to be low, and therefore, persons 102, 104 are determined to be prioritized candidates for feature extraction (by Figure 1 106 in boldface rectangles 108, 106). The third person 112 is determined not to be a prioritized candidate object for feature extraction because the third person is sufficiently far away from any of the other objects 102, 104 as discussed herein.
[0045] In the third image frame 100c, both the first person 102 and the second person 104 still at least partially overlap the ROI 110. However, unlike the second image frame 100b, the persons 102, 104 now overlap each other (i.e., the bounding boxes 106, 108 of the persons 102 and 104 overlap). Figure 5 As will be discussed, in the case where objects overlap beyond a threshold range, these objects are not considered as prioritized candidates for feature extraction because the risk of leaking the visual appearance of these overlapping objects is considered too high when determining the feature vectors of the overlapping objects. Figure 1In the example of FIG. 1 , in the third image frame 100 c, the first person 102 and the second person 104 are considered to have too high an overlap range, and therefore, the persons 102, 104 are not determined to be prioritized candidates for feature extraction. Since the third person 112 is far enough away from any of the other objects 102, 104, the third person 112 is determined not to be a prioritized candidate for feature extraction, as discussed herein.
[0046] In the fourth image frame 100d, both the first ROI 110 and the second ROI 116 are associated with two objects in the scene, which means that the two objects at least partially overlap with each of the ROIs 110, 116. Both the first person 102 and the second person 104 still at least partially overlap with the first ROI 110. In addition, both the first person 102 and the third person 112 at least partially overlap with the second ROI 116. For the first ROI 110, it is determined that the associated objects 102, 104 do not overlap with each other (or do not overlap with the object 112), and therefore, the objects 102, 104 are added to the list of prioritized candidate objects. For the second ROI 116, it is determined that the associated objects 102, 112 do not overlap with each other (or do not overlap with the object 104), and therefore, the objects 102, 112 are added to the list of prioritized candidate objects. It should be noted that in some embodiments, if an object has already been added to the list of prioritized candidate objects, it cannot be added again. In other embodiments, when at least one object in the prioritized list of candidate objects is selected for determining a feature vector, any duplicates are removed, as further described below.
[0047] In the fifth image frame 100e, all objects are far enough apart that no prioritized candidate objects are determined for feature extraction. In other words, none of the ROIs 110, 116 is associated with at least two objects, as described herein.
[0048] Figures 2 to 4 By way of example, it is shown how to determine the ROI 110 based on the motion region 202 in the image frame. Figures 2 to 4 In FIG. 1 , a cropped portion of an image frame is shown. The cropped portion includes an object 104, along with a motion region 202 from the image frame. Figures 2 to 4 In the example of FIG. 2 , the motion region 202 is shown as a pixel mask 202. However, other ways of representing the motion region in the image frame may be used (e.g., polygons, rectangles, etc.). The motion region 202 may be determined by identifying changes in the scene over time (e.g., using frame differences, background subtraction, temporal filters, etc.), typically by comparing consecutive or adjacent frames.
[0049] exist Figure 2 , four examples of how to determine the ROI 110 based on the motion region 202 are shown. The upper left example shows a case where the ROI 110 has the shape of the pixel mask 110. Figure 2 In the upper left example of FIG. 8 , ROI 110 has the same form and extent as motion region 202 . Figure 2 The lower left example in shows a case where the ROI 110 has a circular form and completely overlaps the motion region 202 . Figure 2 The upper right example in shows a case where the ROI 110 has the form of an ellipse and completely overlaps the motion region 202 . Figure 2 The lower right example in shows a case where the ROI 110 has a rectangular form and completely overlaps the motion region 202 .
[0050] Figure 3 Four other examples of how to determine the ROI 110 based on the motion region 202 are shown. In some embodiments, determining the ROI for object feature extraction includes extending the motion region by a predetermined range in each direction. Figure 3 In the example of , the motion region 202 is expanded uniformly in each direction. However, in other examples, other strategies may be used to expand the motion region 202, such as expanding to a greater extent in the horizontal direction than in the vertical direction. Typically, motion in a scene has a greater velocity in the horizontal direction than in the vertical direction, which may be advantageously captured by such an embodiment. Figure 3 In each of the four examples, Figure 2 corresponds to the example of Figure 2 Compared with ROI 110, Figure 3 The ROI 110 in is expanded uniformly in all directions.
[0051] Figure 4 Four other examples of how to determine ROI 110 based on motion region 202 are shown. In some embodiments, motion region 202 in an image frame is associated with an indication of the speed of the detected motion. There are several options on how to achieve this, for example:
[0052] Motion detection via optical flow: This method tracks how different parts of an image move between frames and calculates the speed of the motion.
[0053] Spatiotemporal motion detection: This method separates foreground elements from background elements, looks for gradients in the “image volume” that change over time, and uses motion information and direction to identify blobs to calculate velocity.
[0054] • Solution via neural network: This approach involves training a neural network to recognize and track motion in an image, and then using the network to detect and calculate the velocity of the moving areas.
[0055] • Output from the tracker from the previous frame: The method consists in associating the tracked object from the previous frame with the motion region in the current frame to assume that the velocity of the object is equal to the velocity of the motion region.
[0056] exist Figure 4 , the speed is represented by arrow 402. Therefore, Figure 4 The velocity in the example of 402 is horizontal and has a direction to the right. Thus, the ROI 110 can be expanded based on the velocity, for example, to a greater extent in a direction corresponding to the direction of the velocity 402 than in a direction not corresponding to the direction of the velocity 402. In another example, the ROI can be expanded in the direction of the velocity and in a direction opposite to the direction of the velocity so as to prioritize the objects 104 both before and after occlusion. In yet another embodiment, the ROI can be expanded based on both the direction and magnitude of the velocity, such that a larger magnitude causes a greater expansion than a smaller magnitude.
[0057] exist Figure 4 In each of the four examples, Figure 3 corresponds to the example of Figure 3 The ROI 110 is uniformly expanded in every direction. Figure 4 The ROIs 110 in FIG. 1 are all expanded according to the speed. Therefore, the size of the ROI expansion can be a function of the speed of the object, that is, how far the object is expected to move in a specific time frame. In addition, the analysis speed of the system can be considered, that is, how quickly the system can perform the necessary analysis. In order to ensure that the system has enough time to analyze the object before it is occluded (or leaves the scene captured by the video sequence including the image frames), a margin can be added to the size of the ROI expansion. This margin allows the system to have more time to perform the necessary analysis before the object is lost or occluded.
[0058] Figure 5Three overlapping objects, schematically represented by bounding boxes 106, 108, 114, are shown by way of example. As discussed herein, visual appearance leakage may occur when overlapping objects are not properly separated during the feature extraction process. For example, if one object partially covers another object, the feature extraction process may inadvertently include information from both objects in the overlapping region. As a result, the feature vector of the partially occluded object may contain visual appearance characteristics of the occluded object. In some cases, overlaps below a certain threshold may still produce sufficiently accurate feature vectors. This is because, despite the overlap, the majority of the image data used to determine the object's feature vector may still be derived from the correct object. In Figure 5 In FIG. 5 , the bounding box 106 of the first object overlaps the bounding box 108 of the second object by a certain range 506, which means that the area 504 does not overlap with any other object. The bounding box 108 of the second object overlaps the bounding boxes 106 and 114 of both objects, and the overlapping range is Figure 5 The areas 506 and 510 in , indicate that this means that the area 502 does not overlap with any other object. Therefore, the bounding box 114 of the third object overlaps the bounding box 108 of the second object to a certain extent 510, which means that the area 508 does not overlap with any other object.
[0059] According to an embodiment, when it is determined that an object does not overlap with any other object by more than a threshold amount, the object can be considered as a prioritized candidate object for feature extraction. In some embodiments, the threshold amount is 0. In some embodiments, the threshold amount (i.e., the allowed overlap range) is a predetermined percentage of the spatial range of one of the overlapping objects. Examples of predetermined percentages may be 1%, 5%, 7%, 10%, 25%, etc. For example, for the case of overlap between a first object and a second object, the overlapped range 506 may be considered too high compared to the total area of the bounding box 106 of the first object, which means that the first object is not included in the prioritized candidate objects as described herein. For the case of overlap between a second object and a third object, the overlapped range 510 (i.e., the overlapped area 510 compared to the total area of the bounding box 114 of the third object) may be considered to be no more than the threshold, which means that the third object is included in the prioritized candidate objects as described herein. For a second object that overlaps with two objects, in some embodiments, the total overlapped area (506+510) is compared to the entire area of the bounding box 108 to determine whether the second object should be included in the prioritized candidate objects. In other embodiments, if one of the areas 506 , 510 is deemed to be above a threshold compared to the total area of the second object's bounding box 108 , the second object is not included among the prioritized candidate objects.
[0060] Figure 6 A flowchart of a method 600 for prioritizing feature extraction for object re-identification in an object tracking application is shown by way of example. Figure 8 The method 600 is described in detail. The method 600 comprises a step of receiving S602, by a receiving means 808, an image frame 100a depicting a scene comprising a plurality of objects. The image frame may be, for example, a frame from a video sequence 802 depicting a scene captured by a camera.
[0061] Method 600 further includes a step of receiving S604 object detection data 805 (the object detection data 805 includes positioning data indicating the position and spatial range of the object in the image frame for each of the multiple objects) by a receiving component 808. Object detection data can be received from the object detector component 804. The object detector component can implement any appropriate traditional algorithm for object detection (e.g., Haar cascade classifier, oriented gradient histogram (HOG), region-based convolutional neural network (R-CNN), etc.). The object detector component may include an artificial intelligence (AI) or machine learning (ML) algorithm trained to detect an object of interest in an image. AI / ML is an appropriate technique for detecting objects in an image and can be relatively easily trained using a large labeled / annotated image data set containing an object type of interest. It should be noted that there are several object detection algorithms that do not rely on neural networks. These methods can be based on classical computer vision such as scale-invariant feature transform (SIFT), template matching, etc.
[0062] The method 600 further comprises receiving S606, by a receiving component 808, motion data 807 indicating one or more motion regions in the image frame, each motion region corresponding to an area in the scene in which motion has been detected. In some embodiments, the motion data further comprises, for each motion region in the image frame, an indication of the speed of the motion detected in the corresponding area of the scene. The motion data 807 may be received from a motion detector component 806. The motion detector may generally implement any suitable algorithm (as described herein) for detecting motion in a scene (captured by a video sequence) by comparing consecutive or adjacent frames.
[0063] The method 600 further includes determining, for each motion region, a ROI in the image frame for object feature extraction by a region of interest ROI determination component 810 S608, wherein the ROI for object feature extraction overlaps with the motion region. The ROI can be determined by the ROI determination component 810 (as described above in conjunction with Figures 2 to 4) is further expanded S610 (eg, by expanding the ROI derived from the motion region).
[0064] The method 600 further includes determining, for each ROI for object feature extraction, a list of objects at least partially overlapping the ROI for object feature extraction using the object detection data, and associating the ROI for object feature extraction with the list of objects by the ROI associating component 812 S612 .
[0065] The method 600 further includes determining, by the candidate object determination component 814, a list of prioritized candidate objects for feature extraction, as follows: Figure 7 Will be described further.
[0066] Method 600 further includes selecting at least one object from the list of prioritized candidate objects by candidate object selection component 816. Selecting S616 may include removing duplicates from the list of prioritized candidate objects. Selecting S616 may limit the number of objects used to determine feature vectors to a specific number (N). Such a restriction may further include comparing each prioritized candidate object with one or more selection criteria, and selecting (at most) N objects that meet at least one of the one or more selection criteria. The selection criteria may be configured based on the application or the requirements of the application. For example, the selection criteria may include selecting an object of a specific object type and / or an object of a specific size and / or an object associated with the speed of motion, etc. In an embodiment, the object may be associated with the speed of motion when the motion region associated with the speed of motion overlaps a high range (complete overlap, 95% overlap, etc.). In some embodiments, such a restriction may include sorting each prioritized candidate object according to one or more sorting criteria, and selecting the highest sorted N objects. The sorting criteria may, for example, include sorting based on object size, sorting based on confidence scores from an object detector, sorting based on the speed of the object, etc.
[0067] The method 600 further includes determining, by a feature vector determining component 818, for each selected object, a feature vector of the object based on image data of the image frame according to the positioning data of the selected object. Any suitable algorithm for determining a feature vector from image data may be used. Example algorithms include:
[0068] Scale Invariant Feature Transform (SIFT): SIFT is a feature extraction algorithm that detects and describes local features in an image. It can be considered robust to changes in scale and rotation.
[0069] Speeded Up Robust Features (SURF): SURF is a feature extraction algorithm similar to SIFT, but SURF can be considered faster and more robust to changes in lighting and viewpoint.
[0070] • Histogram of Oriented Gradients (HOG): HOG is a feature extraction algorithm that counts the occurrences of gradient orientations in local parts of an image.
[0071] Convolutional Neural Network (CNN): CNN is a deep learning algorithm that can be used to extract features from images. CNN learns features from images by using multiple layers of convolution and pooling.
[0072] Local Binary Pattern (LBP): LBP is a texture descriptor that extracts local features from an image by comparing each pixel with its surrounding neighbors.
[0073] Feature extraction using object or sub-object classification functions: This technique involves detecting, for example, a person’s clothing, accessories (the color of a sweater, the handbag they are wearing, the glasses they are wearing, the hat they are wearing, etc.), the type of vehicle (red pickup truck, black SUV, etc.), or similar features of an object.
[0074] The feature vector may then be used by a re-identification component 820, for example, to identify a selected object in an image frame as the same object previously detected in one or more previous frames of the same video sequence 802 or detected in another video sequence. In another embodiment, the feature vector may be transmitted to a tracker, which may use the feature vector to create a more robust track or to recreate a track that has been lost due to occlusion. An example of using the feature vector for this purpose is described in "Simple Online and Real-Time Tracking Using Deep Association Metrics" (published by Wojke et al. at https: / / arxiv.org / abs / 1703.07402 at the time of filing this application).
[0075] In some embodiments, the determined feature vector can be associated with at least part of the positioning data of the corresponding object selected. For example, the position of the selected object can be associated with the feature vector. Such information can be used by the re-identification component 820 to determine the relevance of the feature vector of the track that has been lost in the video sequence. For example, some object movements may be considered unreasonable. Very large movements (such as crossing the image only in one frame) may instead look like two different objects that are similar. The tracker including the re-identification component 820 usually has a limited area based on the latest detection of the object (the detection expects the object to reappear therein), based on the speed of the object and a certain margin of error. If the object is lost (for example, when a person walks behind a bush), the radius is usually increased to achieve re-association. The association with a given track based on the detection outside the radius is usually "expensive" (requiring very safe matching) or is prohibited.
[0076] In some embodiments, the determined feature vector may be associated with a timestamp indicating the capture time of the image frame. Such information may be used by the re-identification component 820 to determine the relevance of the feature vector for a track that has been lost in the video sequence. For example, the timestamp may be used to determine an acceptable distance between the location of a selected object and the last instance of a track that was lost. Additionally, when performing long-term tracking, the timestamp may be used to determine the relevance of the feature vector.
[0077] Figure 7 By way of example, Figure 6 More details of the step of determining the list of prioritized candidate objects for feature extraction in S614 are provided in FIG. Figure 6 Step S614 is performed for all ROIs determined in step S608 of the method (and possibly expanded in S610). Thus, when all ROIs have been checked S702, the step of determining S614 a prioritized list of candidate objects for feature extraction ends S712. For each ROI, it is checked S704 whether the ROI is associated with two or more objects (i.e., the number of objects that at least partially overlap with the ROI as determined above in the associating step S612). If so, each of the two or more objects associated with the ROI is analyzed S708 to determine whether the object should be added S710 to the prioritized list of candidate objects. Thus, the method includes an analyzing step S708 of checking whether the object overlaps with any other object in the plurality of objects by more than a threshold amount. A plurality of objects detected in an image frame (i.e., as determined in the Figure 6If no other object among the objects identified in the object detection data received in step S604 overlaps the object currently being analyzed by more than a threshold amount, the object currently being analyzed is added S710 to the prioritized list of candidate objects. The analyzing step S708 (and adding step S710, if applicable) is performed until all objects (such as those determined in step S706) have been analyzed. When it is determined that all objects associated with the ROI have been analyzed S706, the next ROI is checked and so on.
[0078] Figure 6 and Figure 7 The methods shown in and any other methods or functions described herein may be stored as instructions on a non-transitory computer-readable storage medium so that when these instructions are executed on a device or system with processing capabilities, these methods are implemented. Such a device or system (e.g., Figure 8 ) may include one or more processors. In an example, system 800 may be implemented in a single device such as a camera. In other examples, some or all of the different components (modules, units, etc.) 804 to 820 may be implemented in a server or in the cloud. In some examples, the camera implements Figure 8 All components except the re-identification component 820 which can be implemented by, for example, a server. In some examples, the camera implements Figure 8 All components except the feature vector determination component 818 and the re-identification component 820 which can be implemented by, for example, a server. In other examples, Figure 8A subset of the components (e.g., image capture (of captured video sequence 802) and motion detector 806) are implemented on the camera, while other components are implemented in the server. In general, one or more devices (camera, server, etc.) implementing components 804 to 820 may each include circuits configured to implement components 804 to 820 and more specifically implement their functions. Therefore, the features and methods described herein may be advantageously implemented in one or more computer programs (the one or more computer programs are executable on a programmable system), which may include at least one programmable processor coupled to receive data and instructions from a data storage system and transmit data and instructions to the data storage system, at least one input device such as a camera for capturing image frames / video sequences, and at least one output device such as a display for displaying images, potentially implemented in an object tracking application including the tracked objects described herein. By way of example, suitable processors for executing the program of instructions include both general-purpose microprocessors and special-purpose microprocessors and a sole processor or one of multiple processors or cores of any type of computer. The processor may be supplemented by an ASIC (Application Specific Integrated Circuit) or may be incorporated into an ASIC.
[0079] The above embodiments should be understood as illustrative examples of the present invention. Other embodiments of the present invention are contemplated. For example, motion data can be received from an object tracker component and tracked between image frames based on the movement of the object. It should be understood that any feature described with respect to any one embodiment can be used alone or in combination with other features described, and can also be used in combination with one or more features of any other embodiment or any combination of any other embodiment. In addition, equivalents and modifications not described above may also be adopted without departing from the scope of the present invention defined in the appended claims.
Claims
1. A method for prioritizing objects for feature extraction for the purpose of object re-identification in an object tracking application, comprising the following steps: receiving an image frame depicting a scene including a plurality of objects; receiving object detection data, the object detection data comprising, for each of the plurality of objects, localization data indicating a position and a spatial extent of the object in the image frame; receiving motion data indicating one or more motion regions in the image frame, each motion region corresponding to an area in the scene where motion has been detected; For each motion region, determining a region of interest ROI in the image frame for object feature extraction, wherein the ROI for object feature extraction overlaps with the motion region; for each ROI for object feature extraction, using the object detection data to determine a list of objects that at least partially overlap with the ROI for object feature extraction, and associating the ROI for object feature extraction with the list of objects; For each ROI for object feature extraction associated with two or more objects, determining a prioritized list of candidate objects for feature extraction by: for each of the two or more objects, upon determining that the object does not overlap with any of the other objects in the plurality of objects by more than a threshold amount, adding the object to the prioritized list of candidate objects for feature extraction; At least one object from the list of prioritized candidate objects is selected, and for each selected object, a feature vector of the object is determined based on image data of the image frame according to the positioning data of the selected object.
2. The method according to claim 1, wherein: The step of determining the ROI for object feature extraction includes extending the motion region by a predetermined range in each direction.
3. The method according to claim 1, wherein: The motion data further comprises, for each motion region in the image frame, an indication of a speed of the motion detected in the corresponding region of the scene, wherein the step of determining a ROI for object feature extraction comprises expanding the motion region based on the speed.
4. The method according to claim 3, wherein: The ROI for object feature extraction is determined by extending the motion region to a larger range in a direction corresponding to the direction of the speed than in a direction not corresponding to the direction of the speed.
5. The method according to claim 1, wherein: The shape of the ROI used for object feature extraction is one of a pixel mask, a circle, an ellipse and a rectangle.
6. The method according to claim 1, wherein: The threshold amount is zero.
7. The method according to claim 1, wherein: The threshold amount is a predetermined percentage of the spatial extent of one of the objects that overlap.
8. The method according to claim 1, wherein: A maximum of N objects from the list of prioritized candidate objects are selected, where N is a predetermined number.
9. The method according to claim 8, wherein: When the list of prioritized candidate objects consists of more than N objects, the step of selecting comprises comparing each prioritized candidate object with one or more selection criteria and selecting the N objects satisfying at least one of the one or more selection criteria.
10. The method according to claim 8, wherein: When the list of prioritized candidate objects consists of more than N objects, the step of selecting comprises ranking each prioritized candidate object according to one or more ranking criteria and selecting the N objects with the highest ranking.
11. The method according to claim 1, further comprising the steps of: For each selected object, the determined feature vector is associated with at least part of the positioning data of the selected object.
12. The method according to claim 1, further comprising the steps of: For each object selected, the determined feature vector is associated with a timestamp indicating the capture time of the image frame.
13. A system for prioritizing feature extraction for object re-identification in an object tracking application, comprising: one or more processors; as well as One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform actions including: receiving an image frame depicting a scene including a plurality of objects; receiving object detection data, the object detection data comprising, for each of the plurality of objects, localization data indicating a position and a spatial extent of the object in the image frame; receiving motion data indicating one or more motion regions in the image frame, each motion region corresponding to an area in the scene where motion has been detected; For each motion region, determining a region of interest ROI in the image frame for object feature extraction, wherein the ROI for object feature extraction overlaps with the motion region; for each ROI for object feature extraction, using the object detection data to determine a list of objects that at least partially overlap with the ROI for object feature extraction, and associating the ROI for object feature extraction with the list of objects; For each ROI for object feature extraction associated with two or more objects, determining a prioritized list of candidate objects for feature extraction by: for each of the two or more objects, upon determining that the object does not overlap with any of the other objects in the plurality of objects by more than a threshold amount, adding the object to the prioritized list of candidate objects for feature extraction; At least one object from the list of prioritized candidate objects is selected, and for each selected object, a feature vector of the object is determined based on image data of the image frame according to the positioning data of the selected object.
14. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein: The instructions, when executed, cause the one or more processors to perform operations including: receiving an image frame depicting a scene including a plurality of objects; receiving object detection data, the object detection data comprising, for each of the plurality of objects, localization data indicating a position and a spatial extent of the object in the image frame; receiving motion data indicating one or more motion regions in the image frame, each motion region corresponding to an area in the scene where motion has been detected; For each motion region, determining a region of interest ROI in the image frame for object feature extraction, wherein the ROI for object feature extraction overlaps with the motion region; for each ROI for object feature extraction, using the object detection data to determine a list of objects that at least partially overlap with the ROI for object feature extraction, and associating the ROI for object feature extraction with the list of objects; For each ROI for object feature extraction associated with two or more objects, determining a prioritized list of candidate objects for feature extraction by: for each of the two or more objects, upon determining that the object does not overlap with any of the other objects in the plurality of objects by more than a threshold amount, adding the object to the prioritized list of candidate objects for feature extraction; At least one object from the list of prioritized candidate objects is selected, and for each selected object, a feature vector of the object is determined based on image data of the image frame according to the positioning data of the selected object.
Citation Information
Patent Citations
System and method for boosting object detection performance in videos
CN103914702A
Object detection and tracking method and device
CN110458861A
Determining region-of-interest of object using image-based object tracking
CN113196297A
Intra-mode region-of-interest video object segmentation
US20070183663A1
Cited By
Image extraction method based on image occlusion, computer equipment and storage medium
CN120894302A
