Method of tracking an object

Abstract

The present application provides methods of tracking objects. According to an aspect, a computer-implemented method of tracking objects in a video sequence of a scene is provided, the method comprising: determining a location in the scene at which an object exits a sink of the scene and a location at which the object enters a source of the scene; tracking a first object moving in the scene using a re-identification algorithm, wherein the first object is associated with a re-identification threshold of the re-identification algorithm; detecting that the first object has exited the scene at the sink; and in response to detecting that the first object has exited the scene at the sink, adjusting the re-identification threshold associated with the first object to reduce a probability that the re-identification algorithm will re-identify a second object as the first object, the second object entering the scene at the source after the first object has exited the scene at the sink.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to a computer-implemented method of tracking objects in a video sequence of a scene. BACKGROUND

[0002] One common application of video surveillance systems is to track objects moving in a monitored scene. One computer vision-based object tracking technique is re-identification ("ReID"), in which an object of interest (e.g. a person, a vehicle, an animal) is identified in one frame and then re-identified in successive frames. The trajectory of an object (e.g. the coordinates of the object in the frames) can be recorded in a "tracklet" maintained for the object.

[0003] Re-identification has a wide range of applications, including surveillance, traffic monitoring and crowd analysis. More generally, re-identification can be useful in situations where the tracked object can be temporarily occluded during tracking. If an occluded (partially or completely) object can be re-identified as the same object as before the occlusion, the occluded object (and its trajectory) of the re-identification can be associated with the object and trajectory identified before the occlusion. Corresponding methods can be used when tracking an object that moves from a first camera perspective of a scene to a second camera perspective of the scene. SUMMARY

[0004] The inventors have realised that, in a tracking scene involving tracking objects having similar appearances (e.g. vehicles having similar brands, models and / or colours or people having similar clothing and / or appearances), using re-identification as in conventional implementations can introduce a risk of inadvertently identifying a similar but different object (e.g. a different car of the same model and similar colour) as the same object when one of the similar objects has left the scene at a sink and a similar but different object subsequently appears at a source. It is an object of the present invention to provide a method of tracking objects using re-identification that mitigates this risk.

[0005] Accordingly, according to a first aspect of the present invention, there is provided a computer-implemented method of tracking objects in a video sequence of a scene, the method comprising:

[0006] determining a location of a sink in the scene at which the object leaves the scene and a location of a source in the scene at which the object enters the scene;

[0007] tracking a first object moving in the scene using a re-identification algorithm, wherein the first object is associated with a re-identification threshold of the re-identification algorithm;

[0008] detecting that the first object has left the scene at the sink; and

[0009] in response to detecting that the first object has left the scene at the sink, adjusting a re-identification threshold associated with the first object to reduce a probability that a re-identification algorithm will re-identify a second object as the first object, the second object entering the scene at the source after the first object has left the scene at the sink.

[0010] By adjusting the re-identification threshold, a greater similarity between the second object and the first object is required for the re-identification algorithm to re-identify the second object as the first object. Thus, the method is able to reduce the risk of inadvertently re-identifying an object as a tracked object that enters the scene at the source and is visually similar to a tracked object that has previously left the scene at the sink, but corresponds to a different physical object than the tracked object.

[0011] In the case of a temporary occlusion moving through the scene, similar objects detected in the video sequence before and after the occlusion can correspond to the same physical object. Thus, in this case, re-identification is appropriate and desirable. However, the likelihood that a physical object re-enters the scene shortly after having left the scene at the sink is generally low. However, some scenes comprise a source and a sink with a relative position such that a re-entry of the same physical object at the source is still possible after the physical object has left the scene at the sink for some time. Thus, a simple approach of terminating the tracking of an object directly upon detecting that the tracked object leaves at the sink does not properly handle such leave and re-entry situations. That is, identifying the re-entering object as a new object can result in a loss of potentially relevant tracking information. In contrast, by adjusting the re-identification threshold associated with the first object, rather than terminating the tracking of the leaving object directly, the inventive method allows for correctly re-identifying an object that re-enters at the source, which has previously left the scene at the sink.

[0012] Thus, the method is able to improve the reliability of object tracking, in particular in cases where the tracking can involve visually similar objects. Furthermore, the improved reliability is provided without sacrificing the benefits related to re-identification, i.e. handling temporary occlusions of tracked objects and movements of tracked objects between different camera perspectives of the scene.

[0013] Further utility of the inventive method can be particularly apparent when tracking objects in a scene that comprises a source and a sink that are arranged next to each other or overlap.

[0014] The sink and the source can be arranged along respective lanes of a road, respectively at an exit and an entrance of a parking spot, or along an exit ramp and an entrance ramp of a road, respectively.

[0015] The sinks and sources can also be located (e.g., in applications of tracking individuals such as pedestrians and / or cyclists) on a portion of a sidewalk or footpath leading out of the scene or a respective adjacent portion, or on the exit and entrance, respectively, of a building, indoor space (e.g., a room) or outdoor space (e.g., a park).

[0016] The "re-identification threshold" here refers to a respective threshold associated with each respective tracked object, which is used by a re-identification algorithm to re-identify an object detected in a frame of the video sequence as the respective tracked object. That is, in order to re-identify an object detected in a second frame as an object identified in a previous first frame, object features extracted from the second frame need to match object features extracted from the first frame to an extent defined by the re-identification threshold.

[0017] The term "object" (e.g., "first object," "second object," etc.) refers here to a depiction in a video sequence (more specifically, in one or more frames of the video sequence) of a physical object in a scene (the physical object being of a type that is tracked, such as a vehicle or an individual). Accordingly, the term "physical object" refers to an actual physical object moving in the monitored scene. Thus, as can be understood from the foregoing discussion, two "objects" (e.g., "first object" and "second object") in a video sequence can depict the same physical object or different physical objects in the scene, depending on the scene.

[0018] In some embodiments, the method further comprises, after adjusting the re-identification threshold associated with the first object:

[0019] detecting an entry of the second object at the source;

[0020] re-identifying the second object as the first object by a re-identification algorithm using the adjusted re-identification threshold; and

[0021] subsequently continuing to track the second object as the first object.

[0022] Thus, by re-identifying the second object as the first object, a situation in which the second object is sufficiently similar to the first object that is currently being tracked (which can especially be the case when the first object and the second object actually correspond to the same physical object) can be addressed, so that the first object can be continued to be tracked.

[0023] In some embodiments, the method further comprises, after re-identifying the second object as the first object, restoring the re-identification threshold and continuing to track the second object as the first object using the restored re-identification threshold. To "restore" the re-identification threshold here means to restore or revert the re-identification threshold to a value prior to the adjustment, e.g. a predetermined default re-identification threshold. The method is based on the concept that after the exit and re-entry scenario has been resolved (i.e. the second object has been re-identified as the first object using the adjusted re-identification threshold), the original value of the re-identification threshold can be used to continue tracking the first object.

[0024] In some embodiments, re-identifying the second object as the first object comprises comparing the second object feature set of the second object (i.e. the second object features extracted from the video frames comprising the second object) with the first object feature set of the first object (i.e. the first object features extracted from the video frames comprising the first object) to generate a match score, and comparing the match score with the adjusted re-identification threshold. In this way, the match score is generated between the first object features and the second object features such that the re-identification can be assessed using a threshold test.

[0025] The match score can be defined such that the more similar the first object feature set and the second object feature set are, the higher the match score is (e.g. the match score increases as the distance between the first object feature set and the second object feature set decreases). In this case, the re-identification threshold can be adjusted by an increase. Thus, in response to the match score reaching or exceeding the adjusted (increased) re-identification threshold, the second object can be re-identified as the first object.

[0026] Alternatively, the match score can be defined such that the more similar the first object feature set and the second object feature set are, the lower the match score is (e.g. the match score decreases as the distance between the first object feature set and the second object feature set decreases). In this case, the re-identification threshold can be adjusted by a decrease. Thus, in response to the match score reaching or being lower than the adjusted (decreased) re-identification threshold, the second object can be re-identified as the first object.

[0027] In some embodiments, tracking the first object comprises maintaining a tracking context for the first object, and wherein re-identifying the second object as the first object comprises associating the second object with the tracking context of the first object.

[0028] Thus, by associating the second object with the existing tracking context of the first object, the second object can be simply "re-identified" as the first object. Thus, in a sense, the second object can inherit the previously determined tracking data (e.g. trajectory segment) of the first object.

[0029] In some embodiments, the entering of the second object at the source is detected within a predetermined time from detecting that the first object has left the scene at the sink. That is, a further condition for re-identifying the second object as the first object can be that the second object enters the scene at the source within a predetermined time from the first object leaving the sink.

[0030] In some embodiments, the method further comprises, after determining that more than the predetermined time has passed since detecting that the first object has left the scene at the sink:

[0031] detecting an entering of the second object at the source; and

[0032] tracking the second object as a different object from the first object using the re-identification algorithm and the re-identification threshold associated with the second object.

[0033] Hence, if it is determined that more than the predetermined time has passed since detecting that the first object has left the scene at the sink, it can no longer be meaningful to attempt to re-identify a new object entering the scene at the source as the first object. In this case, the method can simply determine to track the second object as a new object. In embodiments in which tracking the first object comprises maintaining a tracking context for the first object, the method can further comprise, in response to determining that more than the predetermined time has passed since detecting that the first object has left the scene at the sink, deleting the tracking context for the first object from the set of active tracking contexts maintained by the re-identification algorithm. Hence, by terminating tracking of the first object, both computational resources and memory resources can be conserved.

[0034] Furthermore, as mentioned above, there can be situations in which the second object is not sufficiently similar to the currently tracked first object to be re-identified as the first object. Hence, in some embodiments, the method can further comprise, after adjusting the re-identification threshold associated with the first object:

[0035] detecting an entering of the second object at the source;

[0036] determining that the second object is different from the first object using the re-identification algorithm and the adjusted re-identification threshold; and

[0037] subsequently tracking the second object as a different object from the first object using the re-identification algorithm and the re-identification threshold associated with the second object.

[0038] The second re-identification threshold can typically be set to the same value as the (unadjusted) re-identification threshold associated with the first object, e.g. a predetermined default re-identification threshold.

[0039] In some embodiments, the method includes determining a location of a set of sinks at which the object entered the scene, wherein the sink is one of the set of sinks, and wherein, after adjusting the re-identification threshold, the probability that the re-identification algorithm will re-identify an object entering the scene at any of the set of sinks as the first object is reduced.

[0040] That is, it is sufficient to adjust and maintain a single common re-identification threshold associated with the first object, which the re-identification algorithm can use regardless of which of the multiple sinks in the scene the second object enters the scene.

[0041] In some embodiments, the method further includes, after adjusting the re-identification threshold associated with the first object:

[0042] detecting an entry of a second object at any of the set of sinks;

[0043] re-identifying the second object as the first object by the re-identification algorithm using the adjusted re-identification threshold; and

[0044] subsequently continuing to track the second object as the first object.

[0045] Thus, the situation in which a second object sufficiently similar to a currently tracked first object enters the scene at any of the set of sinks can be resolved by re-identifying the second object as the first object, so that the first object can continue to be tracked.

[0046] In some embodiments, the sink is a first sink, and the method further includes determining a location of a second sink at which the object exited the scene, and the method further includes:

[0047] tracking a third object moving in the scene using the re-identification algorithm, wherein the third object is associated with a respective re-identification threshold of the re-identification algorithm;

[0048] detecting that the third object has exited the scene at the second sink; and

[0049] terminating tracking of the third object in response to detecting that the third object has exited the scene at the second sink, thereby preventing a fourth object entering the scene at the sink from being re-identified as the third object.

[0050] Thus, the method recognizes the fact that some scenes can include a sink (“the second sink”) such that the likelihood of a object (“the third object”) re-entering the scene at a sink after the third object exited the scene at the second sink is substantially zero. Thus, computational resources can be conserved by terminating tracking of the third object.

[0051] In some embodiments, tracking the third object comprises maintaining a tracking context for the third object, and wherein terminating tracking of the third object comprises deleting the tracking context for the third object from the set of active tracking contexts maintained by the re-identification algorithm. Thus, memory resources used to maintain the tracking context for the third object can be released.

[0052] According to a second aspect of the application, there is provided a computer program product comprising computer program code portions configured to perform the method of tracking an object in a video sequence of a scene according to the first aspect or any embodiment thereof when executed by a processing device.

[0053] According to a third aspect of the application, there is provided an object tracking system comprising:

[0054] at least one camera for capturing a video sequence of a scene; and

[0055] a processing device configured to track an object in the video sequence according to the method of the first aspect or any embodiment thereof.

[0056] The second and third aspects feature the same or equivalent benefits as the first aspect. Any functionality described in relation to the first aspect can have a corresponding feature in the system, and vice versa. BRIEF DESCRIPTION OF DRAWINGS

[0057] This and other aspects of the present application will now be described in more detail, with reference to the appended drawings, showing embodiments of the application.

[0058] Figure 1 A view of a scene is schematically illustrated.

[0059] Figure 2 An implementation of an object tracking system is shown.

[0060] Figure 3 A flowchart of a method of tracking an object in a video sequence of a scene.

[0061] Figure 4 A flowchart of an extension of the method of Figure 3 DETAILED DESCRIPTION

[0062] Figure 1 An example view of a scene 1 monitored by a camera is schematically illustrated. The view can correspond to a video frame of a video sequence captured by the camera. At the time point shown in Figure 1 At the time point shown in, scene 1 comprises a plurality of objects in the form of vehicles moving in scene 1, each object being tracked by the re-identification algorithm.

[0063] ​Scene 1 comprises a plurality of sources 2a-c and a plurality of sinks 4a-c. It is noted that Figure 1 The respective number of sources and sinks is merely a non-limiting example, and the scene can more generally comprise any number of sources and sinks, but comprises at least one source and at least one sink. Each source 2a-c represents a region of Scene 1 at or in which an object can enter Scene 1. Conversely, each sink 4a-c represents a region of Scene 1 at or in which an object can leave Scene 1. Thus, each source 2a-c represents a possible entry point for an object into Scene 1, while each sink 4a-c represents a possible exit point for an object from Scene 1.

[0064] In the illustrated example, each of sources 2a-c and sinks 4a-c is disposed along a respective lane of a road segment leading to or away from the monitored view of Scene 1. The sources and sinks can be disposed adjacent to each other as shown, e.g., along respective road lanes, e.g., in a city or town. Figure 1 In the illustrated example, source 2a and sink 4a, source 2b and sink 4b, and source 2c and sink 4c.

[0065] In the illustrated example, source 2a and sink 4a, source 2b and sink 4b, and source 2c and sink 4c. Figure 1 At the illustrated moment in time, one of the tracked objects, e.g., car 10a, has just entered Scene 1 at source 2c and is moving toward sink 4a. As car 10a moves through Scene 1, car 10a can be tracked in successive video frames using re-identification. During the movement, car 10a can be temporarily fully or partially occluded by another tracked object (e.g., another car) or by a fixed obstacle or structure (e.g., a tree canopy, a traffic light, or a building). The use of re-identification here allows car 10a to be re-identified as the same object before and after the occlusion event, so that car 10a can be associated with the currently identified object and its trajectory (e.g., a trajectory segment recording previous coordinates of the object in the frames).

[0066] More specifically, the re-identification algorithm can be implemented by extracting a set of object features of an object detected in a video frame from a plurality of consecutive video frames of a video sequence. The object features extracted from a given video frame can be compared with object features extracted from one or more previous video frames to generate a match score. The re-identification algorithm can perform a threshold test that includes comparing the match score with a re-identification threshold associated with the respective tracked object. In response to the match score passing the threshold test, it can be determined that the object features extracted from the given frame and the one or more previous frames belong to the same (physical) object, and thus the object in the given video frame can be re-identified as such. Accordingly, the location of the re-identified object in the given frame can be recorded in a track segment associated with the object. In the event that the match score fails the threshold test, the object features extracted from the object in the given video frame can be determined to be a different object from the object identified in the previous one or more video frames, and thus its location in the given frame can not be associated with the previously tracked object.

[0067] A respective tracking context can be maintained for each tracked object. The tracking context can include an object identifier and a track segment that defines a trajectory traveled by the object. When a new object is detected and identified in a video sequence, a new tracking context can be created. When tracking of an object is to be terminated at a point, further attempts to re-identify a detected object with the currently tracked object can be stopped, and the track segment of the tracking context associated with the currently tracked object can be stopped from being updated.

[0068] The re-identification algorithm can maintain a set of active or current tracking contexts, each associated with a respective object currently tracked by the re-identification algorithm. Each time a new object is detected and identified in a video sequence by the re-identification algorithm, a new tracking context can be created and added to the set of active tracking contexts. Once tracking of a given object associated with a tracking context in the set of active tracking contexts is terminated, the associated tracking context can be removed from the set of active tracking contexts. Thus, a new object that subsequently enters the scene at the source can be prevented from being re-identified as the given object (i.e., because the tracking context associated with the given object has been removed from the set of active tracking contexts). Tracking can be terminated in response to determining that the given object has exited the scene at the sink such that the likelihood of the object re-entering the scene at the source is substantially zero. Tracking can also be terminated in response to determining that more than a predetermined time has passed since the object was detected to have exited the scene at the sink.

[0069] Video sequences such as the scenario of Scenario 1 are subject to dynamic conditions (e.g., changing lighting conditions, variable weather and visibility conditions, etc.). Accordingly, each tracked object is associated with a re-identification threshold that can be set to provide a certain tolerance for appearance changes of the tracked object. The respective re-identification threshold can be included or stored in the tracking context of the tracked object. For example, over the course of several video frames, the lighting conditions in the scene can change from sunny to overcast, causing a noticeable change in the color of a tracked object, such as car 10a. As another example, an object can move into and out of an area illuminated by a streetlight. In order to be able to reliably track the object, e.g., car 10a, during such changes, the re-identification threshold can be set accordingly such that despite some changes in the appearance color of car 10a, the re-identification algorithm can still re-identify car 10a as the same object.

[0070] However, once a tracked object leaves the scene at a sink, the tolerance provided by the re-identification algorithm and the re-identification threshold can create problems. For example, consider Figure 1 car 10b in FIG. 1B is about to leave scene 1 at sink 4a. If after car 10b leaves scene 1 at sink 4a, a different car having a similar make, model, and / or color (i.e., similar within the tolerance defined by the re-identification threshold associated with car 10b) enters scene 1 at a source such as source 2a, there is a risk that the re-identification algorithm will re-identify this new, different car as the same object as car 10b, and thus continue to track the new car as the currently tracked car 10b (e.g., record further detected locations of the new car in the track segment of the tracking context of the currently tracked car 10b). On the other hand, in some cases, e.g., for adjacent source 2a and sink 4a, car 10b can make a U-turn after leaving scene 1 at sink 4a, and thus re-enter scene 1 at source 2a. In this case, it is desirable that car 10b be re-identified and continuously tracked as the same object when it enters scene 1 at source 2a.

[0071] As described in this disclosure, these seemingly competing objectives can be addressed by adjusting the re-identification threshold associated with the first tracked object in response to detecting that the first tracked object has left the scene at a sink, such that the probability that the re-identification algorithm will re-identify a second object that enters the scene at a source after the first tracked object has left the scene at the sink as the first object is reduced. In other words, the tolerance of the re-identification algorithm in attempting to re-identify the second object as the first tracked object can be reduced, such that the re-identification algorithm requires a greater visual similarity between the second object and the first object to re-identify the second object as the first tracked object.

[0072] Reference will be made to the following examples.Figure 1 And further with reference to Figures 2 to 4 Example implementations of the method are discussed in more detail.

[0073] Although Figure 1 Example implementations of the method are discussed in more detail.

[0074] Figure 2 An example implementation of an object tracking system 20 is shown. The system 20 comprises a video surveillance camera 22 for capturing a video sequence V of a scene, e.g. scene 1. Although Figure 1 A single view scene is shown, and Figure 2 A single camera 22 is shown, but the system 20 can also comprise a system of two or more cameras each monitoring a respective sub-view of the scene.

[0075] The system 20 further comprises a processing device 24 and an associated memory 26. The memory 26 can be coupled to or contained in the processing device 24. The processing device 24 is configured to receive the video sequence V from the camera 22 in the form of a sequence of video frames. As Figure 2 The video sequence V can be stored in the memory 26 as shown in which the processing device 24 can retrieve and process the video frames from the memory 26 to track objects in the video sequence in the sequence of video frames.

[0076] The processing device 24 can maintain in the memory 26 a set of active tracking contexts comprising a respective tracking context for each currently tracked object. Figure 2 Two tracking contexts 28a-b are indicated schematically, but the number of tracking contexts 28a-b will depend on the number of currently tracked objects. The tracking contexts can comprise a set of data fields as shown, e.g. one or more of an object ID, a set of object features representing the object, a Re-Identification (ReID) threshold and a track segment. The tracking contexts can optionally further comprise a timer field as will be described in further detail below. The data fields of the tracking contexts can be updated by the processing device 24 during the tracking process as will be described further below.

[0077] The object tracking method can be implemented in hardware and software. In a software implementation, the processing device 24 can be implemented in the form of one or more processors, such as one or more central processing units and / or graphics processing units, which are associated with computer program code instructions stored on a (non-transitory) computer-readable medium (e.g. a non-volatile memory) to cause the processing device 24 to perform the steps of the object tracking method. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, and optical discs, etc. In a hardware implementation, the processing device 24 can instead be implemented by dedicated circuitry configured to carry out the steps of the object tracking method. The circuitry can be in the form of one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs) or one or more field-programmable gate arrays (FPGAs). It will be appreciated that a combination of hardware and software implementation is also possible, meaning that some method steps can be implemented in dedicated circuitry, while other steps can be implemented in software.

[0078] Figure 3 A flowchart illustrating an example implementation of the object tracking method is shown.

[0079] At step S1, the processing device 24 determines respective locations in the scene 1 at which objects can leave at least one sink (e.g. one or more of the sinks 4a-c) of the scene 1, and respective locations at which objects can enter at least one source (e.g. one or more of the sources 2a-c) of the scene 1. The locations of the sources 2a-c and sinks 4a-c can be determined manually by an operator, e.g. by indicating via a graphical user input interface where the sources 2a-c and sinks 4a-c are in the scene 1. However, the processing device 24 can also implement an automatic determination of the respective locations of one or more sources and sinks, e.g. using image recognition algorithms. As a non-limiting example, the image recognition algorithms can be configured or trained to identify road lanes in the scene 1 and entry and exit points of the road lanes at the periphery of the scene 1.

[0080] At step S2, the processing device 24 tracks one or more objects detected in the scene 1 using a re-identification algorithm. For example, when an object such as the car 10a enters the scene 1 at a source such as the source 2c, the processing device 24 attempts to determine whether the detected object (e.g. the car 10a) corresponds to an object that has already been tracked. Accordingly, the processing device 24 extracts a set of object features from a video frame in which the object (e.g. the car 10a) is detected at the source 2c, and compares the set of object features to respective sets of object features associated with one or more currently tracked objects (e.g. the processing device 24 maintains a respective tracking context 28a-b for each object).

[0081] The processing device 24 can determine a distance (e.g. Euclidean distance or other distance suitable for comparing sets of object features, possibly multi-dimensional) between the set of object features of the newly detected object and the set of object features of the currently tracked object to generate a match score. The extracted object features can generally include visual features as well as latent (hidden) features. Non-limiting examples of object features include feature vectors extracted by a convolutional neural network (CNN), color histograms, and computer vision feature descriptors such as histogram of oriented gradients (HOG) or speeded up robust features (SURF). The processing device 24 can compare the match score to a re-identification threshold (e.g. stored in the tracking context of the respective currently tracked object) associated with the currently tracked object. The match score can be defined to increase (e.g. monotonically, generally strictly monotonically) with increasing similarity (e.g. decreasing distance) between the compared sets of object features, wherein the newly detected object can be re-identified as the currently tracked object in response to the match score reaching or exceeding the re-identification threshold. Alternatively, the match score can be defined to decrease (e.g. monotonically, generally strictly monotonically) with increasing similarity (e.g. decreasing distance) between the compared sets of object features, wherein the newly detected object can be re-identified as the currently tracked object in response to the match score reaching or falling below the re-identification threshold. In either case, upon passing the threshold test, the processing device 24 can continue to track the object (e.g. the car 10a) as the currently tracked object in subsequent frames, e.g. recording successive positions of the object in a trajectory segment of the respective tracking context (e.g. the tracking context 28a).

[0082] The car 10b is an example of an object (hereinafter interchangeably referred to as "first object") that has been tracked by the processing device 24 in a sequence of previous video frames using a re-identification algorithm at the time instant shown. Figure 1 The processing device 24 can therefore maintain a tracking context 28b for the first object 10b in the memory 26 at the time instant shown (i.e. the tracking context 28b is maintained in the set of active tracking contexts 28a-b). For example, the first object 10b has entered the scene, e.g. at the source 2b or the source 2c. As shown, the first object 10b is about to leave the scene 1 at the sink 4a. Accordingly, at step S3, the processing device 24 detects that the first object 10b has left the scene at the sink 4a in a video frame after the frame shown. Figure 1 The processing device 24 can therefore maintain a tracking context 28b for the first object 10b in the memory 26 at the time instant shown (i.e. the tracking context 28b is maintained in the set of active tracking contexts 28a-b). For example, the first object 10b has entered the scene, e.g. at the source 2b or the source 2c. As shown, the first object 10b is about to leave the scene 1 at the sink 4a. Accordingly, at step S3, the processing device 24 detects that the first object 10b has left the scene at the sink 4a in a video frame after the frame shown. Figure 1 The processing device 24 can therefore maintain a tracking context 28b for the first object 10b in the memory 26 at the time instant shown (i.e. the tracking context 28b is maintained in the set of active tracking contexts 28a-b). For example, the first object 10b has entered the scene, e.g. at the source 2b or the source 2c. As shown, the first object 10b is about to leave the scene 1 at the sink 4a. Accordingly, at step S3, the processing device 24 detects that the first object 10b has left the scene at the sink 4a in a video frame after the frame shown. Figure 1 The processing device 24 can therefore maintain a tracking context 28b for the first object 10b in the memory 26 at the time instant shown (i.e. the tracking context 28b is maintained in the set of active tracking contexts 28a-b). For example, the first object 10b has entered the scene, e.g. at the source 2b or the source 2c. As shown, the first object 10b is about to leave the scene 1 at the sink 4a. Accordingly, at step S3, the processing device 24 detects that the first object 10b has left the scene at the sink 4a in a video frame after the frame shown.

[0083] At step S4, the processing device 24 adjusts the re-identification threshold associated with the first object 10b (e.g. the ReID threshold of the tracking context 28b) in response to detecting that the first object 10b has left the scene at the sink. Prior to the adjustment, the re-identification threshold can be set to a default re-identification threshold. If the match score is defined to increase with increasing similarity, the re-identification threshold can be increased by applying an additive offset to the default value of the re-identification threshold, or by scaling the default value of the re-identification threshold by a factor larger than 1. If the match score is defined to decrease with increasing similarity, the re-identification threshold can be decreased by applying a subtraction (negative) offset to the default value of the re-identification threshold, or by scaling the default value of the re-identification threshold by a factor smaller than 1.

[0084] The default value of the re-identification threshold and the amount of re-identification threshold adjustment each represent a design parameter, the value of which can be determined from prior knowledge of the type of tracked objects, the scene, the locations of the source and the sink, the allowable deviation required for reliable re-identification of the tracked objects, and the acceptable risk of re-identifying a similar but different object as the object that has left the scene at the sink, etc. The match score used by the re-identification algorithm can typically be determined as a normalized value. Thus, the match score and the re-identification threshold can be in the range [0, 1]. If the match score is defined to increase when the object features are more similar, the default value of the re-identification threshold can be in the range 0.6-0.7 as a non-limiting example. If the match score is defined to decrease when the object features are more similar, the default value of the re-identification threshold can be in the range 0.3-0.4. In either case, the re-identification threshold can be adjusted (increased or decreased) by 10-30% for example. In either case, the re-identification threshold is thus adjusted such that the probability that the re-identification algorithm will re-identify a second object entering the scene at the source after the first object has left the scene at the sink as the first object is reduced.

[0085] After adjusting the re-identification threshold associated with the first object 10b, at step S5, the processing device 24 can detect a second object entering the scene 1 at the source. The second object can be detected at any one of the sources 2a-c of the scene (e.g. at the source 2a adjacent to the sink 4a).

[0086] In response to detecting the second object at the source, the processing device 24 attempts to re-identify the second object as the first object 10b at step S6 using the re-identification algorithm and the raised re-identification threshold. As described above, the processing device 24 can compare the second object feature set of the second object to the first object feature set of the first object 10b to generate a match score. The processing device 24 can perform a threshold test that includes comparing the match score to the adjusted re-identification threshold. In response to the match score passing the threshold test (e.g., the match score meets or exceeds the raised re-identification threshold in the case of a raised re-identification threshold, or the match score meets or falls below the lowered re-identification threshold in the case of a lowered re-identification threshold), the second object is re-identified as the first object 10b, where the method proceeds along the“yes” branch of FIG. 6. Figure 3 In response to the match score failing the threshold test (e.g., the match score is less than the raised re-identification threshold in the case of a raised re-identification threshold, or the match score exceeds the lowered re-identification threshold in the case of a lowered re-identification threshold), the second object is determined to be different from the first object 10b, where the method proceeds along the“no” branch of FIG. 6. Figure 3

[0087] If the method continues to the“yes” branch, after re-identifying the second object as the first object 10b, the processing device 24 restores the adjusted re-identification threshold to the value of the pre-adjustment re-identification threshold, e.g., the default value, at step S7. The re-identification threshold can be restored by subtracting the additive offset, adding the subtractive offset, or by rescaling the raised re-identification threshold by the inverse of the scaling factor described above.

[0088] Subsequently, at step S8, the processing device 24 can continue to track the second object as the first object 10b using the restored re-identification threshold.

[0089] If the method proceeds to the“no” branch, the processing device 24 continues by tracking the second object as a different object from the first object 10b at step S9. Assuming the processing device 24 fails to re-identify the second object as any of the currently tracked objects, the second object can be tracked as a new object, e.g., a new tracking context associated with the second object is created and added to the set of active tracking contexts, as described with reference to the entry of the object 10a at the source 2c in FIG. 1. Figure 1

[0090] ​​Therefore, the method described above can reduce the risk of unintentionally continuing to track a second object as the tracked object 10b by adjusting the re-identification threshold. This second object may correspond to a different physical object than the tracked object 10b, thus requiring a greater visual similarity between the second object and the first object 10b for successful re-identification. However, if the second object and the first object 10b are similar enough to generate a matching score through a threshold test utilizing the adjusted re-identification threshold (in which case the second object is more likely to correspond to the same physical object as the first object 10b), the second object may still be re-identified as the first object 10b and tracked accordingly.

[0091] To avoid unnecessarily re-identifying an object entering scene 1 at the source as the first object 10b, a further condition for re-identifying the second object as the second object 10b can be applied: the second object enters scene 1 at the source within a predetermined time from the departure of the first object 10b from the sink. If this additional condition is met, the method can continue according to the "yes" branch. If this additional condition is not met, the method can continue according to the "no" branch. For example, the processing device 24 can maintain a timer for each tracking context 28a-b. Figure 2 (The "Timer" field in the text). In response to detecting that the first object 10b has left scene 1 at sink 4a, the processing device 24 can begin incrementing the value of the timer. If, when the timer reaches a predetermined time limit, a second object that has not entered the scene is re-identified as the first object 10b, the tracking of the first object 10b can be terminated, wherein the tracking context 28b of the first object 10b can be deleted from the active tracking context set. If the first object 10b later re-enters scene 1, it can therefore be detected, identified, and tracked as a new object, for example, using a new tracking context. The value of the predetermined time can be based on scene knowledge. For example, the predetermined time can be set according to the type of tracking application, the specific type and location of the source and sink in the scene, etc. The predetermined time can also be set according to the number of tracking contexts that can be maintained by the available computing resources of the object tracking system 20.

[0092] Figure 4 yes Figure 3 The flowchart of the optional extension of the method can be combined with... Figure 3 The methods and steps in the code are executed in parallel.

[0093] As mentioned above, a scene can include a source and a sink. For this source and sink, the probability that an object that leaves the scene at the sink will re-enter the scene at the source shortly thereafter is essentially zero, or at least too small to inspire an attempt to re-identify. Figure 4 This method can solve the problem.

[0094] At step S10, the processing device 24 determines the location of a second sink at which the object leaves the scene. Referring to Figure 1 And relative to the source 2a, the second sink can for example be represented by sink 4b.

[0095] At step Sll, the processing device 24 tracks a "third object" moving in the scene 1 using a re-identification algorithm and a respective re-identification threshold associated with the third object. Referring to Figure 1 The third object can for example be represented by a car or truck 10c. As Figure 1 indicated, the third object 10c is about to leave the scene 1 at the second sink 4b. Accordingly, at step S12, the processing device 24 detects that the third object 10c has left the scene at the second sink 4b in a frame subsequent to the frame Figure 1 indicated.

[0096] In response to detecting that the third object 10c has left the scene 1 at the second sink 4b, the processing device 24 terminates tracking of the third object 10c at step S13, thereby preventing a fourth object entering the scene 1 at any one of the sources 2a-c from being re-identified as the third object. As previously mentioned, terminating tracking can include deleting the tracking context associated with the third object from the set of active tracking contexts maintained by the re-identification algorithm.

[0097] As mentioned above, the location of the sink 4b can be determined manually by an operator or automatically using an image recognition algorithm. The decision to determine the sink 4b as the "second sink" at which to terminate tracking can be based on scene knowledge. For example, in the example scene 1 knowledge, it can be considered that it is forbidden or even impossible to make a U-turn along the stretch of road following the sink 4b.

[0098] The person skilled in the art realizes that the present invention is in no way limited to the embodiments described above. On the contrary, many modifications and changes are possible within the scope of the appended claims.

Claims

1. A computer-implemented method of tracking an object in a video sequence of a scene, the method comprising: determining a location in the scene at which an object leaves a sink of the scene and a location at which an object enters a source of the scene; tracking a first object moving in the scene using a re-identification algorithm, wherein the first object is associated with a re-identification threshold of the re-identification algorithm; detecting that the first object has left the scene at the sink; in response to detecting that the first object has left the scene at the sink, adjusting the re-identification threshold associated with the first object to reduce a probability that the re-identification algorithm will re-identify a second object as the first object, the second object entering the scene at the source after the first object has left the scene at the sink; the method further comprising, after adjusting the re-identification threshold associated with the first object: detecting the entry of the second object at the source; and in response to detecting the entry of the second object at the source, attempting to re-identify the second object as the first object using the re-identification algorithm and the adjusted re-identification threshold, wherein attempting to re-identify the second object as the first object comprises comparing a second object feature set of the second object to a first object feature set of the first object to generate a match score, and performing a threshold test by comparing the match score to the adjusted re-identification threshold; in response to the match score passing the threshold test: re-identifying the second object as the first object, and subsequently restoring the re-identification threshold, and continuing to track the second object as the first object using the restored re-identification threshold; in response to the match score failing the threshold test: determining that the second object is different from the first object, and subsequently tracking the second object as an object different from the first object using the re-identification algorithm and a re-identification threshold associated with the second object; wherein the match score is defined such that the more similar the first object feature set and the second object feature set, the higher the match score, and wherein the re-identification threshold is adjusted by increasing the re-identification threshold, or wherein the match score is defined such that the more similar the first object feature set and the second object feature set, the lower the match score, and wherein the re-identification threshold is adjusted by decreasing the re-identification threshold.

2. The method of claim 1, wherein, tracking the first object comprises maintaining a tracking context of the first object, and wherein re-identifying the second object as the first object comprises associating the second object with the tracking context of the first object.

3. The method of claim 2, wherein, the entry of the second object at the source is detected within a predetermined time from detecting that the first object has left the scene at the sink.

4. The method of claim 1, wherein, Tracking the first object includes maintaining a tracking context for the first object, and the method further includes, in response to determining that more than a predetermined time has passed since detecting that the first object has exited the scene at the sink, removing the tracking context for the first object from a set of active tracking contexts maintained by the re-identification algorithm.

5. The method of claim 1, wherein, The method includes determining a location of a set of sources at which objects enter the scene, wherein the source is one of the set of sources, and wherein, after adjusting the re-identification threshold, the probability that the re-identification algorithm re-identifies an object entering the scene at any of the set of sources as the first object is reduced.

6. The method of claim 5, wherein, Entry of the second object is detected at any of the set of sources.

7. The method of claim 1, wherein, The sink is a first sink, and the method further includes determining a location of a second sink at which objects exit the scene, and the method further includes: tracking, using the re-identification algorithm, a third object moving in the scene, wherein the third object is associated with a respective re-identification threshold of the re-identification algorithm; detecting that the third object has exited the scene at the second sink; and in response to detecting that the third object has exited the scene at the second sink, terminating tracking of the third object, thereby preventing a fourth object entering the scene at the source from being re-identified as the third object.

8. The method of claim 7, wherein, Tracking the third object includes maintaining a tracking context for the third object, and wherein terminating tracking of the third object includes removing the tracking context for the third object from a set of active tracking contexts maintained by the re-identification algorithm.

9. The method of claim 1, wherein, The source and the sink are adjacent to or overlap each other.

10. An object tracking system comprising: at least one camera for capturing a video sequence of a scene; and a processing device configured to track objects in the video sequence according to the method of claim 1.

11. A computer program product comprising computer program code portions configured to perform the method of tracking objects in a video sequence of a scene according to claim 1 when executed by a processing device.

Images