Method for improving tracking

Abstract

To provide a method and a camera system for improving tracking of an object in a scene, using an overview monitoring camera.SOLUTION: A method comprises: tracking objects in a plurality of image frames depicting a scene to produce current object tracks; detecting object candidates in an image frame subsequent to the plurality of image frames depicting the scene; calculating, for each object candidate, an association measure which is indicative of a likelihood that the object candidate is associated with a current object track; correlating a view of an overview camera with a heat map of the scene; adjusting the association measure or an association threshold for object candidates which, according to the heat map, are located in areas in the scene having an elevated degree of occurrence of historically verified object tracks so as to increase their probability of being associated with a current object track; and associating each object candidate with a current object track if the association measure is above an association threshold.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a method for improving tracking and a camera system configured to perform said method.

Background Art

[0002] The functionality of tracking is a standard function in video surveillance applications. It is used for the obvious task of tracking objects within the monitored scene, but also as a means to provide robustness against functions such as people counting and line crossing detection in such scenes. As a result, the quality of the tracking method affects several functionalities of the surveillance system.

[0003] Tracking an object involves correlating several detections of the object to a single object track. This is generally performed by predicting where the currently tracked object will be observed next based on its current movement. If the next detection is close enough to the prediction, it is considered to be related to the same object. The actual method is slightly more complex than this, some of which will be detailed in the detailed description, but the basic methodology is included in the prior art. For example, the track may be lost due to occlusion by another object or confusion with another object.

[0004] With the advent of the possibility of extracting object features, i.e., being able to define the identity of an object through its identifying features, it is possible to correct broken tracks by using re-identification in appropriate situations. This basically corresponds to performing an automatic verification of the object identity and reconnecting two tracklets to a single track if they are related to the same object. Re-identification requires a reliable and appropriate image quality, and is thus more commonly used in camera systems with better resolution and / or the ability to zoom in on the object in order to achieve an appropriate pixel density.

[0005] The latter type of camera system has a significant advantage with respect to tracking objects, especially when it is also possible to pan and tilt to keep the object being tracked within the field of view. However, a related drawback is that while one object is being tracked, the rest of the scene may be without surveillance coverage. Therefore, a camera system aimed at providing both a continuous overview of the scene and the ability to zoom in on individual objects or locations may comprise both one or more overview cameras with a wide field of view and one or more PTZ cameras (pan-tilt-zoom cameras) to enable a detailed view. Briefly stated, it is very common for overview cameras to have a higher pixel density than PTZ cameras, but PTZ cameras are capable of acquiring an image with a higher pixel density of the object due to their zooming ability.

[0006] As a result, there is a need for improved methods and configurations for tracking, especially for tracking using overview cameras. The term "overview camera" can be defined by its limitations in a particular camera setting. This relates to cameras that are unable to acquire an image of sufficient quality to perform re-identification for a particular scene or for a part of a particular scene. A typical example is that the object is too far away for the camera to extract an image with sufficient pixel density. This can be rephrased as tracking where image quality can limit tracking performance, which applies to any camera at a given point in time. In a camera implementation, an overview camera typically corresponds to a fixed camera (fixed orientation and fixed zoom or no zoom) with a wide field of view aimed at providing an overview of the scene and situation awareness. Several overview cameras can be used in combination to cover an even larger field of view, and one or more PTZ cameras can be placed to provide a detailed view of a part of the scene being monitored. SUMMARY OF THE INVENTION

[0007] The object of the present invention is to provide an improved method for tracking objects in a scene, especially in situations where re-identification is not available. According to a first aspect, these and other objects are achieved fully or at least in part by the method according to claim 1. Another object is to provide a camera system configured to execute a method as detailed in the subsequent independent claims and their dependent claims, as described in the following detailed description.

[0008] Advantages of the present invention include, according to some embodiments thereof, improving tracking, particularly in tracking situations where it is difficult for the current tracking algorithm to follow an object being tracked from one position to another. According to claim 1, this is achieved by a method for improving the tracking of an object in a scene using an overhead monitoring camera. The method includes tracking an object in a plurality of image frames depicting the scene to generate a current object track. Part of the tracking is detecting object candidates in an image frame following the plurality of image frames depicting the scene, with the aim of finding which of these object candidates belong to the current track. To that end, an association measure is calculated for each object candidate, the association measure indicating the likelihood that the object candidate is associated with the current object track. In situations where the quality of the image data is low or other causes introduce uncertainty into the tracking, it is correlating the view of the overhead camera with a heatmap of the scene, the heatmap providing data indicating regions in the scene where the degree of occurrence of past object tracks has increased, and adjusting the association measure or association threshold for object candidates located in regions in the scene where the degree of occurrence of past object tracks has increased according to the heatmap so as to increase the probability of being associated with the current object track. This can be enhanced by introducing further steps including these steps being performed, the tracking algorithm being able to proceed to the step of associating each object candidate with the current object track if the association measure exceeds the association threshold. In this way, object candidates along the conventional path have a higher likelihood of being associated with the current track.

[0009] The view of the overhead camera is correlated with the heatmap, i.e., the position in the overhead camera can be converted to the position in the heatmap, so it is easily possible to extract values from the heatmap and apply them to the appropriate positions in the view of the overhead camera.

[0010] The heatmap can preferably be generated using a PTZ camera that tracks objects in a scene over time and remembers the tracks followed by the objects. By using a PTZ camera, improved tracking performance for the monitored scene is enabled, thereby increasing the amount of high-quality data of past tracks.

[0011] To further improve the quality of the heatmap data, the tracking is performed using intermittent or continuous re-identification to guarantee the verified tracks from each individual object being tracked.

[0012] As will be described in more detail below, the heatmap includes position measurements of the recorded object tracks, but may also include representations of the measurements rather than the actual measurement results. In this way, the position data of past tracks is readily available for queries. The heatmap may also include information regarding the typical speed of past tracks, object classes, or object speeds to enable higher accuracy during tracking, all data that may be used when filtering information from the heatmap prior to use.

[0013] In this way, the method may also include selecting heatmap data corresponding to the identified object class or the identified object speed. As an example, if a tracked object is identified as a vehicle, it may be beneficial to use only past data regarding vehicles rather than past data including, for example, how humans were moving within the scene. The object speed may be used as a filtering parameter for selecting the lower part of the heatmap.

[0014] A typical system for executing this method includes a PTZ camera and an overhead camera. For the present invention to operate properly, the view of the overhead camera is position - calibrated with the view of the PTZ camera. Since the PTZ camera is often used to zoom in on selected events within the field of view of the overhead camera, such a system enables the execution of the method according to any embodiment, and most such systems are position - calibrated anyway.

[0015] Apart from including information assembled using the PTZ camera, the heatmap may also include data from one or more overhead cameras. One advantage of this is that it can speed up the generation of the functional heatmap, and the more data there is, the better as long as the data is sufficiently robust. Another advantage is that it allows taking advantage of the opportunity to monitor how the actual association metric changes for the overhead camera with respect to the scene being monitored. In situations where the data from the overhead camera is not sufficiently robust for tracking, for example, verification from the PTZ camera can confirm that it is actually the same object as before. When in use, the recorded data can be used when the association metric or association threshold is adjusted.

[0016] The association metric can increase within the region of the heatmap containing the verified track, for example, the distance between the object detection and the path of the heatmap, or with the heatmap intensity, so as to increase the likelihood that the object detection is associated with the current track. This can be performed before the track is initiated, which can have the effect of increasing the likelihood of track placement, for example, when the object is located far away from the overhead camera. A similar effect can be achieved by lowering the detection threshold of the corresponding region. The associated region gets closer to the tracks verified in the past, and "close" can be quantified as "within the tolerance of the position error", which means that if there is an object candidate detection that may potentially be located along the track verified in the past, it should be more likely to be associated with the track. Also, in practice, such detections are likely to be actual objects rather than false detections.

[0017] The system that executes this method preferably includes a PTZ camera and an overhead camera, but the principle of the present invention can be partially achieved with only a PTZ camera or multiple PTZ cameras in that regard. The reason is that the PTZ camera can be used in a fully zoomed-in mode that enables the collection of heatmap data, or any appropriate degree of zoom, and the (fully) zoomed-out mode during normal operation. The reason for using the PTZ camera in the zoomed-out mode during normal operation is to enhance the situation awareness, which is an important feature of the overhead camera, because the field of view covered in the zoomed-out mode is larger.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0019] Figure 1 is a schematic diagram of a camera system 100 that can be used when implementing the present invention. The camera system 100 includes a single PTZ camera 102, i.e., a camera with pan, tilt, and zoom capabilities. With this capability, the camera can cover a wide area, zoom in on details, and follow objects moving over a long distance. The camera system 100 also includes one or more overhead cameras 104 that can have a fixed focal length that generally covers a wider field of view compared to the PTZ camera 102. Again, in contrast to the PTZ camera, the overhead camera 104 typically has a fixed orientation during use, i.e., lacks motor-assisted pan and tilt during operation. In the illustrated embodiment, there are four overhead cameras 104, which together can cover a 360-degree field of view of the scene. The overhead cameras 104 are used for the obvious purpose of covering a wide field of view and providing an overall scene understanding. In a dedicated system 100 of the type shown in Figure 1, the overhead cameras 104 are aligned and calibrated so that a single stitched overhead image can be formed by combining the images from each overhead camera. Also, the overhead cameras 104 are aligned with the PTZ camera 102 and vice versa, enabling the PTZ camera 102 to immediately move to image an area within the field of view of the overhead camera 104 and enabling the overhead camera to show the current field of view of the PTZ camera as a graphic overlay, such as a rectangle.

[0020] The camera system of Figure 1 can be a system available within the Axis Q-60-61 series, including all its capabilities, such as the Q6010-E or Q6100-E systems by the applicant. However, the camera system can also be any tailor-made combination of individual cameras.

[0021] The resolution for pixels per image may be greater for the overhead camera than for the PTZ camera, or vice versa, but in this camera system, the zoom capability of the PTZ camera is far superior with respect to the pixel density of the imaged object, particularly an object located further away, which means that when an object is imaged, a greater pixel density is dedicated to it.

[0022] Leaving this particular embodiment aside for a moment, a typical embodiment can comprise a PTZ camera 102 used when assembling a heat map and at least one overhead camera 104 (with one exception disclosed below) used during the actual tracking process. These cameras, the PTZ camera 102 and the (at least one) overhead camera 104 do not need to be combined in a single unit 100, but need to be calibrated such that it is possible to perform a position transformation between the position within the coordinate system within the overhead camera 104 and the PTZ camera 102, and vice versa. The at least one overhead camera 104 can be the PTZ camera 102 in a zoomed-out state, so in a very specific and perhaps not at all common embodiment, the PTZ camera 102 and the overhead camera 104 can be exactly the same. Although not considered the most preferred embodiment, the capabilities of different camera types are used at different times, which means that in the context of the present invention, a single camera can first perform the tasks of a PTZ camera and then zoom out to perform the tasks of an overhead camera. This is a rather unlikely embodiment but is emphasized at a later stage for the purpose of further clarification.

[0023] Figure 2 is a schematic diagram of a surveillance scene. A camera system 100, such as the type shown in Figure 1, is arranged at the upper left corner of the apartment building 106. Then, the overhead camera of the camera system 100 can provide an overhead view of all parts of the scene, or at least the parts not blocked by the building 106 itself. As a result, the PTZ camera can more selectively image the same part of the scene. The scene includes several roads and sidewalks 108, several vehicles 110, several pedestrians 112, cyclists 114, and several trees and bushes 116, all of which may be moving objects (or, in the case of plants and trees, swaying objects).

[0024] The PTZ camera 102 can follow individual objects 108-114 in the scene and increase the zoom level as the distance from the camera system 100 increases. The overhead camera 104 can be used for the same purpose, except that the zoom level cannot be increased (or changed), and the same image analysis can be utilized. Examples of image analysis include the following. - Motion detection, where the shifting pixel input within the area can be interpreted as motion. This analysis responds to moving objects including not only vehicles and individuals, but also swaying trees and moving shadows. For example, it is possible to identify specific types of motion even with this basic technique by filtering based on, for example, the size of the coherent area, the characteristics of the detected changes, etc.

[0025] - Object detection, where the characteristics of the object can be extracted so as to enable not only motion detection but also classification of the type of object, even regarding size or object class (human, car, bicycle, tree, etc.).

[0026] - Re-identification, where a feature vector representing the appearance of the object is extracted from the moving object so as to enable the provision of a unique identifier for the object.

[0027] Today, re-identification is typically used in the context of deep learning and neural networks and requires a high level of detail (resolution or pixel density) to be reliable. Currently, the computational cost of performing re-identification using neural networks is too high to perform live for each frame and object at a higher frame rate. Instead, re-identification is generally performed intermittently, at a lower frame rate and / or when triggered by the input, such as when a more basic tracker loses track. When performed on recorded material, such procedures can be easily performed by accessing a more powerful or dedicated CPU, so the computational cost is not as important a consideration.

[0028] In the case of an object tracking algorithm, motion detection and possible object detection are only the first part of the tracking. For individual scenes and individual points in time; there may be multiple detections. The second part is to filter all these detections in order to estimate whether the new detections are related to previous tracks. This can basically be done using an "association metric" that corresponds to the likelihood of whether a new detection belongs to a previous track. In traditional object tracking, the Kalman filter, which is considered to be the best linear unbiased estimator, is often used. The Kalman filter is an algorithm that makes predictions about where an object will appear in subsequent situations based on previous measurements. There is a lot of literature on the Kalman filter and tracking, but in the simplest case, it tracks an object over several frames and uses parameters such as "last known position", "direction", and "speed" (i.e., the "state" of the object) to make a prediction about where the object will appear, including the uncertainty involved in the prediction. For example, the prediction may include a probability density function indicating the likelihood that the object will appear in different locations, calculated using statistical considerations based on the input data. Then, a measurement of where the object is, i.e., a detection or observation, is made, and this measurement itself contains its own uncertainty. Then, the predicted and measured locations are used to determine an updated position of the object, etc. In the context of video surveillance, the measured position will be based on image processing, and there is an additional problem of knowing whether the detected object belongs to the tracked object. For example, there may be several object detections in the vicinity of the predicted position, so there is an additional problem of choosing which object detection to use. When verifying the prediction, the above-mentioned association metric is used, and the closer to the simply predicted position, the higher the association metric, i.e., the higher the likelihood that the detection is related to the previous track. For example, if the association metric exceeds an association threshold, the detected object may be associated with the previous object track, and when there are several detections to consider, the one with the highest association metric is selected.Since other parameters of detection can also be taken into account, the term "association metric" is used in the present invention. In this way, the possibility of maintaining the tracking of exactly the same object increases, in contrast to methods where objects appearing in the vicinity have an equal chance of being associated with the current track.

[0029] In a simple version, the association metric can be purely based on the distance from the predicted position, for example, by using the probability density function described above. This means that if detection candidates are equidistant from the predicted position, they have the same association metric and thus the same probability of belonging to the current track. However, the association metric can be more complex and can include parameters such as size and speed to further increase the likelihood of associating a correct detection with the current track. This can also include or be used in combination with a re-identification part, which means that the appearance of the object is taken into account, for example, by including the feature vector described above. The weight given to this re-identification part can vary. For example, if several object detections are equally likely to belong to a previous track based on a standard association metric, a greater weight can be given. The re-identification part can also be used to reconnect a new tracklet or detection to an existing track in a situation where the track has been lost for a certain period. In this context, a "tracklet" is a part of a track that has not yet been associated with another track.

[0030] The above list of image analysis is arranged more or less in the order of the required details (i.e., the level of detail / resolution of the observed object), which is converted into the distance from the camera for each individual camera. The PTZ camera 102 can adjust the zoom and thus can utilize re-identification further away from the camera. This is done at the expense of a reduced field of view, and in situations where there are multiple potentially interesting objects, it is necessary to prioritize one or a few objects or shift the orientation of the PTZ camera 102 between objects. On the other hand, the overhead camera 104, without available zoom options, has tracking robustness that is likely to decrease more rapidly with the distance from the camera system 100 while being able to follow multiple objects simultaneously.

[0031] A direct attempt at a solution to improve tracking could be to adjust the association measure or association threshold with respect to distance for the overhead camera. This essentially introduces more leeway for associating new detections with existing tracks. This approach may be suitable for some use cases but has the potential to increase the occurrence of incorrect tracking associations to the extent that it affects the reliability of tracking.

[0032] The idea of the present invention is to provide an improvement to the known art in situations such as those exemplified above. For that purpose, an embodiment of the present invention can start from the data assembly period. During that period, the PTZ camera is used to assemble data regarding the movement within the scene, particularly the movement associated with moving objects, the movement of which can form tracks within the scene as it is information likely to be relevant to the surveillance scenario. This is carried out by tracking the objects, and the data includes the object position and effectively the object position over time (which is also an indication of speed). The data can also include the object class (bus, bike, car, individual, etc.) and even the identity of the object. Identity does not necessarily mean that it is exactly known who the individual is, etc., but only means that it is verified that it is always the same individual and that the individual can be separated from other individuals. What is said about "individual" also applies to other object types with respect to identity, an example being that a particular track can be associated with a single bike or a single car, etc.

[0033] A PTZ camera assembles data by tracking moving objects. This can be a fully automated tracking that is performed while the PTZ camera is not occupied with a user-defined task, or it can be data collected during a user-defined task. The user-defined task can be an operator who manually moves the PTZ camera (using a graphic or physical user interface) to follow an object. During such manual tracking, the tracking algorithm may still be executed to extract the object position over time. The data obtained for a tracked object is at least the object position, which can be stored in a common position reference system so that it can be immediately retrieved by any camera of the system (or actually by the control system of the camera system). Each track may be stored as a multi-sampling point in an appropriate coordinate system, preferably the common coordinate system of any camera of the system (this simplifies later use). This can also be stored as a multi-trajectory curve, or any other appropriate format. Since several tracks follow the same or essentially the same path, the weight ("temperature") for that path can be increased, or simply as an additional functionality, a path that has moved basically twice is given a value of "2", while a path that has moved 600 times is given a value of "600". Such considerations are not closely related to the present invention, but rather to how data is assembled in a heat map.

[0034] Over time, the assembly of data results in at least one heatmap across the monitored area. The heatmap can be separated by object category, time, etc., and thus is the use of "at least one" heatmap. Of course, this may be a single multi-dimensional heatmap where various parameters are represented in different layers of the heatmap, but if different parameters are stored in different heatmaps, different parts of vectors, different cells of matrices, etc., it is equivalent in terms of effect. The heatmap corresponds to a representation of a scene where the degree of movement is quantified and presented by the intensity within the image of the heatmap. More specifically, it is not movement itself, but rather the generation of verified movement tracks over time, and thus, if there is no trackable movement, it is not registered in the heatmap. This has the advantage of eliminating most of the effects of false movement, movement of vegetation back and forth, and movement detection due to noise. The scale of the heatmap can be relative or absolute with respect to the number of verified object tracks. At the start, this can be a linear dependence such that an area with 10 verified object tracks is twice as "hot" or rises compared to an area with 5 verified object tracks. Over time, there may be an effect of saturation, whereby smaller differences in object track frequency cannot be distinguished. Furthermore, when using the heatmap data, one can choose to simply distinguish between an area with verified movement tracks and an area without movement tracks. In such a situation, there may be a threshold function, whereby a certain number of verified movement tracks are required for an area to be classified as an area "with verified movement tracks". These verified movement tracks are called "paths". By introducing a distance threshold similar to the effect of segmentation, it is possible to reduce the dynamics of the heatmap, where verified movement tracks running parallel to each other within the threshold distance can be represented by exactly the same path. Such an approach can be beneficial for paths that do not move much.In the case of a path with a high frequency of verified movement tracks, the tracks are most likely to be distributed according to a normal distribution along the path.

[0035] Next, the resulting heatmap can be stored and used by an overhead camera as described in relation to two scenarios. The first is that the object moves away from the overhead camera to a distance, and the second is that the object approaches the overhead camera from a distance. From the perspective of the present invention, the two scenarios are more or less the same, but within the field of camera installation, the two scenarios are extreme because in the first, it starts with the maximum image quality and in the second, it starts with the minimum image quality.

[0036] In the first example, the overhead camera can track the object relatively easily. The resolution is at least good enough at the start to perform re-identification if necessary. As the object moves away, the possibility of performing re-identification eventually disappears, and the tracking has to rely on other parameters such as conventional Kalman filter tracking or similar tracking of that type. At this stage, there are at least two uses of the heatmap. One is to adjust the association metric along the detected path. As an example, if the standard association metric for object detection is calculated at 75%, this can be adjusted upwards to 77% or 85% etc. depending on its distance from the path. This makes it easier to maintain the trajectory moving along the path. There is a risk that false object detections along the path are included in the track, but the scene-specific statistics indicate that the object is along the path where it moves most frequently, so the advantages should outweigh this risk. This association metric should be used in combination with other tracking techniques, which means that, for example, a Kalman filter can still be used to predict the future position of the tracked object, but the possibility of finding the object along the path increases. This has the effect of making the tracking more robust and reducing the possibility of false movements outside the path that would be involved. Such false movements can, for example, arise from trees or bushes moving in the wind, but since these have never caused tracking by the PTZ camera, they do not affect the heatmap.

[0037] Another approach that can be used in combination with the former is to adjust the association threshold, which in practice is to lower the threshold that the association metric should exceed to be associated with an existing track.

[0038] Numbers given in relation to a scale or threshold only mean to indicate a change as a qualitative scale. All trackers have a parameter indicating the confidence regarding the association of objects. The phrase "adjusted association scale" means that this parameter is changed in a direction that makes the association of the track more likely. As described above, for example, a general change of the threshold as a function of the distance from an overhead camera can have the effect of increasing the number of motion detections in an undesirable way. On the other hand, the spatial accuracy with which this is carried out in embodiments of the present invention, supported by a heatmap, enables an improved tracking ability.

[0039] Another option could be to finely tune a Kalman filter (or any other tracker filter used) according to a heatmap. This fine tuning could mean that the estimated value of the association scale for associating object detections changes asymmetrically with the distance from the predicted position. More specifically, there may be a steeper attenuation of measurements in a direction orthogonal to the path than in a direction that coincides with the path. This makes it more likely to find an object along the path that is expected while also acting as a filter regarding false motion detections to both sides of the path.

[0040] These two options can be combined, and the latter example also includes adjusting the detection threshold within the reshaped region of uncertainty.

[0041] The above-described method can be executed based on only the heatmap and the normal tracking algorithm. However, in one embodiment of the present invention, the adjustment of the detection threshold is executed based on additional data collected during the training period but collected by an overhead camera rather than a PTZ camera. Such additional data is the actual tracking data from the overhead camera and can benefit from enhanced information from the PTZ camera that verifies the identity of the object. This additional data can then be used to adjust the threshold to a level suitable for a particular region of the monitored scene. It should be noted that in this context, the "training period" does not necessarily have a defined endpoint. To enable the use of the heatmap, it needs to contain sufficient information for the purposes of the present invention. The term "sufficient" may be considered ambiguous, but the truth of the matter is that there is no clear cutoff when there is sufficient information. For example, for the present invention to have a positive impact, it may be sufficient that the object is tracked along some roads and sidewalks. Still, for obvious reasons, a larger statistical basis is better. Further, the accumulation of information in the heatmap does not even need to have a separate endpoint, and any tracking executed during the life of the camera system can be added to the heatmap.

[0042] Returning to the hand - held embodiment, a more detailed example of the "further data" described in the previous paragraph is shown. By utilizing the verification performed by the PTZ camera, it is possible to authenticate an object tracked by the overhead camera, particularly in situations where the tracking algorithm used by the overhead camera fails. In the setting of this embodiment, which is rather an alternative when assembling the data for the heatmap, the PTZ camera and the overhead camera track the same object. During the procedure of tracking by the overhead camera, due to changes such as distance, occlusion, and shadow, the tracking reliability of the overhead camera (i.e., that of the tracking algorithm being executed on the image data acquired by the overhead camera) can change. These fluctuations can be recorded, especially when the association metric may fall below the association threshold of the overhead camera's tracker. The changing association metric can be added as one piece of data stored in the heatmap or a related lookup table. By using this data, it becomes possible to adjust either the tracking parameter of the overhead camera, i.e., the association metric or the association threshold as a function of the position along the path in a more controlled manner. Similar to the case of other data related to the heatmap, this data can vary along with the object type, time, etc. In any case, this embodiment enables the tailor - made adjustment of the association threshold along the area of the heatmap. Regarding the fluctuations over time, they are not experiences that seem significant. In fact, there is a difference in performance between daylight conditions and the pitch - black conditions at night, but in modern society, it is rare for it to become completely dark enough to affect modern digital surveillance cameras.

[0043] With this in mind, the scenario shifts to an object approaching the camera system from afar. Essentially, the same approach can be used, but when the object appears in the distance, (in contrast to the previous scenario where the starting point was an optimal tracking situation with high-resolution image data of an object close to the camera system) by default, it starts with uncertain object detection. Therefore, object detection closer to the path may be more likely to be the starting point of the track than object detection far from the track. This can be exploited by the tracking algorithm in terms of having a higher tendency to start tracking an object on or close to the path. If the track is not verified by several consecutive related object detections, the started track can be cancelled. The tracking algorithm can be made to be "more inclined" by adjusting the association measure or the association threshold, or a combination of both, as previously disclosed.

[0044] Also, when an object is detected and a track is started, parameters such as class, size, and speed can be utilized to access the correct layer of the heatmap (or the correct heatmap, etc.) to further refine the threshold. The refinement corresponds to filtering data from the heatmap to obtain data more relevant to the currently tracked object. If the operator wants to search for a specific object type (such as an object class, an object moving at / under / above a specific speed), of course, the most appropriate layer can be used from the beginning. This can be applied to any of the above scenarios.

[0045] What is common to all of these techniques is that the adjustment does not always need to be applied to all positions in the scene. The application can be triggered by necessities such as, for example, the loss of the object being tracked, a change in visibility, etc. Another commonality is that since objects found outside the path may also be relevant in the surveillance scenario, the adjustment means supporting or enhancing the tracking along the path rather than canceling the detection outside the path.

[0046] FIG. 3 is a perspective view of a simplified scene essentially comprising several roads 108, a junction 118 within the roads, several vehicles 110, and surrounding fields 120.

[0047] Figure 4 is a virtual heatmap of the scene of FIG. 3. The dot density indicates the level of track generation (not the generation of movement, but particularly track generation). As can be seen, some regions within the scene have a higher level of track generation than other regions, that is, in some regions, the degree of track generation is increasing. There are mainly tracks along the road. For illustrative purposes, the generation or frequency of tracks gradually decreases towards the sides of the road, represented by a decreasing intensity (or dot density in the captured example). In an actual scenario, since it is less likely to find vehicles traveling along the lane but not on the lane, the decrease is likely to be steeper. Such a distribution can clearly differ from scene to scene or within a scene. Some movements are detected in the fields, such as animals and agricultural vehicles, but in the most likely scenario, this movement is not as important compared to the movement along the road. Although not essential, it is preferable to verify the tracks assembled in the heatmap. In this context, "verified" can indicate movement by an object that may be relevant from a monitoring perspective. Also, this can be expressed as the object following the track and being verified to be the same object throughout the track. Then, it is verified to the extent possible using, most preferably, the tracking algorithm of a PTZ camera, which is used to assemble the heatmap. The significant optical performance of the PTZ camera functions as the first authentication for verification, and more specific verification steps may also be included. An example of verification can include re-identification, which can be performed continuously, intermittently, or in situations necessary to verify the track. This may follow a temporary loss of the track, where older tracks are associated with newer tracks, and it is necessary to ensure that they are actually related to different instances of the same object before combining those tracks into one track. Re-identification can be performed using object feature vectors and neural networks, but other means for re-identification are also possible.In an example where data resulting from manual tracking by an operator is used, it can be said that verification is achieved by a human, but in most practical cases, even in that case, it is still an algorithm that operates in the background. In either method, as a result, only data regarding the track formed by the moving object is assembled into the heatmap or at least made available for later tracking. In contrast, there may be a lot of movement within the field from crops that generate a "wave" of movement that sways back and forth and even moves along the entire field. However, such movement is not tracked by the PTZ camera, and it is easy to prevent such movement from being confused with the movement of related objects by using re-identification and tracking. The heatmap may be used as is, but it is also possible to set either an absolute or dynamic threshold (e.g., in relation to the surrounding area) so that the path is more clearly defined. In a simplified heatmap, a value of 1 can be given to the center of such a path while a value of 0 is given to all other areas, but in other embodiments, more of the dynamic range of the heatmap can be used so that better resolution can be obtained in the adjustment or so that priorities can be set between paths with different intensities (i.e., having different object track frequencies) as required by the situation.

[0048] Here, with reference to FIGS. 5 and 6, a method for adjusting an association measure or an association threshold during tracking using the heat map of FIG. 4 will be described. In FIG. 5, how an object is tracked when it moves from right to left within an image is shown. The black circles 124 indicate the previous positions of the object. That is, each black circle 124 shows the state of the object, basically the position, over time, and the object moves from right to left (this type of data is assembled into a heat map). The last verified position is in front of a road bifurcation point and is the black circle 124'. Next, the tracker predicts the successive positions. Assuming that the object is moving in the same direction at the same speed, the tracker predicts the next position as indicated by the black diamond 126. The tracker then investigates whether there is a detection nearby, and (in a general situation) the closer it is, the better. In a given example, two candidates in the form of ring shapes 128 and 130, each at an equal distance d from the predicted position, are detected. In the prior art method, these two detections belong to an existing track, that is, they have an equal likelihood of being associated with the prediction and thus with the existing track. At this stage, the information from the heat map enters the equation, and an example of how this can be done is presented with reference to FIG. 6. FIG. 6 shows several curves 132, 134 that can be used to estimate the likelihood of a detection being associated with a prediction based on the distance d between the prediction and the detection. In a simplified case, the likelihood can correspond to an association measure in that the higher the value, the more appropriate it is than something with a lower association.

[0049] Here, the use of the present invention according to an embodiment of the present invention is to select a probability density function based on the position of the detection related to the heat map. The upper detection 130 is not in or near the region where the occurrence has increased, for example, based on the distance from the route (illustrated by the road 108) or based on the intensity of the heat map at that position. Therefore, it is considered that there is no effect of any heat map. Based on this, the original or slightly modified probability density function 132 can be used. The probability density function 132 may correspond to the predicted probability density from the tracker. When the distance d is input, in this virtual case, a score (or likelihood, or actually related scale) of about 0.30 (arbitrary unit) is generated. However, in the case of other detection candidates 128, the distance between the detection and the route is small, and as a result, the intensity of the heat map increases at the position of the detection. Therefore, the adjusted probability distribution 134 is used, and as a result, the distance from the prediction is the same, but the likelihood increases to a score of 0.70. This means that among the detection candidates, the detection candidates along the route have the highest likelihood and the highest association scale, and are therefore associated with the track.

[0050] This functionality is also useful in a situation where there is only one detection candidate 128 while the detection 130 in FIG. 5 does not exist. Based on the same example, an unexpected shift in orientation for the tracker may result in the association scale being too low for the only detection candidate 128, and its score not reaching the threshold, causing the tracking to be aborted. However, using the present invention, the association scale may be adjusted by the distance from the route, and as a result, the association threshold is met and the tracking continues. Instead of adjusting the association scale, the association threshold can be adjusted so that it is easier for the detection candidate along the heat map route to reach the threshold. The method used may vary depending on the type of tracker used.

[0051] In particular, even if the heatmap is binary (path or no path), the effect on the association measure and / or the association threshold can be gradual. This can be expressed as a direct or more complex function of the distance from the detection to the path, taking into account matters such as the inherent uncertainty when determining the location of the detection, for example.

[0052] In this context, throughout this description, it may be noted that the term "association measure" is used to describe a parameter that determines the likelihood that a detection is associated with an existing track. An equivalent parameter may also be referred to as an "association cost", in which case a lower cost corresponds to a higher likelihood, and terms such as "association probability" or simply "probability", or some other term, is used in the tracking model in question. Regardless of the term used, the present invention can be applied to enhance tracking, and regardless of the term used, this term can be converted to an "association measure".

[0053] The method described is a simple and step - by - step way to implement the present invention, as illustrated in the flowchart of FIG. 7. First, a heatmap is assembled at step 136. This is performed for the actual camera installation, i.e., with the camera system arranged to overlook the intended field of view. Once the heatmap is present, it can be used in the normal operation of the camera system which starts with objects being tracked using the cameras of the camera system at step 138. In the tracking process, at step 140, object candidates are detected and an association metric is calculated for the object candidates. This is shown as a separate step 142, but in practice, the tracker's evaluation and calculation may be intertwined in a more complex way. Following this step, at steps 144 and 146, the association metric or the association threshold is adjusted respectively. In other embodiments, both of these means are taken, i.e., both the association metric and the association threshold are adjusted. In the final step 148, additional information and adjustment values are used as input when associating object candidates with existing tracks. The present invention can also be implemented in other ways as described above. An example already mentioned is that a multi - dimensional probability distribution is generated based on the input from the heatmap. Depending on the specific tracker used, other implementations of the present invention may also be more suitable than others. Such a determination is considered to be within the capabilities of those skilled in the art based on the description of the present invention. Further, it should be emphasized again that depending on the metric or threshold used, the present invention can be applied by adjusting the metric or threshold up or down according to the metric / threshold. Therefore, unless related to a specific embodiment, referring to the direction for adjustment in the following claims has no meaning. Therefore, when the desired effect of the adjustment is described in the claims, it should be read in this context and should not be confused with the effort to define the present invention by the result to be achieved.

Claims

**Claim 1** A method for improving the tracking of objects in a scene using an overhead monitoring camera, comprising: tracking objects in a plurality of image frames depicting the scene to generate a current object track; detecting object candidates in an image frame following the plurality of image frames depicting the scene; calculating an association metric for each object candidate that indicates the likelihood that the object candidate is associated with the current object track; and the method further comprises: correlating the view of the overhead camera with a heatmap of the scene, wherein the heatmap of the scene provides data indicating regions in the scene where the degree of occurrence of past object tracks has increased; adjusting the association metric or association threshold for object candidates located within regions in the scene where the degree of occurrence of past object tracks has increased, according to the heatmap, so as to increase the probability of association with the current object track; when the association metric exceeds an association threshold, associating each object candidate with the current object track. A method further comprising the above steps. **Claim 2** The method according to claim 1, wherein the heatmap is generated using a PTZ camera that tracks objects in the scene over time and stores the tracks followed by the objects. **Claim 3** The method according to claim 1, wherein the heatmap is generated using a PTZ camera that tracks objects in the scene over time, and the tracking is performed using intermittent or continuous re-identification to ensure a verified track from each individual object being tracked. **Claim 4** The method according to claim 1, wherein the heatmap includes position measurements of recorded object tracks. **Claim 5** The method according to claim 1, wherein the heatmap includes velocity information about recorded object tracks. **Claim 6** The method according to claim 1, wherein the heatmap includes the object class or object speed of recorded object tracks to enable filtering for object class or object speed. **Claim 7** The method according to claim 6, further comprising selecting heatmap data corresponding to the identified object class or the identified object speed.

8. The method according to claim 7, wherein tracking the current object includes identifying the object class to which the tracked object belongs or the object speed of the object.

9. The method according to claim 1, which is executed by a system comprising a PTZ camera and an overhead camera, wherein the view of the overhead camera is position-calibrated with the view of the PTZ camera.

10. During the assembly of the heatmap, tracking is also performed using image data from the overhead camera, and the association metric for the tracking performed by the overhead camera is monitored for the tracks verified by the PTZ camera. The method according to claim 2.

11. The method according to claim 1, wherein the association threshold is lowered within the region of the heatmap containing the verified track.

12. The method according to claim 1, wherein the association metric increases, for example, the distance between the object detection and the path of the heatmap or the heatmap intensity within the region of the heatmap containing the verified track so as to increase the likelihood that the object detection is associated with the current track.

13. The method according to claim 1, wherein the functionality of the overhead camera is provided by a PTZ camera in a zoom-out mode.

14. A camera system comprising a PTZ camera and an overhead camera configured to execute the method according to claim 1.

Images