Method for improving tracking

Abstract

The present application relates to a method for improving tracking, in particular to a method for improving tracking of an object in a scene using a panoramic surveillance camera, comprising: tracking an object in a plurality of image frames depicting the scene to produce a current object trajectory, detecting an object candidate in an image frame following the plurality of image frames, and calculating, for each object candidate, an association metric indicating a likelihood that the object candidate is associated with the current object trajectory. The method further includes associating a view of the panoramic camera with a heat map of the scene, the heat map providing data indicating a region in the scene that the occurrence of the historically verified object trajectory has a higher degree, and adjusting, according to the heat map, a degree of association or a correlation threshold of object candidates located in a region in which the occurrence of the historically verified object trajectory in the scene has a higher degree to increase their probability of association with the current object trajectory, and if the degree of association is higher than the correlation threshold, determining whether the object candidates are associated with the current object trajectory. Each object candidate is associated with a current object trajectory.

Description

Technical Field

[0001] The present invention relates to a method for improving tracking and a camera system configured to perform the method. Background Art

[0002] The function of tracking is a standard function in video surveillance applications. It is used for the obvious task of tracking objects in a surveillance scene, but it is also used as a means to provide robustness to functions such as people counting and crossing line detection in such a scene. Therefore, the quality of the tracking method will affect multiple functions of the surveillance system.

[0003] Tracking an object involves associating multiple detections of the object into an object trajectory. This is typically achieved by predicting where the currently tracked object will be observed next based on the current movement of the currently tracked object. If the next detection is close enough to the prediction, it is considered to be related to the same object. Although the basic method is included in the prior art, the actual method is slightly more complex than this, and some of its parts will be elaborated in detail in the detailed description. There are situations where the trajectory is lost due to, for example, occlusion by another object or confusion with other objects.

[0004] Since the possibility of extracting object features emerged, which means that the identity of an object can be defined by its identifying features, it is possible to repair broken trajectories by using re-identification in appropriate cases. This basically corresponds to performing an automatic verification of the object identity, and if two trajectory segments are related to the same object, the two trajectory segments are reconnected into a single trajectory. Re-identification will require sufficient image quality to be reliable, so it is more commonly used in camera systems with better resolution and / or the ability to zoom in on objects to achieve sufficient pixel density.

[0005] The latter type of camera system, especially if they are also able to pan and tilt to keep the tracked object in the field of view, has a significant advantage in tracking objects. However, the related drawback is that while one object is being tracked, the rest of the scene may not be monitored. Therefore, a camera system designed to provide a continuous scene overview and the ability to zoom in on individual objects or locations may include one or more panoramic cameras with a wide field of view and one or more PTZ cameras (pan-tilt-zoom cameras) to achieve a detailed view. In short, panoramic cameras are likely to have a greater pixel density than PTZ cameras, but due to the ability to zoom in, PTZ cameras will be able to acquire images with a higher object pixel density.

[0006] Accordingly, there is a need for an improved method and configuration for tracking, particularly tracking using panoramic cameras. The term "panoramic camera" can be defined by its limitations in a particular camera setup. It refers to a camera where, for a particular scene or part of a particular scene, an image of sufficient quality cannot be obtained for re-identification. A typical example would be where an object is too far away for the camera to extract an image with sufficient pixel density. This can be expressed as tracking where the image quality may limit the tracking performance, which is true for any camera at one point or another. In camera implementations, a panoramic camera typically corresponds to a fixed camera with a wide field of view (fixed orientation and fixed zoom or no zoom), designed to provide an overview of the scene and situational awareness. Several panoramic cameras can be used in combination to cover a larger field of view, and one or more PTZ cameras can be arranged to provide a detailed view of parts of the monitored scene. Summary of the Invention

[0007] The object of the present invention is to provide an improved method for tracking objects in a scene, particularly in cases where re-identification cannot be utilized. According to a first aspect, these and other objects can be achieved in whole or at least in part by the method according to claim 1. Another object is to provide a camera system configured to perform such a method, as detailed in the following independent claims and their dependent claims, as described in the detailed description below.

[0008] According to several embodiments of the present invention, advantages of the present invention include improved tracking, especially in cases where current tracking algorithms have difficulty tracking a tracked object from one location to another. According to claim 1, this is achieved by a method for improving the tracking of an object in a scene using a panoramic surveillance camera. The method includes tracking an object in a plurality of image frames depicting the scene to generate a current object trajectory. Part of the tracking is to detect object candidates in an image frame subsequent to the plurality of image frames depicting the scene, with the aim of finding which of these object candidates belong to the current trajectory. To this end, an association metric indicating the likelihood of an object candidate being associated with the current object trajectory is calculated for each object candidate. In cases where the quality of the image data is poor or other reasons introduce uncertainty in the tracking, the likelihood can be enhanced by introducing further steps, including: associating the view of the panoramic camera with a heatmap of the scene, the heatmap providing data indicating regions with a high degree of occurrence of historical object trajectories in the scene, and adjusting the association metric or association threshold of object candidates located in regions with a high degree of occurrence of historical object trajectories in the scene according to the heatmap, so as to increase the probability of an object candidate being associated with the current object trajectory. Once these steps are taken, if the association metric is higher than the association threshold, the tracking algorithm can proceed with the step of associating each object candidate with the current object trajectory. In this way, object candidates along frequently traveled paths will have a higher likelihood of being associated with the current trajectory.

[0009] Since the view of the panoramic camera is related to the heatmap, which means that positions in the panoramic camera can be converted to positions in the heatmap, it is thus possible to easily extract values from the heatmap and apply them to appropriate positions in the view of the panoramic camera.

[0010] The heatmap can preferably be generated by using a PTZ camera to track an object in the scene over time and storing the trajectory followed by the object. Using a PTZ camera enables improvement of the tracking performance in the monitored scene, which increases the amount of high-quality data of historical trajectories.

[0011] To further improve the quality of the heatmap data, intermittent or continuous re-identification is used for tracking to ensure verified trajectories from each individual object being tracked.

[0012] As will be described in further detail below, the heatmap includes position measurements of the recorded object trajectories, but it may also contain representations of measurements rather than actual measurements. In this way, the position data of historical trajectories will be easily available for querying. The heatmap can also include information about the typical speed, object category, or object velocity of historical trajectories, and all data can be used when filtering information from the heatmap before use, so as to achieve higher precision during the tracking process.

[0013] In this way, the method may also include selecting heatmap data corresponding to the identified object class or the identified object speed. An example could be that if the object being tracked is identified as a car, it may be beneficial to use only the historical data related to cars rather than including, for example, historical data on how humans move in the scene. The object speed can be used as a filtering parameter for selecting a sub - portion of the heatmap.

[0014] A typical system for performing the method will include a PTZ camera and a panoramic camera, and in order for the present invention to operate correctly, the view of the panoramic camera is position - calibrated using the view of the PTZ camera. Such a system will be able to perform the method according to any embodiment, and in any case, most such systems will perform position - calibration because the PTZ camera is typically used to zoom in on selected events in the view of the panoramic camera.

[0015] In addition to including information assembled using the PTZ camera, the heatmap may also include data from one or more panoramic cameras. One advantage of this is that it can speed up the generation of the functional heatmap, and as long as the data is robust enough, the more data the better. Another advantage is that this opportunity can be used to monitor how the actual association metric will change for the panoramic camera in the surveillance scene. As a simple example, in a case where the data from the panoramic camera is not robust enough for tracking, verification from the PTZ camera can confirm that it is indeed the same object as before. In use, the recorded data can be used when adjusting the association metric or the association threshold.

[0016] The association metric can be increased in the region of the heatmap that includes the verified trajectory, where the verified trajectory has, for example, a distance or heatmap intensity between the object detection and the heatmap path, to increase the likelihood of associating the object detection with the current trajectory. This can be done before tracking starts, and if, for example, the object is located far from the panoramic camera, this may have the effect of increasing the chance of the trajectory being located. A similar effect can be achieved by lowering the detection threshold in the corresponding region. The relevant region will be close to the historically verified trajectory, and "close" can be quantified as "within a position error range", meaning that if there is an object candidate detection that may be on the historically verified trajectory, it should have a greater chance of being associated with the trajectory. In addition, in fact, such a detection is more likely to be the actual object rather than a false detection.

[0017] While the system for performing the method preferably includes a PTZ camera and a panoramic camera, the principles of the present invention can be implemented in part by only a PTZ camera or by multiple PTZ cameras. The reason is that a PTZ camera can be used in a fully zoomed-in mode or at any suitable zoom level capable of collecting data for a heat map, as well as in a (fully) zoomed-out mode during normal operation. The reason for using a PTZ camera in the zoomed-out mode during normal operation is that the field of view covered in the zoomed-out mode is larger, thus improving the situational awareness ability, which is a key function of a panoramic camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a camera system that can be used in an embodiment of the present invention.

[0019] Figure 2 FIG. illustrates a scene overviewed by a camera system configured to perform a method according to an embodiment of the present invention.

[0020] Figure 3 FIG. illustrates Figure 2 another scene similar to

[0021] Figure 4 is Figure 3 a schematic heat map of verified object trajectories in the scene of

[0022] Figure 5 FIG. illustrates the use of a heat map according to an embodiment of the present invention Figure 4 of

[0023] Figure 6 is a diagram according to one or more embodiments showing how selecting different probability distribution curves can be a way to implement the present invention.

[0024] Figure 7 is a flowchart of an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Figure 1It is a schematic diagram of a camera system 100 that can be used when implementing the present invention. The camera system 100 includes a single PTZ camera 102, that is, a camera with panning, tilting, and zoom functions. These functions enable it to cover a large area while magnifying details and tracking objects moving at a distance. The camera system 100 also includes one or more panoramic cameras 104. Compared with the PTZ camera 102, the panoramic camera 104 can be a camera with a fixed focal length that generally covers a larger field of view. Similarly, compared with the PTZ camera, the panoramic camera 104 generally has a fixed orientation during use, that is, they lack motor-assisted panning and tilting during operation. In the illustrated embodiment, there are four panoramic cameras 104, which can together cover a 360-degree field of view of the scene. The panoramic cameras 104 are for the obvious purpose of covering a large field of view and providing an overall understanding of the scene. In Figure 1 In a dedicated system 100 of the type shown, the panoramic cameras 104 are aligned and calibrated so that a single stitched panoramic image is formed by combining the images from each panoramic camera. In addition, the panoramic cameras 104 are aligned with the PTZ camera 102 and vice versa, enabling the PTZ camera 102 to immediately move to image an area within the field of view of any of the panoramic cameras 104 and enabling the panoramic cameras to indicate, for example, with a graphic overlay such as a rectangle, the current field of view of the PTZ camera.

[0026] Figure 1 The camera system can be a system including all its functions such as those available in the Axis Q-60-61 series (such as the Q6010-E or Q6100-E of the present applicant). However, it can also be any custom combination of single cameras.

[0027] The resolution of the panoramic camera represented by each image pixel may be higher than that of the PTZ camera and vice versa, but for the current camera system, the zoom ability of the PTZ camera makes it superior in terms of the pixel density of the imaged object (especially an object at a greater distance), which means that when imaging an object, there will be more concentrated pixels dedicated to it.

[0028] Putting aside this specific embodiment, typical embodiments may include a PTZ camera 102 used when assembling the heat map, and at least one panoramic camera 104 used during the actual tracking process (one exception will be disclosed below). These cameras, the PTZ camera 102 and the (at least one) panoramic camera 104, do not have to be combined into a single unit 100, but they do need to be calibrated so that position conversion can be performed between the position in the coordinate system of the panoramic camera 104 and the PTZ camera 102, and vice versa. The at least one panoramic camera 104 can be a PTZ camera 102 in a reduced state, so in a very specific and perhaps not common at all embodiment, the PTZ camera 102 and the panoramic camera 104 can be the same. Although this is not considered the most preferred embodiment, the capabilities of different camera types are used at different times, which means that in the context of the present invention, a single camera can first perform the tasks of a PTZ camera and then reduce and perform the tasks of a panoramic camera. Although this is an unlikely embodiment, it will also be emphasized at a later stage for further clarification.

[0029] Figure 2 is a schematic overview of the monitored scene. The camera system 100, for example Figure 1 the type of camera system shown, is arranged in the upper left corner of the apartment building 106. Then, the panoramic camera of the camera system 100 will be able to provide an overview of each part of the scene, or at least an overview of the parts not blocked by the building 106 itself. Therefore, the PTZ camera will be able to image the same parts of the scene in a more selective manner. The scene contains a pair of roads and sidewalks 108, some vehicles 110, some pedestrians 112, a cyclist 114, and some trees and bushes 116, all of which may be moving objects (or in the case of greenery, swaying objects).

[0030] The PTZ camera 102 can follow the various objects 108 - 114 in the scene and increase the zoom level as the distance from the camera system 100 increases. The panoramic camera 104 can be used for the same purpose, except that it cannot increase (or change) the zoom level and can utilize the same image analysis. Examples of image analysis include:

[0031] - Motion detection, where the input of shifted pixels in a region can be interpreted as motion. This analysis will respond to moving objects including vehicles and individuals as well as swaying trees and moving shadows. By filtering, for example, the size of the coherent region, the characteristics of the detected changes, etc., certain types of motion can be distinguished even using this basic technique.

[0032] - Object detection, where features of an object can be extracted so that not only motion detection can be performed, but also the type of the object can be classified according to its size and even object category (person, car, bicycle, tree, etc.).

[0033] - Re-identification, where a feature vector representing the appearance of an object is extracted from a moving object in order to provide a unique identifier for the object.

[0034] Today, re-identification is typically used in the context of deep learning and neural networks, and it does require a high level of detail (resolution or pixel density) to be reliable. Currently, the computational cost of performing re-identification using neural networks is too high to re-identify each frame and object in real time at a higher frame rate. Instead, re-identification is typically performed intermittently, at a lower frame rate, and / or when triggered by an input (e.g., if a more basic tracker loses a track). If performed on recorded material, the computational cost is less of a concern because these processes can be easily carried out by accessing a more powerful or dedicated CPU.

[0035] For object tracking algorithms, motion detection and possibly object detection are only the first part of tracking. For a single scene and a single moment, there may be multiple detections. The second part is to filter all these detections to infer whether a new detection is related to a previous trajectory. This can be done using an "association metric", which basically corresponds to the likelihood that a new detection belongs to a previous trajectory. In traditional object tracking, the Kalman filter is commonly used, and the Kalman filter is considered the best linear unbiased estimator. The Kalman filter is an algorithm that predicts where an object will be in subsequent situations based on previous measurements. There is a lot of literature on the Kalman filter and tracking, but in the simplest case, it tracks an object over multiple frames and uses parameters such as the "last known position", "direction", and "speed" (i.e., the "state" of the object) to predict where the object will be, including the uncertainty involved in the prediction. For example, the prediction can include a probability density function that indicates the likelihood that the object will be in different positions, and this probability density function is calculated using statistical considerations based on the input data. Then, it will measure the position of the object, i.e., detect or observe, and the measurement itself involves its inherent uncertainty. Then, the predicted and measured positions will be used to determine the updated position of the object, and so on. In the case of video surveillance, the measured position will be based on image processing, and there is an additional challenge of knowing whether the detected object belongs to the object being tracked. For example, there can be multiple object detections near the predicted position, so there is also the additional problem of choosing which object detection to use. When using the above-mentioned association metric to verify the prediction, simply put, the closer to the predicted position, the higher the association metric, i.e., the higher the likelihood that the detection is related to the previous trajectory. For example, if the association metric exceeds an association threshold, the detected object can be associated with the previous object trajectory, and when there are multiple detections to consider, the one with the highest association metric will be selected. Since other parameters of the detection can also be considered, the present invention will use the term "association metric". In this way, compared with a method where any object that appears nearby has an equal chance of being associated with the current trajectory, the chance of maintaining the tracking of the same object will increase.

[0036] In a simple version, the association metric can be based purely on the distance to the predicted location, for example, by using the probability density function already mentioned. This would mean that if detection candidates have equal distance to the predicted location, they will have the same association metric and thus the same probability of belonging to the current trajectory. However, the association metric can be more complex and include parameters such as size and speed to further increase the chance of associating the correct detection with the current trajectory. It can also include or be used in combination with a re-identification part, which means taking into account the appearance of the object, for example by involving the feature vectors already mentioned. The weight given to this re-identification part may vary. For example, if based on the regular association metric, some object detections are equally likely to belong to the previous trajectory, a greater weight can be given to it. In case a trajectory has been lost for some time, the re-identification part can also be used to reconnect new trajectory segments or detections to the existing trajectory. In this case, a "trajectory segment" is a part of a trajectory that has not yet been associated with another trajectory.

[0037] The above list of image analysis is more or less in the order of the required level of detail (i.e., the level of detail / resolution of the observed object), which for a single camera would translate to the distance from the camera. The PTZ camera 102 will be able to adjust the zoom and thus utilize re-identification at a greater distance from the camera. This comes at the cost of a reduced field of view, and in the presence of multiple potentially interesting objects, one or several objects must be prioritized, or the direction of the PTZ camera 102 must be changed between the objects. At the same time, the panoramic camera 104 can track multiple objects simultaneously, but due to the lack of available zoom options, its tracking robustness may decrease faster with the distance from the camera system 100.

[0038] A straightforward attempt at an improved tracking solution could be to adjust the association metric or the association threshold with distance for the panoramic camera. This would essentially introduce more leeway for associating new detections with existing trajectories. This approach may be suitable for some use cases, but it may increase the occurrence of false tracking associations, which to some extent affects tracking reliability.

[0039] The idea of the present invention is to provide an improvement over the prior art in the context of the above examples. To this end, embodiments of the present invention may start from the data assembly period. During this period, a PTZ camera is used to assemble data on the movement in the scene, in particular the movement related to moving objects, whose movement may form a trajectory in the scene, as this is information that may be directly relevant to the monitored scene. This is done by tracking the objects, and the data will include the object position and the valid object position over time (i.e., which is also an indication of speed). The data may also include the object class (bus, bicycle, car, person, etc.), and even the identity of the object. Identity does not necessarily mean exactly knowing who the person is, etc., only that it is verified as always being the same person, and this person can be separated from another person. The same applies to the identity of other object types with respect to the term "person", for example, a specific trajectory may be related to a single bicycle or a single car, etc.

[0040] The PTZ camera will assemble data by tracking the moving objects. It can be a fully automatic tracking performed by the PTZ camera when it is not busy with user-defined tasks, but it can also be data collected during user-defined tasks. The user-defined task can be that the operator manually (using a graphical or physical user interface) moves the PTZ camera used to track the object to track the object. During this manual tracking, the tracking algorithm can still be running to extract the object position over time. The data obtained for the tracked object is at least the object position, which can be stored in a common position reference system so that it can be immediately retrieved by any camera of the system (or actually by the control system of the camera system). Each trajectory can be stored as multi-sampling points in a suitable coordinate system, preferably a common coordinate system for any camera of the system (which simplifies later use). It can also be stored as a multi-trajectory curve or any other suitable format. Since multiple trajectories follow the same or substantially the same path, the weight ("temperature") of this path can be increased, or it can just be an additional function where basically a path traveled twice will be given the value "2", while a path traveled 600 times will be given the value "600". These considerations are not closely related to the present invention, but rather to how data is assembled in a heat map.

[0041] Data assembly will generate at least one heatmap over the monitored area over time. The heatmap can be divided by object category, time of day, etc., hence the use of "at least one" heatmap. Of course, it can be a single multi-dimensional heatmap where various parameters are represented in different layers of the heatmap, but the effect will be equivalent if different parameters are stored in different heatmaps, different parts of a vector, different cells of a matrix, etc. The heatmap will correspond to a representation of a scene where the degree of motion will be quantified and can be presented by the intensity in the image of the heatmap. More specifically, it is not the motion itself but the appearance of the verified motion trajectories over time, so if there is no trackable motion, it will not be recorded in the heatmap. This has the advantage of eliminating spurious motion, the effect of vegetation moving back and forth, and most of the effect of motion detection caused by noise. The scale of the heatmap can be relative or absolute with respect to the number of verified object trajectories. At the beginning, it can be a linear dependency such that an area with ten verified object trajectories will be twice as "hot", or elevated, as an area with five verified object trajectories. Over time, a saturation effect may occur such that smaller differences in object trajectory frequency cannot be distinguished. Additionally, when it comes to using the data of the heatmap, one can choose to distinguish only between areas with verified motion trajectories and areas without motion trajectories. In this case, there may also be a threshold function such that the area requires a certain number of verified motion trajectories to be classified as an area "with verified motion trajectories". These verified motion trajectories will be referred to as "paths". The dynamics of the heatmap can be reduced by introducing a distance threshold, which is similar to the effect of segmentation where verified motion trajectories running parallel and within a threshold distance of each other can be represented by one identical path. This method may be beneficial for less traveled roads. For paths with a high frequency of verified motion trajectories, the trajectories will most likely be distributed along the path according to a normal distribution.

[0042] Then, the generated heatmap can be stored and used by the panoramic camera in a way related to two scenarios to be explained. The first is an object moving away from the panoramic camera into the distance, and the second is an object approaching the panoramic camera from the distance. From the perspective of the present invention, these two situations are more or less the same, but within the field of camera installation, these two situations are extreme because in the first case, it starts with the highest image quality, while in the second case, it starts with the lowest image quality.

[0043] In the first example, the panoramic camera will be able to track the object relatively easily. If needed, the solution is even good enough for re-identification, at least at the beginning. As the object moves away, the possibility of re-identification will eventually disappear, and the tracking will have to rely on other parameters, such as traditional Kalman filter tracking or similar types of tracking. At this stage, the heatmap has at least two uses. One is to adjust the association metric along the detected path. For example, if the conventional association metric used for object detection is calculated as 75%, it can be adjusted upward to 77% or 85% etc. according to its distance from the path. This makes it more likely to maintain the trajectory of moving along the path. There is a risk that false object detections along the path are included in the trajectory, but the benefits should outweigh this risk because the statistics of a specific scenario show that objects most often move along the path. This association metric will be used in combination with other tracking techniques, which means that, for example, the Kalman filter can still be used to predict the future position of the tracked object, but the chance of finding the object along the path will increase. This will have the effect of making the tracking more robust and reducing the chance of false movements outside the path involved. Such false movements may result from, for example, trees or shrubs moving in the wind, but since these movements never cause the tracking of the PTZ camera, they do not affect the heatmap.

[0044] Another method that can be used in combination with the former is to adjust the association threshold and actually lower the threshold that the association metric has to exceed in order to be associated with an existing trajectory.

[0045] The numbers assigned to be related to the metric or threshold only mean to indicate the change, as a qualitative measure. Each tracker has parameters representing the confidence about object association. The phrase "adjusted association metric" means that this parameter changes in the direction of making trajectory association more likely. As mentioned before, a general change in the threshold, for example as a function of the distance from the panoramic camera, can have the effect of increasing the number of motion detections in an unwanted way. At the same time, with the support of the heatmap, the spatial accuracy of performing this operation in the embodiments of the present invention can enhance the tracking ability.

[0046] Another option could be to fine-tune the Kalman filter (or any other tracking filter used) according to the heatmap. Fine-tuning may mean that the estimation of the association metric used to associate object detections changes asymmetrically with the distance from the predicted position. More specifically, in the direction orthogonal to the path, the attenuation of the measurement value may be steeper than in the direction consistent with the path. This makes it more likely to find the object along the path, which is expected, and also acts as a filter against false motion detections on either side of the path.

[0047] These two options can be combined, where the latter example also includes adjusting the detection threshold within the reshaped uncertainty region.

[0048] The above method can be carried out based only on the heatmap and conventional tracking algorithms. However, in an embodiment of the present invention, the adjustment of the detection threshold is carried out based on further data collected during the training time, but collected by a panoramic camera rather than a PTZ camera. Such further data can be actual tracking data from the panoramic camera, benefiting from the enhanced information of the PTZ camera to verify the object identity. Then, these further data can be used to adjust the threshold to a level suitable for a specific area of the monitoring scenario. In this case, it should be mentioned that the "training period" does not necessarily have a definite end point. For the heatmap to be usable, it must contain sufficient information for the purposes of the present invention. The term "sufficient" may be considered vague, but the fact is that there is no clear boundary for when there is sufficient information. For example, tracking objects along some roads and sidewalks is sufficient to have a positive impact on the present invention. Nevertheless, for obvious reasons, a larger statistical basis is better. In addition, the accumulation of information in the heatmap does not even have to have any distinct end points, and any tracking carried out during the life cycle of the camera system can be added to the heatmap.

[0049] Returning to the embodiment at hand and giving a more detailed example of the "further data" mentioned in the previous paragraph. The verification that can be carried out using the PTZ camera can be used to confirm the object tracked by the panoramic camera, especially in the case where the tracking algorithm used by the panoramic camera fails. In the setup of this embodiment, when assembling data for the heatmap, the PTZ camera and the panoramic camera will track the same object, which is more like an alternative. During the process of tracking using the panoramic camera, due to changes in distance, occlusion level, shadows, etc., the tracking confidence of the panoramic camera (i.e., the confidence of the tracking algorithm running on the image data acquired by the panoramic camera) can change. These changes can be recorded, and in specific cases where the association metric is lower than the association threshold of the tracker of the panoramic camera. The changing association metric can be added as a piece of data stored in the heatmap or a related lookup table. Using this data enables the tracking parameters of the panoramic camera to be adjusted in a more controllable manner, i.e., the association metric or association threshold as a function of the position along the path. Like other data related to the heatmap, this data may vary with the object type, time of day, etc. In any case, this embodiment enables a customized adjustment of the association threshold along the area of the heatmap. Regarding the changes over time of day, they are not significant experiences. There are indeed performance differences between the bright conditions of day and the pitch - black conditions of night, but in modern society, it is rarely pitch - black enough to affect modern digital surveillance cameras.

[0050] With this in mind, we turn to the scenario where an object approaches the camera system from a distance. Essentially, the same approach can be used, but the difference is that when the object appears in the distance, by default, it will start with an uncertain object detection (contrary to the previous scenario, where the starting point was the best tracking situation with high-resolution image data of an object close to the camera system). Therefore, object detection along the approaching path is more likely to be the starting point of a trajectory than object detection along the departing path. This can be utilized by the tracking algorithm as it is easier to initiate object tracking on or near the path. If the tracking is not verified using multiple consecutive relevant object detections, the initiated tracking can be cancelled. As mentioned before, the tracking algorithm can be made "easier" by adjusting the association metric or the association threshold or a combination of both.

[0051] Furthermore, once an object is detected and tracking is initiated, parameters such as category, size, and speed can be used to access the correct layer of the heatmap (or the correct heatmap, etc.) to further refine the threshold. This refinement corresponds to filtering the data from the heatmap to obtain data more relevant to the current object being tracked. If the operator wants to look for a specific object type (object class, objects traveling at a specific speed / below / above a specific speed, etc.), the most appropriate layer can of course be used starting from the origin. This may apply to any of the above situations.

[0052] One commonality among any of these methods is that the adjustment does not have to be applied all the time or at all locations in the scenario. The application can be triggered by requirements such as losing an object during tracking, changes in visibility, etc. Another commonality is that these adjustments are aimed at supporting or enhancing tracking along the path, rather than canceling detections outside the path, as objects found outside the path may be equally important in the monitoring scenario.

[0053] Figure 3 is a perspective view of a simplified scenario, essentially including several roads 108, a road junction 118, some vehicles 110, and surrounding fields 120.

[0054] Figure 4 is Figure 3A fictional heatmap of the scenario. The concentration of points represents the degree to which the trajectories appear (especially the appearance of trajectories, rather than the appearance of movement). It can be seen that some areas in the scenario have a higher degree of trajectory appearance than other areas, that is, some areas have a higher level of trajectory appearance. There will mainly be trajectories along the road. For the sake of illustration, the appearance or frequency of trajectories represented by reducing the intensity (or the concentration of points in the example of imaging) will gradually decrease to one side of the road. In a practical situation, the decrease may be relatively sharp because it is less likely to find a vehicle driving along the side of the road rather than on the road. This distribution may obviously vary depending on the scenario and may also differ within the scenario. Some movement will be detected in the fields, such as animals and agricultural vehicles, but in the most likely case, compared with the movement along the road, this movement will be negligible. Although not necessary, it is preferably to verify the trajectories assembled in the heatmap. In this case, "verified" may mean that the movement of the object may be relevant from a monitoring perspective. It can also mean that the object follows the trajectory and it can be verified that it is the same object throughout the trajectory. Then, the most preferred tracking algorithm of the PTZ camera used to assemble the heatmap is used to verify as much as possible. The excellent optical performance of the PTZ camera will serve as the first confirmation of the verification, but it can also involve more specific verification steps. Examples of verification can include re-identification that can be continuous, intermittent, or in cases where the trajectory needs to be verified. This may be after a temporary loss of the trajectory, where it is necessary to associate the old trajectory with the new trajectory to ensure that they are indeed related to different instances of the same object before merging them into one trajectory. Object feature vectors and neural networks can be used for re-identification, but other methods can also be used for re-identification. In an example of using data from manual tracking by an operator, it can be said that the verification is done by a person, although in most practical cases, even in this case, there are still algorithms working in the background. Either way, the result is that only data related to the trajectories formed by moving objects will be assembled in the heatmap or at least can be used in subsequent tracking. In contrast, there can be a lot of movement in the fields, such as the crops swaying back and forth, or even the movement that generates a moving "wave" throughout the field. However, the PTZ camera will not track this movement, and by using re-identification and tracking, it is easy to avoid confusing this movement with the movement of relevant objects. The heatmap can be used as it is, but thresholds can also be set, either absolute or dynamic (e.g., relative to the surrounding area), in order to more clearly define the paths. In a simplified heatmap, the centers of these paths can be given a value of 1, while all other areas are given a value of 0, and in other embodiments, a greater dynamic range of the heatmap can be used in order to obtain better resolution in the adjustment or to be able to prioritize between paths with different intensities (i.e., with different object tracking frequencies) if applicable.

[0055] Now, Figure 5 and Figure 6 will be used to explain how to use the heatmap of Figure 4 to adjust the association metric or the association threshold during the tracking process. In Figure 5 , it is illustrated how to track a moving object when the moving object in the image moves from right to left. The solid circle 124 represents the previous position of the object. That is, each solid circle 124 illustrates the state of the object changing over time, essentially the position, and the object moves from right to left (this type of data is the data assembled in the heatmap). It is found that the last verified position is before the fork in the road, i.e., the solid circle 124'. Next, the tracker will predict the consecutive positions. Assuming that the object has been moving in the same direction and at the same speed, the tracker will predict the next position, as shown by the filled diamond 126. Then, the tracker will investigate whether there are any detections nearby, the closer the better (in general). In the given example, two candidates are detected, namely the rings 128 and 130 at an equal distance d from the predicted position. In the method according to the prior art, these two detections will have an equal chance of belonging to the existing trajectory, i.e., being associated with the prediction and thus with the existing trajectory. At this stage, the information from the heatmap enters the equation, and an example of how to do this is presented by involving Figure 6 . Figure 6 illustrates some curves 132, 134 that can be used to estimate the likelihood of the detection being associated with the prediction based on the distance d between the prediction and the detection. In a simplified case, the likelihood may correspond to the association metric, as a higher value makes the association more appropriate than a lower value.

[0056] Now, according to an embodiment of the present invention, the application of the present invention will be to select a probability density function based on the position of the detection related to the heatmap. The upper detection 130 is considered not to be within the influence range of any heatmap because, for example, based on the distance from the path (taking the road 108 as an example) or based on the intensity of the heatmap at that position, it is not within or near the high-occurrence area. Based on this, the original or only slightly modified probability density function 132 can be used. The probability density function 132 can correspond to the probability density after the prediction from the tracker. For this fictional case, the input distance d will yield a fraction (or likelihood, or actually the association metric) of approximately 0.30 (in arbitrary units). However, for the other detection candidate 128, the distance between the detection and the path is small, so the intensity of the heatmap increases at the position of the detection. Therefore, the adjusted probability distribution 134 will be used, which results in an increase in the likelihood to a fraction of 0.70 despite the same distance from the prediction. This will mean that among the detection candidates, the detection candidate along the path will have the highest likelihood and the highest association metric and will thus be associated with the trajectory.

[0057] ​​​​​​​​​​​​In the case where there is only one detection candidate 128 and Figure 5 the detection 130 does not exist, this function also has its purpose. Based on the same example, for a tracker, an unexpected change in direction may cause the association metric to become too low for the only detection candidate 128, its score will not reach the threshold, and the tracking will be interrupted. However, with the present invention, the association metric can be adjusted according to the distance from the path, so as to reach the association threshold and continue the tracking. Instead of adjusting the association metric, the association threshold can be adjusted so that the detection candidates along the path of the heatmap are more likely to reach the threshold. The method used may vary depending on the type of tracker used.

[0058] It is worth noting that even if the heatmap is binary (path or no path), the influence of the association metric and / or the association threshold may be gradual. For example, it can be represented as a direct or more complex function of the distance from the detected path, taking into account factors such as the inherent uncertainty in determining the position of the detection.

[0059] In this context, it can be mentioned that throughout the description, the term "association metric" is used to describe a parameter that determines the likelihood of a detection being associated with an existing trajectory. An equivalent parameter can also be called "association cost", in which case a lower cost corresponds to a higher likelihood, "association probability" or just "probability", etc., or any term for the tracking model under discussion. Regardless of the term used, the present invention can be applied to enhance tracking, and regardless of which term is used, the term can be translated as "association metric".

[0060] The method is a simple and step-by-step way to implement the present invention, as Figure 7as shown in the flowchart. First, a heat map is assembled in step 136. This is done for the actual camera installation, i.e., the camera system is arranged at a position overlooking the expected field of view. Once the heat map exists, it can be used during the normal operation of the camera system, and the normal operation is initiated once an object is tracked by the camera of the camera system in step 138. During the tracking, object candidates are detected in step 140, and an association metric is calculated for the object candidates. This is depicted as a separate step 142, but in practice, the evaluation and calculation of the tracker may be intertwined in a more complex manner. After this step, the association metric or the association threshold is adjusted in steps 144 and 146, respectively. In other embodiments, both measures, i.e., adjusting the association metric and the association threshold, are taken. In the last step 148, additional information and the adjusted values are used as inputs when associating the object candidates with existing tracks. As mentioned above, the present invention can be implemented in other ways. An example that has been mentioned is generating a multi-dimensional probability distribution based on the input from the heat map. Depending on the specific tracker used, other embodiments of the present invention may also be more suitable than the remaining embodiments. Based on the description of the present invention, such a decision is considered within the capabilities of those skilled in the art. In addition, it should be emphasized again that depending on the metric or threshold used, the present invention can be applied by adjusting the metric or threshold up or down according to the metric / threshold. Therefore, it is meaningless to mention the adjustment direction in the appended claims unless it is related to a specific embodiment. Therefore, when the expected effect of the adjustment is mentioned in the claims, it should be read in this context and not be confused with the effort to define the present invention by the result to be achieved.

Claims

1. A method for improving tracking of objects in a scene using a panoramic surveillance camera, comprising: tracking the object in a plurality of image frames depicting the scene to produce a current object trajectory, detecting object candidates in image frames subsequent to the plurality of image frames depicting the scene, and For each object candidate, computing an association metric indicating the likelihood that the object candidate is associated with the current object track, Wherein the method further comprises: associating the panoramic camera's view with a heat map of the scene, the heat map of the scene providing data indicating areas in the scene having a higher degree of occurrence of historical object tracks, and adjusting an association metric or an association threshold for object candidates located in regions having a higher degree of occurrence of historical object trajectories in the scene according to the heat map so as to increase the probability of the object candidates being associated with the current object trajectory, Each object candidate is associated with the current object track if the association metric is above an association threshold.

2. The method according to claim 1, wherein: The heat map is generated using a PTZ camera to track objects in the scene over time and store the trajectories followed by the objects.

3. The method according to claim 1, wherein: The heat map is generated using a PTZ camera to track objects in the scene over time, and wherein the tracking is performed using intermittent or continuous re-identification to ensure a verified track from each individual object being tracked.

4. The method according to claim 1, wherein: The heat map includes position measurements of recorded object trajectories.

5. The method according to claim 1, wherein: The heat map includes velocity information of the recorded object trajectories.

6. The method according to claim 1, wherein: The heat map includes object categories or object speeds of the recorded object trajectories so that filtering of the object categories or object speeds can be performed. 7 . The method of claim 6 , further comprising selecting heat map data corresponding to the identified object category or the identified object speed.

8. The method according to claim 7, wherein: Tracking of the current object includes identifying an object category to which the tracked object belongs or an object speed of the object.

9. The method of claim 1, performed by a system comprising a PTZ camera and a panoramic camera, wherein the view of the panoramic camera is positionally calibrated with the view of the PTZ camera.

10. The method according to claim 2, wherein: During assembly of the heatmap, image data from the panoramic camera is also used for tracking, wherein associated metrics of tracking by the panoramic camera are monitored against trajectories verified by the PTZ camera.

11. The method according to claim 1, wherein: The association threshold is lowered in regions of the heat map that include validated tracks.

12. The method according to claim 1, wherein: The association metric is increased in regions of the heatmap that include verified tracks, for example with the distance between an object detection and a path of the heatmap or with the heatmap intensity, so as to increase the likelihood of the object detection being associated with the current track.

13. The method according to claim 1, wherein: The functionality of the panoramic camera is provided by a PTZ camera in zoomed-out mode.

14. A camera system, comprising a PTZ camera and a panoramic camera, the camera system being configured to perform the method of claim 1.

Images