Method and non-transitory computer-readable storage medium for detecting one or more occluded areas of scene

Abstract

To provide a method for detecting one or more occluded areas of a scene analyzed by an object tracking system.SOLUTION: A method includes building (S506) a map of one or more occluded areas in a scene. Building the map comprises running (S510) a re-identification algorithm on a video sequence to try to resume a lost object track. If the object track is successfully resumed, the method includes determining (S512) an area of the scene where a first object track is lost and an area of the scene where the first object track is resumed. A connection between first and second areas of the scene is added (S514) to the map such that the map identifies that the object track lost in the first area of the scene has been resumed in the second area of the scene.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to object tracking, and in particular to a computer-implemented method for detecting one or more occluded areas of a scene analyzed by an object tracking system. [Background technology]

[0002] Object tracking is an important and widely researched area within computer vision. The goal of object tracking is to keep track of an object (or multiple objects) in sensor data capturing a scene.

[0003] Occlusion presents a significant challenge in object tracking, for example, when the object to be tracked is hidden by static objects in the scene: for example, a tracking algorithm may lose a pedestrian who is momentarily obscured behind a tree or building.

[0004] When an object track is lost, a process called coasting may be used to attempt to resume the object track again. Coasting refers to the practice of predicting the object's position based on its previous movement when the object is temporarily undetected, such as during occlusion or signal loss. Coasting utilizes techniques such as linear extrapolation or more complex motion models to estimate the object's trajectory. Coasting continues to estimate the object's trajectory during periods of loss and maintains a virtual track. When a new object is detected near the virtual track, it may be considered to be the same as the object in the lost track, the track may be considered to be successfully reacquired, and tracking resumes. However, coasting carries the risk of introducing errors over time, especially if the object changes its speed or direction while undetected. For this reason, object track coasting is generally only performed for a certain period of time (called the coasting period). If the object track is not successfully reacquired before the end of the coasting period, the coasting is deemed to have failed and the object track is deemed lost and deleted.

[0005] When the coasting process fails, or in general when an object track is lost, a re-identification process may be used to attempt to resume the lost track. Re-identification refers to the process of identifying an object in a video sequence as the same object previously detected (e.g., before it was occluded) in one or more previous frames of the video sequence or in a frame of another video sequence. When the object is re-identified, the lost track may be continued. Re-identification works by extracting meaningful features from the detected object. When an object needs to be re-identified (e.g., after it is occluded or leaves the field of view), features are extracted from the untracked object in the scene. A similarity measure (such as Euclidean distance or cosine similarity) is then used to compare the feature vector of the new detection with the feature vector previously extracted for the object in the lost track. If a match is detected, the lost track may be continued.

[0006] Re-identification generally requires significant computational resources and, moreover, re-identification is not possible in all object tracking systems, such as systems that track objects in data captured by radar devices.

[0007] Maintaining a map of occluded areas of a monitored scene may improve the coasting and / or re-identification process. However, correctly identifying occluded areas in a scene and using these to predict object behavior while occluded may be complicated.

[0008] There is therefore a need for improvement in this context.

[0009] Hierarchical Databases for Multi-Camera Surveillance Systems (JAMES BLACK et al., PATTERN ANALYSIS AND APPLICATIONS, SPRINGER-VERLAG, LO, vol. 7, no. 4, 1 December 2004, XP019381474) presents a framework for event detection and video content analysis for visual surveillance applications.

[0010] Real-Time Traffic Monitoring with Occlusion Handling (FERNANDEZ-SANJURJO MAURO et al., TOPICS IN CRYPTOLOGY - CT-RSA 2020, September 22, 2019, XP047523419) proposes a traffic monitoring system that combines detection and tracking in a way that can meet the requirements of being robust to occlusions while still operating in real time. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] JAMES BLACK et al., PATTERN ANALYSIS AND APPLICATIONS, SPRINGER-VERLAG, LO, vol. 7, no. 4, December 1, 2004, XP019381474 [Non-Patent Document 2] FERNANDEZ-SANJURJO MAURO et al., TOPICS IN CRYPTOLOGY - CT-RSA 2020, September 22, 2019, XP047523419 Summary of the Invention

[0012] In view of the above, it would be advantageous to overcome or at least reduce one or some of the above-described disadvantages as set forth in the accompanying independent claims.

[0013] According to a first aspect of the present invention, there is provided a computer-implemented method for detecting one or more occluded areas in one or more video sequences analyzed by an object tracking system, the method comprising: providing one or more video sequences, the one or more video sequences depicting a same scene, the one or more video sequences including a plurality of objects; determining the plurality of objects in the one or more video sequences; and detecting the one or more occluded areas in the one or more video sequences.

[0014] Constructing the map includes steps a) and b). a) upon determining that a first object track of the plurality of object tracks corresponding to a first object of the plurality of objects is lost, operating a re-identification algorithm on at least one video sequence of the one or more video sequences to attempt to resume the first object track; b) determining a first area in which the first object track was lost upon successful restart of the first object track, the first area being determined in one or more video sequences of the multiple areas in the one or more video sequences, and a second area in which the first object track is resumed, the second area being determined in one or more video sequences of the multiple areas in the one or more video sequences, wherein each area of ​​the multiple areas in the one or more video sequences refers to an area of ​​a 2D representation of a 3D area of ​​the scene captured by the one or more video sequences, and the one or more video sequences are related to a base coordinate system to which objects and areas represented in image frames of each video sequence are transformed; and adding a connection between the first area and the second area to the map of the one or more occluded areas such that the map identifies that the object track lost in the first area has been resumed in the second area.

[0015] In the following, the expression "area in one or more video sequences representing a scene" and similar expressions referring to an area in a 2D representation of a scene captured by one or more video sequences may be simplified to just "area of ​​the scene" or similar for simplicity and ease of explanation. It should be noted that when several video sequences are used, which are captured, for example, from different positions in the (real) scene, the coordinate system of one of the video sequences may generally serve as a base coordinate system for the multiple video sequences, and thus object detections, areas, etc. in the video sequences may be transformed to the base coordinate system. If only one video sequence generated by a fixed camera captures the scene, each area of ​​the scene may be represented by a certain pixel area in the image frames of the video sequence. If one video sequence is captured by a video camera with a changing field of view, a base coordinate system similar to that described above may be determined and used to transform areas / detections from one image frame to another (e.g., using the pan, tilt, roll, zoom parameters of the moving camera). Generally, as used herein, an area of ​​a scene is a representation of a real-world 3D area in a 2D representation formed by a video sequence capturing a real-world scene.

[0016] The term "occluded area" should be understood in the context of this specification as an area in one or more video sequences representing a scene, where an object tracker system that tracks an object in one or more video sequences loses track of the tracked object. An occluded area is referred to herein as an area between a first area where the track was lost and a second area where the track was resumed. An occluded area is thus an area in the video sequence(s) that extends between an area, a first area ("start area"), where historical tracking data statistically indicates that the tracker is no longer able to track the object (or predicts with reasonable certainty, possibly after a coasting period), and another area, a second area ("end area"), where a re-identification algorithm has successfully aligned the track of the lost object with the track of another object. Thus, as used herein, a map of occluded areas refers to a map of connections between areas in a video sequence, i.e., between a start area and one or more end areas.

[0017] An occluded area may be the result of a static object preventing direct observation or detection (in sensor data from a sensing or imaging system such as a camera or radar) of other objects behind the static object. It may also be the result of a more temporary occlusion of a tracked object in a real-world scene, such as a person being temporarily occluded by another person or car. Other examples of reasons for an occluded area include the area being very dark in an area in one or more video sequences, or a privacy mask being added to an area in one or more video sequences, such that no object detection is received from the object tracking system for objects in these areas.

[0018] Object tracking is the process of identifying and following the location and movement of an object across a series of image frames in a video sequence. Object tracking involves detecting and identifying an object in an initial frame, and then continuously updating the object's location in subsequent frames by matching object detections in subsequent frames to the object in the initial frame. The term "object track is lost" or similar terms should be understood in the context of this specification as an object tracker system not detecting an object in a frame that can be matched to an object track detected in a previous frame for some reason. In some embodiments, an object track may be considered lost after a coasting process fails, as described further below. In other embodiments, an object track may be considered lost as soon as the object tracker system does not detect an object in a frame that can be matched to an object track from a previous frame, for example, when coasting is not implemented, or for other reasons. In some embodiments, an object track is considered lost when the object tracker system has not detected an object that can be matched to an object track from a previous frame for a threshold number of frames.

[0019] The term "resuming a first object track" or similar terms should be understood in the context of this specification as a previously lost object track (e.g., due to occlusion or the object moving out of the frame) being successfully resumed (restored, re-established, continued, aligned with a previously lost track, etc.) using a re-identification algorithm. The goal of re-identification is to correctly associate or match a given object across different video sequences or image frames, even when there is a time gap between these observations. With the help of a re-identification algorithm, the initial object tracking process can thus be resumed by tracking the same object that has been re-identified following a period of loss (e.g., due to occlusion by another object).

[0020] As described above, in object tracking applications, coasting is commonly used to attempt to resume an object track when a tracked object is occluded. The coasting process has a set coasting period that defines how long the virtual track is maintained before the coasting process times out. After the coasting process times out, or for other reasons described above, the track may be considered lost and re-identification may be employed to attempt to resume the object track. The re-identification algorithm may be run on subsequent image frames in the video sequence as where the latest observation of the object in the lost object track was made, or on other video sequences capturing the same scene. Upon successful re-identification, the object track may be resumed by continuing tracking the re-identified object.

[0021] The inventors have realized that successful re-identification may indicate the location and size of the occluded area in the monitored scene, as well as the trajectory of the object when occluded. In particular, the location / area in the monitored scene where the object track was lost may be identified and registered (e.g., stored in a database structure, in a log file, or in a memory) using the location in the scene where the tracked object was detected. When the lost track is successfully resumed using the re-identification process, the location / area in the monitored scene where the track was resumed (i.e., an object determined to match the object in the lost track was detected) may be registered as well. Advantageously, it may be determined that the object lost in the first area of ​​the scene has been re-identified in the second area of ​​the scene. Thus, it may be determined that an occluded area exists between the first and second areas of the scene. Thus, a map of one or more occluded areas of the scene may be determined. In the following, the map of one or more occluded areas may be referred to simply as the map of occluded areas for simplicity.

[0022] The map of one or more occluded areas may include connections between areas of the monitored scene, such that for a particular area of ​​the scene (the "start" of the occluded area), the map identifies one or more other areas in the scene (the "end" of the occluded area) where an object track that was lost in that particular area of ​​the scene was resumed. The map may be implemented using a table or similar data structure, where each area where an object track was lost is connected (e.g., connected via a shared ID or the like) to one or more areas where the object track was resumed. The connections may be implemented by creating a table of locations in one or more video sequences where an object track was lost and successfully resumed. These locations may be mapped (e.g., using a predefined area, or a clustering algorithm) to areas in one or more video sequences, as described further below. Each area may be represented by a center point or by a range (e.g., pixel coordinates or pixel area). This may be used to determine whether an object is proximate to an area (e.g., just before the object tracker loses track of the object), as described further below.

[0023] By applying re-identification to all newly appearing objects (i.e., objects not currently being tracked), spatial connections are established between the locations in the scene where an object disappeared and the positions in the scene where the same object later reappears. These disappearances can be interpreted as occlusions in different parts of the scene and later used to modify the tracking system behavior in the occluded parts of the scene, thereby making the tracking system scene-adaptive and improved.

[0024] Advantageously, a map of occluded areas may be maintained and updated. For example, the map may be defined "offline" using a video sequence(s) capturing the monitored scene. The map of occluded areas may then be used in real-time object tracking applications to improve object tracking, for example, to reduce the number of lost object tracks, to reduce the number of erroneous object tracks, and / or to reduce the computational resources required to implement object tracking.

[0025] In some embodiments, constructing the map includes performing steps a) and b) for multiple lost object tracks of the multiple object tracks. Thus, the map of one or more occluded areas may identify multiple areas of the scene where one or more object tracks are lost. Moreover, the map of occluded areas may identify, for each area where one or more object tracks are lost, one or more areas where the lost object(s) resume. Advantageously, a more flexible map of one or more occluded areas may be determined, identifying multiple occluded areas in a scene and / or taking into account that an occluded object may take different trajectories while being occluded. In some embodiments, an area of ​​a scene is not determined to be a start or end of an occluded area until at least a threshold number of object tracks are lost or resumed in this area. Advantageously, a more reliable map of occluded areas may be determined.

[0026] In some embodiments, the map of one or more occluded areas further indicates the probability that an object track lost in a first area is resumed in a second area. For example, if 5 of 10 object tracks lost in a first area of ​​a scene are resumed in a second area of ​​the scene, the probability for that connection in the map of occluded areas may be indicated as 50%.

[0027] In some embodiments, the map indicates that an object track lost in a first area is resumed in a subset of areas in one or more video sequences, the subset including at least two areas in one or more video sequences of the multiple areas in the one or more video sequences, and the map further indicates, for each area in the subset, a probability that an object track lost in the first area is resumed in that area in the subset. In other words, if an object track lost in a first area of ​​a scene is determined to be resumed, for example, in a second and third area of ​​the scene (hence a subset of areas in the scene, this subset including two areas of the scene in this example), the map may indicate a probability for each of these connections. For example, if five of the ten object tracks lost in a first area of ​​a scene are resumed in a second area of ​​the scene, the probability for that connection (between the first and second areas of the scene) in the map of occluded areas may be indicated to be 50%. Of the remaining five lost tracks, three may be resumed in the third area of ​​the scene, and thus the probability for that connection (between the first and third areas of the scene) in the map of occluded areas is indicated as 30%. The remaining two lost tracks may not be successfully resumed using, for example, a re-identification process, or may not be determined to be resumed in yet another area of ​​the scene.

[0028] In an example, the method further includes providing sensor data capturing a scene; tracking a second object in the sensor data to determine a second object track; and upon determining that the second object is not trackable in the sensor data, determining a location in the sensor data of a latest observation of the second object in the second object track; and upon determining that the location of the latest observation of the second object in the second object track is within a threshold distance from the first area, identifying from the map an area in the one or more video sequences where the object track that was lost in the first area was resumed and determining a distance between the first area and the identified area; and setting a coasting period for the second object track based on the determined distance, where a larger distance results in an equivalently longer coasting period.

[0029] Advantageously, the map of occluded areas can be used to adapt the coasting period / time based on where the object of the object track is located in the scene when coasting is initiated for the object track. When coasting is initiated, the object is considered untrackable in the sensor data, for example, due to the object being occluded. When an object is occluded or otherwise undetectable in the current image frame of a video sequence (or similar for other types of sensor data), there is no detection of the object in the current image frame that can be matched with an existing object track for that object (i.e., determined for a previous image frame). In other words, the second object is considered untrackable when coasting is initiated for the second object track.

[0030] In general, the coasting period is pre-determined for the object tracking system. However, based on the size of the occluded area, the coasting period may need to be extended in some areas of the scene and may be reduced for other areas. In this example, if a second object is detected in or near a first area of ​​the scene before coasting is initiated, the coasting period may therefore be adapted based on an estimated distance in the scene between the start (first area) and the end (determined from the map of the occluded area) of the occluded area into which the second object has entered (and thus is not trackable in the sensor data). For a second object (which has a certain speed before coasting is initiated), a longer distance results in a longer coasting period compared to a shorter distance.

[0031] Advantageously, this may improve object tracking and reduce the number of lost tracks, which reduces the need for computationally demanding re-identification processes. Moreover, as described above, re-identification may not be available for all types of sensor data. For example, re-identification is not available for radar (radio detection and ranging) data or lidar (light detection and ranging) data. Using the techniques described herein, video data capturing a scene may be used to define a map of occluded areas, and object tracking using radar data may be improved using the map of occluded areas, as described herein. In this embodiment, the map of occluded areas is transformed into radar coordinates. If the sensor data comprises a video sequence captured from another location or otherwise has a different coordinate system compared to the map of occluded areas, a transformation is required to map locations, etc., from the video sequence to the map of occluded areas, as further described above.

[0032] Furthermore, by reducing the coasting time for some areas of a scene, the removal of tracks that are no longer relevant may be improved, which may reduce computations and reduce the risk of erroneous results from coasting.

[0033] In an example, the step of identifying from the map an area in the one or more video sequences where an object track that was lost in the first area has been resumed includes selecting an area in the one or more video sequences from the subset that has the greatest probability as indicated by the map.

[0034] Thus, if the map of one or more occluded areas identifies two or more areas of a scene where a lost object track in a first area of ​​the scene has been resumed, the most likely area from the subset of areas may be selected and used to determine the coasting period. Alternatively or additionally, the angle of motion that the second object has in the most recent observation of the second object in the second object track (before the coasting process is started) may be used when selecting which of the subset of areas to use to determine the coasting period, such that an area of ​​the subset of areas having a most similar angle with respect to the first area may be prioritized (e.g., has increased probability) in the selection process.

[0035] In some embodiments, setting the coasting period includes determining a speed of the second object in a most recent observation of the second object in the second object track, and setting the coasting period for the second track further based on the speed of the second object, where a higher speed results in an equivalently shorter coasting period.

[0036] Thus, both the estimated distance from the start to the end of the occluded area (i.e., the estimated size of the occluded area) and the velocity of the second object are used to determine the coasting period, resulting in a more accurate estimate of how long the second object may be occluded.

[0037] In some embodiments, the estimated trajectory and position of the second object during occlusion may also be improved using the techniques described herein. For example, in some embodiments, upon determining that the second object is not trackable in the sensor data, the method includes predicting the position of the second object while it is not trackable in the sensor data based on the determined speed. For example, the time at which the second object is considered not trackable + the determined speed may result in an improved coasting process, thus resulting in the virtual track of the second object being more accurate.

[0038] In some examples, the method further includes, upon determining that the second object is not trackable in the sensor data, determining an angle between the first area and the identified area in the one or more video sequences, and predicting a position of the second object while it is not trackable in the sensor data further based on the angle.

[0039] Advantageously, the position of the second object in the virtual track can be predicted as described herein, so that the virtual track of the second object can be more accurately estimated while the second object is occluded, based on the behavior of the object previously occluded in the same area of ​​the scene.

[0040] According to some embodiments, the sensor data is one of radar data, lidar data, or video data. As previously described, the techniques described herein may be used to improve object tracking in radar / lidar data, where re-identification algorithms are not applicable. Moreover, for both radar, lidar, and video data, a map of occluded areas may be employed to improve the coasting process, as described herein.

[0041] According to some embodiments, the multiple areas in the one or more video sequences include multiple predetermined areas in the one or more video sequences. For example, a scene may be divided into areas based on the format and resolution of sensor data capturing the scene. In some examples, each area corresponds to a certain area of ​​pixels (X*Y pixel size) in an image frame of a video sequence capturing the scene. In some examples, each area corresponds to a certain physical size of the scene, such as Z*W meters (or other size units) represented by the one or more video sequences.

[0042] In some examples, one or more video sequences are iteratively divided into multiple areas in one or more video sequences based on the locations in the one or more video sequences where object tracks of the multiple object tracks are lost and resumed. For example, the construction stage (offline or online) may comprise a first stage in which objects are tracked to determine the locations in the scene where object tracks are lost and resumed to determine the areas of the scene to be used in a second stage in which a map of occluded areas is determined (based on the multiple areas of the scene), as described above. In other embodiments, the first and second stages are implemented simultaneously, such that when the area of ​​the scene changes based on the collected statistics on lost and resumed (re-identified) tracks, the map of occluded areas is updated accordingly.

[0043] In some embodiments, the step of iteratively dividing the one or more video sequences into multiple areas in the one or more video sequences includes determining a first plurality of locations, each location indicating a location in the one or more video sequences where an object track of the multiple object tracks is lost, clustering the first plurality of locations into a first plurality of clusters, and for each cluster of the first plurality of clusters, determining an area in the one or more video sequences based on its location in the cluster; and determining a second plurality of locations, each location indicating a location in the one or more video sequences where an object track of the multiple object tracks is resumed, clustering the second plurality of locations into a second plurality of clusters, and for each cluster of the second plurality of clusters, determining an area in the one or more video sequences based on its location in the cluster.

[0044] Any suitable clustering algorithm may be used depending on the requirements of the object tracking application. Examples of clustering algorithms include OPTICS (Ordering of Points to Identify Clustering Structure), Mean Shift, and Density-Based Spatial Clustering of Applications with Noise (DBSCAN).

[0045] As noted above, maps of occluded areas may have many uses in object detection or tracking applications.

[0046] In some embodiments, the method includes providing sensor data capturing a scene, tracking a third object in the sensor data to determine a third object track, determining that the location of the third object in the sensor data is within a threshold distance from a first area in one or more video sequences, and determining a feature vector for the third object for the purpose of object re-identification in an object tracking application. Advantageously, the method may provide low complexity and efficient selection criteria by which objects may be analyzed to determine a feature vector for object re-identification. By prioritizing areas in the scene where an object track was previously lost, objects that may be occluded in subsequent image frames may be prioritized for feature extraction. This approach prioritizes objects where the tracking system may lose its track in subsequent image frames.

[0047] According to a second aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium storing instructions for implementing the method according to the first aspect when executed on a device having processing capability.

[0048] According to a third aspect of the present invention, the above object is achieved by a system for detecting one or more occluded areas of a scene analyzed by an object tracking system, comprising one or more processors and one or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the system to perform actions according to the first aspect.

[0049] The second and third aspects may generally have the same features and advantages as the first aspect. Furthermore, it should be noted that the present disclosure relates to all possible combinations of features unless otherwise specified.

[0050] The above, as well as additional objects, features, and advantages of the present invention will be better understood through the following illustrative, non-limiting detailed description of embodiments of the present disclosure, with reference to the accompanying drawings, in which like reference numerals will be used for similar elements, in which: [Brief description of the drawings]

[0051] [Figure 1] FIG. 2 illustrates a scene including multiple objects and multiple occluded areas, according to an embodiment. [Diagram 2] FIG. 2 illustrates a visualization of a map of occluded areas determined for the scene shown in FIG. 1 according to an embodiment. [Diagram 3] FIG. 2 illustrates a visualization of a portion of the map of occluded areas determined for the scene shown in FIG. 1 according to another embodiment. [Figure 4] FIG. 2 shows a visualization of a map of occluded areas determined for the scene shown in FIG. 1 according to a third embodiment; [Diagram 5] FIG. 2 illustrates a method for detecting one or more occluded areas of a scene analyzed by an object tracking system, according to an embodiment. [Figure 6] FIG. 6 illustrates a method for determining coasting parameters using a map of occluded areas detected using the method of FIG. 5. [Figure 7] FIG. 6 illustrates a method for using a map of occluded areas detected using the method of FIG. 5 to prioritize areas in a scene for the purposes of object re-identification in an object tracking application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0052] Occlusion presents a significant challenge in object tracking because occlusion becomes a temporary obstruction of the target object from the perspective of the tracking system. The lack of visual information during occlusion periods introduces uncertainty into the tracking process. Furthermore, sudden changes in the visible parts of the object can cause confusion for the tracking system, especially if the tracking system relies heavily on visual features to identify and track the object. Additional difficulties can arise if the object changes its state, such as speed or direction, during the occlusion. This can make predicting the reappearance of the object particularly challenging, thereby complicating the process of re-identification.

[0053] Object tracking systems may use several techniques to identify occluded areas in a monitored scene and follow the track. For example, using depth information from a stereo camera, lidar, or depth sensor, a tracking system can identify when an object moves behind another, indicating a potential occlusion. However, this technique relies on the availability and accuracy of depth information, which is not always reliable or available. Other techniques include machine learning models that can be trained to predict occlusions based on patterns and features in the input data. These models can be complex to implement and require a significant amount of training data.

[0054] The inventors have realised that using re-identification techniques that are existing components in many object tracking systems, a map of occluded areas can be constructed and used to improve object tracking. Such a technique will now be described with respect to Figures 1-4, supported by the flow chart presented in Figure 5.

[0055] 1 shows a scene 100 analyzed by an object tracking system. The scene includes three static objects 104, 106, 107 that form an occluded area. For example, an individual 112 jogging in the scene 100 may momentarily disappear behind a tree 106. Similarly, another person 114 walking in the same scene 100 may momentarily disappear behind a playground structure 107. A car 110 moving along a road 109 may be obscured by a house 104, and another pedestrian 116 walking adjacent to the road 109 may also be hidden by the same house 104. Depending on the size and shape of the occluding objects 104, 106, 107, how fast the objects 110, 112, 114, 116 move while they are occluded, as well as the direction of those objects 110, 112, 114, 116, all affect how long the duration of time the moving objects 110, 112, 114, 116 are hidden from the view of a tracking system analyzing the scene 100.

[0056] As described above, when an object track is lost, a process called coasting may be used to attempt to resume the object track again. Coasting continues to estimate the object's trajectory during the period of loss, maintaining a virtual track, for example using a Kalman filter. Coasting has the risk of introducing errors over time, especially if the object changes its speed or direction while undetected. Moreover, coasting is generally performed for a specified duration, referred to herein as a coasting period or coasting time. During the coasting period, the system uses prior information and a motion model to estimate the object's current state. Once the coasting period has elapsed, the tracking system will generally mark the track as lost if the object has not been redetected. After the track is lost, a process called re-identification may be performed, as described above and further described below. In other embodiments, for example when coasting is not implemented or for other reasons described above, a track may be considered (determined) to be lost and re-identification performed when the object tracker is no longer able to track the object in the object track, as further described above.

[0057] The scene 100 shown in Fig. 1 is divided into a number of predefined areas 102a-n, represented by dashed lines. For example, each such area corresponds to a pixel area of ​​an image frame of a video sequence capturing the scene 100. These areas 100a-n may be used to identify occluded areas in the scene. Other ways of dividing the scene 100 into areas may be employed, for example, as described below with respect to Fig. 4.

[0058] A method 500 for detecting one or more occluded areas of a scene 100 analyzed by an object tracking system, shown in Figure 5, may begin by providing S502 one or more video sequences representing the scene 100, e.g., the scene 100 in Figure 1. The one or more video sequences may include multiple tracked objects 110, 112, 114, 116. Thus, multiple object tracks are determined S504, each object track relating to a single object of the multiple objects 110, 112, 114, 116.

[0059] It should be noted that a scene may contain any number (1-n) of occluded areas. In the example of Figure 1, the scene 100 contains three occluded areas.

[0060] A map of occluded areas may be constructed S506, as described by the method of Figure 5. Figure 2 shows an example of such a map 200, according to some embodiments. Map 200 includes corresponding areas of the scene (marked by dashed lines in Figure 2) as areas 102a-n from Figure 1, although the areas in map 200 are referred to by specific numbers (202-222) for ease of explanation.

[0061] When the object tracking system marks a tracked object as lost, such as, for example, person 116 from Figure 1 when occluded by building 104, a re-identification algorithm may be run S510. The re-identification algorithm may be run on the same video sequence in which person 116 was last detected, or on another video sequence capturing the scene 100. The purpose of the re-identification algorithm is to attempt to resume the object track associated with person 116.

[0062] Re-identification algorithms aim to recognize the same object across different spatial or temporal gaps. These algorithms may vary widely based on the specific requirements of the application, the type of object being tracked, and the type of features used for identification. Examples include Euclidean distance-based matching or cosine similarity-based matching to measure the similarity between feature vectors. More complex methods, such as deep learning-based re-identification, may be used. For example, a convolutional neural network (CNN) may be trained to extract feature vectors from images of objects and perform re-identification.

[0063] When the object track is successfully resumed, the method 500 includes a step S512 of determining a first area of ​​the scene 100 of the multiple areas of the scene 100 in which the object track was lost and a second area of ​​the scene 100 of the multiple areas of the scene 100 in which the first object track is resumed. A connection between the first area and the second area of ​​the scene may then be added to the map of occluded areas such that the map identifies that the object track lost in the first area of ​​the scene has been resumed in the second area of ​​the scene.

[0064] For example, as shown by map 200 in Fig. 2, an object track associated with person 116 may be lost in a first area 214 (also called a start area) of the scene and resumed in a second area 216 (also called an end area) of the scene. The connection between the first area 214 and the second area 216 is shown as an arrow in Fig. 2. Thus, map 200 identifies that the object track lost in the first area 214 of the scene has resumed in the second area 216 of the scene. In other words, map 200 indicates that there may be an occluded area between the first area 214 and the second area 216 of the scene.

[0065] In some cases, the map 200 indicates that an object track lost in an area of ​​the scene has been resumed in a subset of the area of ​​the scene, the subset including at least two areas of the scene of the multiple areas of the scene. An example of this is shown in FIG. 2 for an occluded area resulting from a tree 106 in the scene 100. According to the map 200, an object track lost in the area 202 of the scene has been resumed in multiple areas 204, 206, 208 of the scene. For example, an object track associated with the jogger 112 may have been re-identified in the area 206 of the scene after being occluded by the tree 106. Another tracked object (not shown in FIG. 1) that was lost in the same area 202 of the scene may have been re-identified in the area 204 of the scene, and a third tracked object (not shown in FIG. 1) that was lost in the same area 202 of the scene may have been re-identified in the area 208 of the scene. Thus, constructing the map may include performing steps S512 and S514 of method 500 for multiple lost object tracks.

[0066] Map 200 further shows two other occluded areas, in that the connection between area 210 and area 212, as well as the connection between area 218 and multiple areas 220, 222, are included in the map.

[0067] FIG. 3 illustrates a map 300 of occluded areas shown according to some embodiments. The map 300 corresponds to a portion of the map 200 shown in FIG. 2 and thus illustrates the map 300 of occluded areas in the scene 100 shown in FIG. 1 using a different representation. In FIG. 3, the map 300 is represented by a table including columns for a start area 302 and an end area 304, and each row represents a connection between a start area and an end area of ​​a scene between which an occluded area is located. For example, the table 300 identifies that an object track lost in the start area 214 of the scene has been resumed in the end area 216 of the scene. The table further identifies a probability 306 that an object track lost in the start area 214 of the scene will be resumed in the end area 216 of the scene. In the example of FIG. 3, the table indicates that this probability is 75%. In other words, 75% of the object tracks lost in the area 214 of the scene have been resumed in the area 216 of the scene. The table 300 further comprises a representation of occluded areas resulting from the trees 106 of the scene 100. The table 300 comprises three rows for occluded areas resulting from the trees 106 of the scene 100. The first row represents a connection between the start area 202 and the end area 204 of the scene. The second row represents a connection between the start area 202 and the end area 206 of the scene. The third row represents a connection between the start area 202 and the end area 208 of the scene. Each of the three rows further comprises an associated probability. In other words, the table 300 indicates that an object track lost in the first area 202 of the scene is resumed in a subset of the areas 204, 206, 208 of the scene, the subset including at least two areas of the scene of the multiple areas of the scene, and the map further indicates, for each area 204, 206, 208 in the subset, a probability 306 that an object track lost in the first area 202 of the scene is resumed in that area in the subset.

[0068] 1-3 show an embodiment in which areas of the scene are pre-determined to be used to identify occluded areas. In another embodiment, the scene is iteratively divided into a number of areas of the scene. This embodiment is shown in FIG. 4, where for ease of illustration, the objects forming the occluded areas are also shown. In these embodiments, the scene is iteratively divided into a number of areas of the scene 401-411 based on positions 412, 414 in the scene where an object track of the number of object tracks is lost and resumed. This can be achieved by a clustering algorithm applied to each of the positions 412 indicative of positions in the scene where an object track of the number of object tracks is lost. In FIG. 4, these positions are represented by black circles. These positions 412 are clustered into five clusters, which are used to determine the areas of the scene 401, 402, 403, 404, 405, respectively. Similarly, a clustering algorithm is applied to each of the positions 414 indicative of positions in the scene where an object track of the number of object tracks is resumed. In FIG. 4, these positions are represented by white circles. These positions 414 are clustered into six clusters, which are used to determine the areas 406, 407, 408, 409, 410, 411 of the scene, respectively. The map of occluded areas therefore contains connections between the areas (not shown in FIG. 4) according to the following table: TIFF2025010520000002.tif47170

[0069] The probabilities in Table 1 assume that all object tracks lost in the starting areas 401-405 are indeed determined to be resumed in one of the ending areas 406-411. However, it should be noted that a lost object track may not be resumed due to the re-identification algorithm not succeeding in finding an object matching the object of the lost object track. The re-identification algorithm may fail, for example, because it was not trained / configured for all special cases. In other examples, the reason may be that the tracked object stayed in an occluded area, for example, entered a house. In these cases, the probability may be lower than 100% in some starting areas of the scene.

[0070] Location 412 and location 414 are clustered each time a new track is lost and / or resumed, resulting in areas 401-411 changing size and location, or new areas being added, or two areas being combined into one area. If such changes to the areas are made, the corresponding connections (and, optionally, probabilities) will change accordingly. For example, if an object track is lost between areas 405 and 404 in FIG. 4, these areas may be merged into one area in some cases. In another example, if an object track is resumed near area 409 in FIG. 4, area 409 may be split into two separate areas.

[0071] 1-4 show an embodiment of how a map of occluded areas may be constructed. The map of occluded areas may be constructed offline during the construction phase, using one or more video sequences capturing the scene. The map of occluded areas may also be constructed and refined in real time while the tracker system tracks objects in the scene, for example, when there are available computational resources for the tracking system to use.

[0072] In some embodiments, the map of occluded areas is not considered to be constructed using techniques described herein until a threshold number of object tracks have been lost and restarted. In these embodiments, the map of occluded areas may not be used to improve object tracking (as described below with respect to Figures 6-7) until a threshold number of objects have been lost and restarted.

[0073] In some embodiments, a connection between a first area and a second area of ​​a scene, such as where the map identifies an object track lost in the first area of ​​the scene as resumed in the second area of ​​the scene, may not be considered verified until a threshold number of object tracks lost in the first area of ​​the scene are actually resumed in that second area of ​​the scene. In these embodiments, the connection between the first area and the second area may not be used to improve object tracking (as described below with respect to Figures 6-7) until the connection is verified.

[0074] As explained above, the map of occluded areas can be used to improve object tracking. An example of how this can be achieved is now described with respect to Figures 6-7.

[0075] 6 shows, by way of example, a method 600 for improving coasting using a map of occluded areas. The map of occluded areas may be provided as described above, for example, as shown in method 500 of FIG. 5 and illustrated in FIGS. 1-4.

[0076] The method 600 includes a step S602 of providing sensor data capturing a scene. The sensor data may be video data, radar data, or lidar data.

[0077] The sensor data is used to track the object to determine the object track. For example, a second object may be tracked in the sensor data to determine the second object track. If the second object is occluded by another object, moves out of the field of view, or the system fails to detect the object due to changes in lighting conditions, noise, or other factors, the second object is therefore determined to be untrackable S604 because the tracking system has lost track of the second object. Thus, coasting may be required, as described above.

[0078] Coasting may be improved using the map of occluded areas. More specifically, the location in the scene of the latest observation of the second object in the second object track is determined S606. The location is compared to the location of the occluded area specified in the map of occluded areas. If the location is inside or close (within a threshold distance) to one of the starting areas (here called the first area) of the occluded area, the connection(s) between the starting area and other areas in the scene may be used to improve coasting. For example, the map of occluded areas may be used to identify an area of ​​the scene where an object track that was lost in the first area of ​​the scene is resumed S608. The area identified in S608 is thus the end area for the relevant occluded area (occluding the second object). Using the example of FIG. 2, if the latest observation of the tracked object is in or near area 214 of FIG. 2, the identified area is equal to area 216. When an area has been identified S608, a distance between the first area of ​​the scene and the identified area of ​​the scene may be determined S610. The distance may be determined, for example, using a coordinate system of the sensor data or by counting pixels between the areas in the representation of the scene. In other embodiments, the map of occluded areas includes a distance for each connection in the map. Such distances may have been determined during the construction stage, for example, using the field of view of a video camera that captures one or more video sequences to determine how far an area of ​​the scene is from another area of ​​the scene.

[0079] Using the distance determined in S610, the tracking system may estimate the size of the occluded area that currently obscures the second object. The tracking system may therefore set the coasting period based on the estimated size, such that a larger distance results in an equivalently longer coasting period S614. Advantageously, the coasting period may be extended or shortened (compared to a predefined global coasting period) based on the assessment, so that it targets the size of the relevant occluded area in the scene, which increases the likelihood that the coasting process will be successful. The coasting period may be set based on a predefined mapping function between the determined distance and the coasting period.

[0080] In some embodiments, the coasting period is further defined by determining a speed of the second object and setting a coasting period for the second track further based on the speed of the second object, where a higher speed results in an equivalently shorter coasting period. The coasting period may thus be determined by dividing the determined distance by the speed (possibly also adding a margin) to estimate a time period in which the second object will be occluded. Moreover, in some embodiments, the position of the second object may be predicted S616 based on the determined speed. For example, if a track has a position x0,y0 at time t and moves with a speed v(v_x,v_y), the position (x1,y1) of that object at time t+dt may be predicted as follows: x1=x0+v_x*dt Equation 1. y1=y0+v_y*dt Equation 2.

[0081] In some embodiments, the map indicates that an object track lost in a first area of ​​the scene has been resumed in a subset of the areas of the scene, the subset including at least two areas of the scene of the multiple areas of the scene. Examples of this are shown in Figures 2-4 and Table 1 for area 202 and area 401, respectively. According to some embodiments, in these cases, the area of ​​the scene from the subset with the greatest probability as indicated by the map may be selected.

[0082] In some embodiments, unless the probability of the identified area (i.e., the end area for the relevant occluded area) meets a threshold probability, the method 600 is aborted, such as without setting a coasting period 614.

[0083] In some embodiments, an estimated trajectory of the second object while it is occluded is also determined based on the map of occluded areas. In these embodiments, an angle between the first area and the identified area in the scene may be determined S612. The position of the second object while coasting may then be predicted further based on the angle S616. Advantageously, the historical data used to construct the map of occluded areas may indicate that an object occluded in an area of ​​the scene generally takes a trajectory when it is occluded, which may differ from the direction in which the object entered the occluded area. Using the techniques described herein, such behavior may be taken into account when estimating / predicting the position of the second object while coasting. In this embodiment, the speed of the second object before it is occluded may need to be mapped to the estimated trajectory.

[0084] The map of occluded areas may be used for other purposes besides improving coasting. For example, as shown in FIG. 7, the map of occluded areas may be used to improve the likelihood of a successful and efficient re-identification process. FIG. 7 shows, by way of example, a method 700 for improving re-identification using a map of occluded areas. The map of occluded areas may be provided as described above, for example, as shown in method 500 of FIG. 5 and illustrated in FIGS. 1-4. The method 700 further includes providing S702 sensor data capturing a scene. The sensor data may comprise video data. Objects may be tracked in the sensor data. For example, a third object may be tracked S704 in the sensor data to determine a third object track. When the third object is approaching a start area of ​​an occluded area in the scene, i.e., when the location of the third object is determined to be within a threshold distance from the first area of ​​the scene S706, the tracking system may prepare for the possibility that the third object is occluded. The preparation may include determining feature vectors for a third object for the purpose of object re-identification in an object tracking application S708. Real-time processing demands and the use of computationally intensive deep learning models for feature extraction from objects may lead to situations where available computational resources are insufficient to keep a database of feature vectors for all tracked objects in a scene to prepare for re-identification, especially when a large number of objects are detected in the scene. Computational resources may be saved by prioritizing areas in the scene where object tracks were previously lost. In other embodiments, a map of occluded objects may be used to determine which areas should not be prioritized for re-identification. As explained above, coasting may be improved for occluded areas of the map, so that re-identification in other areas of the scene may be prioritized, for example, areas that may be temporarily occluded by moving objects.

[0085] The above embodiments should be understood as illustrative examples of the present invention. Further embodiments of the present invention are envisioned. For example, the map of occluded areas may further be used to set up a fixed privacy mask over areas where objects generally become visible again after being occluded. In another example, the map of occluded areas may be used to guide a compression algorithm, for example, to compress the occluded areas less or more (depending on the requirements). Other exemplary uses of the map of occluded areas include changing noise filtering, increasing gain parameters, adjusting local tone mapping, etc., in the sensor data corresponding to the area of ​​the scene where the occluded object is estimated to reappear at the time the object is estimated to reappear. It should be understood that any feature described with respect to any one embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Moreover, equivalents and modifications not described above may also be employed without departing from the scope of the present invention, as defined in the appended claims.

Claims

1. 1. A computer-implemented method for detecting one or more occluded areas in one or more video sequences analyzed by an object tracking system, the method comprising: providing one or more video sequences, the one or more video sequences depicting a same scene, the one or more video sequences including multiple objects; determining a plurality of object tracks in the one or more video sequences; Including, The method further comprises constructing a map of one or more occluded areas in the one or more video sequences, the constructing the map comprising: a) upon determining that a first object track of the plurality of object tracks corresponding to a first object of the plurality of objects is lost, operating a re-identification algorithm on at least one video sequence of the one or more video sequences to attempt to resume the first object track; b) upon successfully resuming said first object track; determining a first area where the first object track is lost, the first area being determined in the one or more video sequences of a plurality of areas in the one or more video sequences, and a second area where the first object track is resumed, the second area being determined in the one or more video sequences of the plurality of areas in the one or more video sequences, wherein each area of ​​the plurality of areas in the one or more video sequences refers to an area of ​​a 2D representation of a 3D area of ​​the scene captured by the one or more video sequences, and the one or more video sequences are related to a base coordinate system to which objects and areas represented in image frames of each video sequence are transformed; adding to the map of one or more occluded areas a connection between the first area and the second area such that the map identifies that an object track lost in the first area has resumed in the second area; 4. A method comprising: constructing a map comprising:

2. The method of claim 1 , wherein constructing the map comprises performing steps a) and b) for a plurality of lost object tracks of the plurality of object tracks.

3. The method of claim 2 , wherein the map of one or more occluded areas further indicates a probability that an object track lost in the first area will be resumed in the second area.

4. 4. The method of claim 3, wherein the map indicates that an object track lost in the first area has been resumed in a subset of areas in the one or more video sequences, the subset including at least two areas in the one or more video sequences of the plurality of areas in the one or more video sequences, and the map further indicates, for each area in the subset, a probability that an object track lost in the first area has been resumed in that area in the subset.

5. providing sensor data capturing the scene; tracking a second object in the sensor data to determine a second object track; determining a location in the sensor data of a most recent observation of the second object in the second object track upon determining that the second object is not trackable in the sensor data; and determining that the location in the sensor data of the most recent observation of the second object in the second object track is within a threshold distance from the first area upon determining that the location is within a threshold distance from the first area. identifying from the map an area in the one or more video sequences where an object track that was lost in the first area has been resumed, and determining a distance between the first area and the identified area; setting a coasting period for the second object track based on the determined distance, a larger distance resulting in an equivalently longer coasting period; The method of claim 1 further comprising:

6. When dependent on claim 4, the step of identifying from the map an area in the one or more video sequences where an object track lost in the first area has been resumed comprises: selecting the area in the one or more video sequences from the subset having the greatest probability as indicated by the map; The method of claim 5 , comprising:

7. Setting the coasting period includes: determining a speed of the second object at the most recent observation of the second object in the second object track; setting the coasting period for the second track further based on the speed of the second object, a higher speed resulting in an equivalently shorter coasting period; The method of claim 5 , comprising:

8. determining that the second object is not trackable in the sensor data; predicting a position of the second object while it is not trackable in the sensor data based on the determined velocity. The method of claim 7.

9. determining that the second object is not trackable in the sensor data; determining an angle between the first area and the identified area in the one or more video sequences; predicting the position of the second object while it is not trackable in the sensor data further based on the angle; The method of claim 8 further comprising:

10. The method of claim 4 , wherein the sensor data is one of radar data, lidar data, or video data.

11. The method of claim 1 , wherein the multiple areas in the one or more video sequences include multiple predetermined areas in the one or more video sequences.

12. 2. The method of claim 1 , wherein the one or more video sequences are iteratively divided into the multiple areas in the one or more video sequences based on positions in the one or more video sequences where object tracks among the multiple object tracks are lost and resumed.

13. The step of iteratively dividing the one or more video sequences into the plurality of areas in the one or more video sequences comprises: determining a first plurality of locations, each location indicating a location in the one or more video sequences where an object track is lost among the plurality of object tracks; clustering the first plurality of locations into a first plurality of clusters; and for each cluster of the first plurality of clusters, determining an area in the one or more video sequences based on the location in the cluster; determining a second plurality of locations, each location indicating a location in the one or more video sequences where an object track of the plurality of object tracks is resumed; clustering the second plurality of locations into a second plurality of clusters; and for each cluster of the second plurality of clusters, determining an area in the one or more video sequences based on the location in the cluster; The method of claim 12, comprising:

14. providing sensor data capturing the scene; tracking a third object in the sensor data to determine a third object track; determining that a location of the third object in the sensor data is within a threshold distance from the first area in the one or more video sequences; determining a feature vector for the third object for object re-identification purposes in an object tracking application; The method of claim 1 further comprising:

15. A non-transitory computer-readable storage medium having stored thereon instructions for implementing the method of claim 1 when executed on a device having processing capabilities.

Images