Method for detecting one or more occluded areas of scene

Abstract

A method for detecting one or more occluded regions of a scene is presented. The method comprises constructing (S506) a mapping of one or more occluded regions in the scene. Construction of the mapping includes running (S510) a re-recognition algorithm on the video sequence to attempt to recover lost object trajectories. If the object trajectory is successfully restored, the method comprises determining (S512) an area in the scene in which the first object trajectory is lost and an area in the scene in which the first object trajectory is restored. A connection between the first region and the second region of the scene is added (S514) into the mapping such that the mapping identifies that the object trajectory lost in the first region of the scene is restored in the second region of the scene.

Description

Technical Field

[0001] The present invention relates to object tracking, and in particular to a computer-implemented method for detecting one or more occluded regions in a scene analyzed by an object tracking system. Background Art

[0002] Object tracking is an important and widely studied area in computer vision. The goal of object tracking is to keep track of an object (or multiple objects) in the sensor data capturing the scene.

[0003] In object tracking, occlusion poses significant challenges, for example, when the tracked object is hidden by static objects in the scene. For example, a tracking algorithm may lose a pedestrian who is temporarily occluded behind a tree or building.

[0004] When an object track is lost, a process called coasting can be used to try to recover the object track again. Coasting refers to the practice of predicting the position of an object based on its previous motion when the object is temporarily undetected (e.g., during occlusion or signal loss). It estimates the object's trajectory using techniques such as linear extrapolation or more complex motion models. During the lost period, coasting continues to estimate the object's track to maintain the assumed track. When a new object is detected near the assumed track, the object can be considered to be the same as the object that lost the track, and the track can be considered to be successfully reacquired and tracked and recovered. However, over time, coasting brings the risk of cumulative errors, especially if the object changes its speed or direction when it is not detected. For this reason, coasting of object tracks is usually performed only within a certain period of time (called the coasting period). If the object track is not successfully reacquired before the end of the coasting period, the coasting is considered to have failed, and the object track is considered to be lost and deleted.

[0005] If the freewheeling process fails, or generally when the object track is lost, a re-identification process can be used to attempt to recover the lost track. Re-identification refers to the process of identifying an object in a video sequence as the same object that was previously detected (e.g., detected before being occluded) in one or more previous frames of the video sequence or in a frame of another video sequence. When the object is re-identified, the lost track can be continued. Re-identification is performed by extracting meaningful features from the detected object. When an object needs to be re-identified (e.g., after the object is occluded or leaves the field of view), features are extracted from untracked objects in the scene. A similarity metric (such as Euclidean distance or cosine similarity) is then used to compare the newly detected feature vector with the feature vector previously extracted for the object in the lost track. If a match is detected, the lost track can be continued.

[0006] Re-identification generally requires a large amount of computing resources. Furthermore, re-identification is not feasible in all object tracking systems (eg, not in systems that track objects in data captured by radar devices).

[0007] Maintaining a map of occluded regions of a monitored scene can improve the freewheeling process and / or the re-identification process. However, correctly identifying occluded regions in a scene and using these to predict object behavior when occluded can be complex.

[0008] Therefore, improvements are needed in this context.

[0009] Hierarchical Databases for Multi-Camera Surveillance Systems (JAMES BLACK et al., Pattern Analysis and Applications, Springer-Verlag, LO, Vol. 7, No. 4, December 1, 2004, XP019381474) proposes a framework for event detection and video content analysis for visual surveillance applications.

[0010] Real-Time Traffic Monitoring with Occlusion Handling (FERNANDEZ-SANJURJO MAURO et al., Topics in Cryptography, CT-RSA 2020, 22 September 2019, XP047523419) A traffic monitoring system is proposed that combines detection and tracking in a way that can meet the requirements of real-time operation while being robust against occlusion. Summary of the invention

[0011] In view of the above, it would be beneficial to address or at least reduce one or several of the above discussed disadvantages, as set out in the accompanying independent patent claims.

[0012] According to a first aspect of the present invention, there is provided a computer-implemented method for detecting one or more occluded regions in one or more video sequences analyzed by an object tracking system, the method comprising: providing one or more video sequences, wherein the one or more video sequences depict the same scene, and the one or more video sequences include multiple objects; determining multiple object trajectories in the one or more video sequences; and constructing a map of the one or more occluded regions in the one or more video sequences.

[0013] Building a mapping includes steps a) and b):

[0014] a) upon determining that a first object track among the plurality of object tracks is lost, executing a re-identification algorithm on at least one video sequence of the one or more video sequences to attempt to recover the first object track, the first object track corresponding to the first object among the plurality of objects;

[0015] b) upon successfully recovering the first object trajectory: determining a first area where the first object trajectory is lost and a second area where the first object trajectory is recovered, wherein the first area is determined in one or more video sequences among a plurality of areas in the one or more video sequences, wherein the second area is determined in one or more video sequences among a plurality of areas in the one or more video sequences, wherein each of the plurality of areas in the one or more video sequences refers to an area represented in 2D as a 3D area of ​​a scene as captured by the one or more video sequences, wherein the one or more video sequences are associated with a reference coordinate system to which objects and areas depicted in image frames of each video sequence are transformed; and adding a connection between the first area and the second area to a mapping of one or more occluded areas such that the mapping identifies that the object trajectory lost in the first area is recovered in the second area.

[0016] In the following, for simplicity and ease of explanation, the expression "region in one or more video sequences depicting a scene" and similar expressions refer to a region of a scene captured by one or more video sequences in 2D which can be simplified to just "region of the scene" or similar expressions. It should be noted that in the case where several video sequences are used, if these video sequences are captured from, for example, different locations in the (real) scene, the coordinate system of one of the video sequences can generally be used as a reference coordinate system for the multiple video sequences, so that object detections, regions, etc. in the video sequences can be converted to the reference coordinate system. In the case where a fixed camera only produces one video sequence to capture the scene, each region in the scene can be represented by a specific pixel region in the image frame of the video sequence. In the case where a video sequence is captured by a video camera with a changing field of view, a similar reference coordinate system as described above can be determined and used to transform regions / detections from one image frame to another (e.g., using pan, tilt and roll, zoom parameters of a moving camera). Typically, a "region of a scene" as used herein is a representation of a real-world 3D region represented in 2D formed by a video sequence capturing the real-world scene.

[0017] In the context of this specification, the term "occluded region" should be understood as a region in one or more video sequences depicting a scene, in which an object tracker system tracking an object in the one or more video sequences has lost track of the object being tracked. In this document, the occluded region is referred to as the region between a first region where the track has been lost and a second region where the track has been recovered. Therefore, the occluded region is a region in a video sequence (multiple video sequences) that extends between a region where historical tracking data statistics indicate (or reasonably reliably predict, possibly after a freewheeling period) that the tracker can no longer track the object (i.e., the first region ("start region")) and another region where the re-identification algorithm has successfully matched the track of the lost object with the track of another object (i.e., the second region ("end region")). Therefore, as used in this document, the "mapping of occluded regions" refers to the mapping of connections between regions in the video sequence (i.e., the connections between the start region and one or more end regions).

[0018] Occluded areas may be the result of static objects that (in sensor data from a sensing or imaging system such as a camera or radar) prevent direct observation or detection of other objects behind them. This may also be the result of a more temporary occlusion of tracked objects in a real-world scene (such as a person being temporarily occluded by another person or car). Other examples of causes of occluded areas include being very dark in an area in one or more video sequences, or adding a privacy mask to an area in one or more video sequences such that objects in those areas may not receive detection of the object from the object tracking system.

[0019] Object tracking is the process of locating and following the position and movement of an object across a series of image frames in a video sequence. It includes detecting and identifying an object in an initial frame, and then continuously updating its position in subsequent frames by matching the object detection in subsequent frames with the object in the initial frame. In the context of this specification, the term "object track loss" or similar terms should be understood as an object tracker system that does not detect an object in a frame that can match the object track detected in a previous frame for some reason. In some embodiments, as further described below, after the coasting process fails, the object track can be considered to be lost. In other embodiments, for example, when coasting is not achieved or for other reasons, as long as the object tracker system does not detect an object in a frame that can match the object track from the previous frame, it can be considered that the object track has been lost. In some embodiments, when the object tracker system does not detect an object that can match the object track from the previous frame during a threshold number of frames, the object track is considered to be lost.

[0020] In the context of this specification, the term "recovering the first object track" or similar terms should be understood as successfully recovering (repairing, re-establishing, persisting, matching with the previously lost track, etc.) an object track that was previously lost (e.g., due to occlusion or object moving out of the frame) using a re-identification algorithm. The goal of re-identification is to correctly associate or match a given object across different video sequences or image frames, even in the case of time gaps between these observations. Thus, with the help of a re-identification algorithm, the initial object tracking process can be resumed by tracking the same object that has been re-identified after being lost for a period of time (e.g., due to occlusion by another object).

[0021] As described above, in object tracking applications, when the tracked object is occluded, freewheeling is often used to attempt to recover object tracking. The freewheeling process has a set freewheeling period that defines how long an assumed trajectory is maintained before the freewheeling process times out. After the freewheeling process times out, or due to other reasons as described above, the trajectory can be considered lost and re-identification can be used to attempt to recover the object trajectory. The re-identification algorithm can be run on subsequent image frames in the video sequence (such as the last observed position of the object in the lost object trajectory) or on other video sequences capturing the same scene. After successful re-identification, the object trajectory can be recovered by continuing to track the re-identified object.

[0022] The inventors have realized that successful re-identification can indicate the location and size of the occluded area in the monitored scene and the trace of the object when it is occluded. Specifically, using the location of the tracked object detected in the scene, the location / area where the object trajectory in the monitored scene is lost can be identified and registered (e.g., stored in a database structure, in a log file, or in a memory). When the lost trajectory is successfully recovered using the re-identification process, the location / area in the monitored scene where the trajectory is recovered (i.e., an object determined to match the object of the lost trajectory is detected) can be similarly registered. Advantageously, it can be determined that an object lost in a first area of ​​the scene is re-identified in a second area of ​​the scene. Therefore, it can be determined that there is an occluded area between the first area and the second area of ​​the scene. Therefore, a mapping of one or more occluded areas of the scene can be determined. In the following, for simplicity, the mapping of one or more occluded areas may be referred to as the mapping of the occluded areas.

[0023] The mapping of one or more occluded regions can include connections between regions of the monitored scene such that it identifies, for a particular region of the scene (the "start" of the occluded region), one or more other regions in the scene where object tracks lost in the particular region of the scene are recovered (the "end" of the occluded region). The mapping can be implemented using a table or similar data structure, where each region where an object track is lost is connected (e.g., via a shared ID or the like) to one or more regions where the object track is recovered. The connection can be implemented by creating a table of locations in one or more video sequences where object tracks were lost and successfully recovered. As will be further described below, these locations can be mapped to multiple regions in one or more video sequences (e.g., predetermined regions, or using a clustering algorithm). Each region can be represented by a center point or range (e.g., pixel coordinates or pixel regions). Using this, it can be determined whether an object is close to an area (e.g., just before an object tracker loses track of the object), which will be discussed further below.

[0024] By applying re-identification to all newly appearing objects (i.e., objects that are not currently being tracked), a spatial connection is established between where objects in the scene disappear and where the same objects later reappear in the scene. These disappearances are interpreted as occlusions in different parts of the scene and can then be used to modify the tracking system behavior in the occluded parts of the scene, making the tracking system scene-adaptive and improved.

[0025] Advantageously, a map of occluded areas can be maintained and updated. For example, a map can be defined "offline" using a video sequence (or multiple video sequences) capturing the monitored scene. Subsequently, the map of occluded areas can be used in real-time object tracking applications to improve object tracking, such as reducing the number of lost object tracks, reducing the number of erroneous object tracks, and / or reducing the computational resources required to achieve object tracking.

[0026] In some embodiments, constructing a mapping includes: performing steps a) and b) for multiple lost object tracks among multiple object tracks. Thus, the mapping of one or more occluded areas can identify multiple areas in the scene where one or more object tracks are lost. In addition, the mapping of the occluded areas can identify one or more areas in which the lost object (multiple objects) are recovered for each area in which one or more object tracks are lost. Advantageously, a more flexible mapping of one or more occluded areas can be determined to identify multiple occluded areas in the scene and / or to take into account that occluded objects may have different traces when occluded. In some embodiments, an area of ​​the scene is not determined as the beginning or end of an occluded area until at least a threshold number of object tracks are lost or recovered in this area. Advantageously, a more reliable mapping of the occluded area can be determined.

[0027] In some embodiments, the mapping of one or more occluded regions further indicates a probability that an object track lost in a first region is recovered in a second region. For example, if 5 of 10 object tracks lost in a first region of a scene are recovered in a second region of the scene, the probability for that connection in the mapping of occluded regions may be indicated as 50%.

[0028] In some embodiments, the mapping indicates that the object track lost in the first area is recovered in a subset of the areas in one or more video sequences, the subset including at least two areas in the one or more video sequences among a plurality of areas in the one or more video sequences, wherein the mapping further indicates, for each area in the subset, the probability that the object track lost in the first area is recovered in the area in the subset. In other words, if the object track lost in the first area of ​​the scene is determined to be recovered in, for example, the second and third areas of the scene (thus, a subset of the areas in the scene, which in this example includes two areas of the scene), the mapping can indicate the probability for each of these connections. For example, if 5 of the 10 object tracks lost in the first area of ​​the scene are recovered in the second area of ​​the scene, the probability for the connection (the connection between the first area and the second area of ​​the scene) in the mapping of the occluded area can be indicated as 50%. 3 of the remaining 5 lost tracks can be recovered in the third area of ​​the scene, so that the probability for the connection (the connection between the first area and the third area of ​​the scene) in the mapping of the occluded area is indicated as 30%. The remaining 2 lost tracks may have been either not successfully recovered using a re-identification process, for example, or were determined to be recovered in yet another area of ​​the scene.

[0029] In an example, the method further includes: providing sensor data of a captured scene; tracking a second object in the sensor data to determine a second object trajectory; and when it is determined that the second object is not trackable in the sensor data, determining a position in the sensor data where the second object was last observed in the second object trajectory, and when it is determined that the position in the second object trajectory where the second object was last observed is within a threshold distance from the first area: identifying an area in one or more video sequences in which object trajectories lost in the first area are recovered based on the mapping, and determining a distance between the first area and the identified area; and setting a coasting period for the second object trajectory based on the determined distance, wherein a larger distance results in a relatively longer coasting period.

[0030] Advantageously, the mapping of occluded areas can be used to configure a freewheeling period / freewheeling time based on the position of an object of an object trajectory in the scene when freewheeling is initiated for that object trajectory. When freewheeling is initiated, the object is considered untrackable in the sensor data, for example because the object is being occluded. When an object is occluded or otherwise undetectable in a current image frame of a video sequence (or similar other types of sensor data), no object is detected in the current image frame that can be matched to an existing object trajectory of the object (i.e., as determined for a previous image frame). In other words, when freewheeling is initiated for a second object trajectory, the second object is considered untrackable.

[0031] Typically, the freewheeling period is predetermined for the object tracking system. However, based on the size of the occluded area, the freewheeling period may need to be extended in certain areas of the scene, while it may need to be shortened for other areas. In this example, when a second object is detected in or near a first area of ​​the scene before freewheeling is initiated, the freewheeling period can be configured based on an estimated distance between the beginning (the first area) and the end (determined based on a mapping of the occluded area) of the occluded area that the second object in the scene enters (and is therefore untrackable in the sensor data). For the second object (which has a certain speed before freewheeling begins), a longer distance causes a longer freewheeling period than a shorter distance.

[0032] Advantageously, this can improve object tracking and reduce the number of lost tracks, which in turn reduces the need for computationally demanding re-identification processing. In addition, as described above, re-identification may not be applicable to all types of sensor data. For example, re-identification cannot be performed for radar (radio detection and ranging) or lidar (light detection and ranging) data. Using the techniques discussed herein, video data of a captured scene can be used to define a mapping of occluded areas, while the mapping of occluded areas described herein can be used to improve object tracking using radar data. In this embodiment, the mapping of occluded areas is converted to radar coordinates. If the sensor data includes a video sequence captured from another location, or otherwise has a different coordinate system than the mapping of the occluded areas, a transformation needs to be performed to map positions, etc. from the video sequence to the mapping of the occluded areas, as further described above.

[0033] Furthermore, by reducing the freewheeling time for certain areas of the scene, the removal of tracks that are no longer relevant can be improved, which in turn reduces calculations and reduces the risk of freewheeling producing erroneous results.

[0034] In an example, the step of identifying from the mapping an area in the one or more video sequences where object trajectories lost in the first area are recovered comprises selecting from the subset an area in the one or more video sequences indicated by the mapping having a maximum probability.

[0035] Thus, where a mapping of one or more occluded regions identifies more than one region of a scene in which a lost object trajectory in a first region of the scene is recovered, the most likely region may be selected from a subset of the regions and used to determine the freewheeling period. Alternatively or additionally, when selecting which of the subsets of regions will be used to determine the freewheeling period, the angle of motion that the second object had when it was last observed in the second object trajectory (before the freewheeling process began) may be used, so that the region in the subset of regions that has the most similar angle relative to the first region may be prioritized (e.g., increased probability) in the selection process.

[0036] In some embodiments, setting the freewheeling period includes: determining a speed of the second object when the second object was last observed in the second object trajectory; and setting the freewheeling period for the second trajectory further based on the speed of the second object, wherein a higher speed results in a relatively shorter freewheeling period.

[0037] Therefore, the estimated distance from the start to the end of the occluded region (ie, the estimated size of the occluded region) and the speed of the second object are used to determine the freewheeling period, thereby more accurately estimating how long the second object may be occluded.

[0038] In some embodiments, the techniques described herein may also be used to improve the estimated track and position of the second object during the occlusion. For example, in some embodiments, when it is determined that the second object is not trackable in the sensor data: the method includes: predicting the position of the second object when it is not trackable in the sensor data based on the determined speed. Thus, for example, the time point when the second object is considered to be untrackable plus the determined speed may result in the assumed track of the second object being more accurate, thereby resulting in improved freewheeling processing.

[0039] In some examples, the method further includes: when it is determined that the second object is not trackable in the sensor data: determining an angle between the first area and the identified area in one or more video sequences; and further predicting a position of the second object when it is not trackable in the sensor data based on the angle.

[0040] Advantageously, based on the behavior of previous objects occluded in the same area of ​​the scene, a hypothesized trajectory of a second object may be more accurately estimated when the second object is occluded because the position of the second object in the hypothesized trajectory may be predicted as described herein.

[0041] According to some embodiments, the sensor data is one of radar data, lidar data, or video data. As previously discussed, the techniques described herein can be used to improve object tracking in radar / lidar data for which re-identification algorithms are not applicable. In addition, for radar data, lidar data, and video data, mapping of occluded areas can be employed to improve freewheeling processing, as described herein.

[0042] According to some embodiments, the multiple regions in one or more video sequences include multiple predetermined regions in one or more video sequences. For example, the scene can be divided into multiple regions based on the format and resolution of the sensor data that captured the scene. In some examples, each region corresponds to a specific area of ​​pixels in an image frame of the video sequence that captures the scene (X×Y pixels in size). In some examples, each region corresponds to a specific physical size of the scene, such as Z×W meters (or other units of size) represented by one or more video sequences.

[0043] In some examples, one or more video sequences are iteratively divided into multiple regions in one or more video sequences based on the locations where object trajectories among multiple object trajectories in the one or more video sequences are lost and recovered. For example, a configuration phase (offline or online) may include a first phase and a second phase, in which objects are tracked to determine the locations where object trajectories in the scene are lost and recovered to determine the regions in the scene to be used in the second phase, in which a mapping of occluded regions is determined as described above (based on multiple regions of the scene). In other embodiments, the first phase and the second phase are implemented simultaneously so that when the region of the scene changes based on the collected statistical information about the lost and recovered (re-identified) trajectories, the mapping of the occluded regions is updated accordingly.

[0044] In some embodiments, the step of iteratively dividing one or more video sequences into multiple regions in the one or more video sequences includes: determining a first plurality of positions, each position indicating a position where an object track among multiple object tracks in the one or more video sequences is lost, clustering the first plurality of positions into a first plurality of clusters, and for each cluster in the first plurality of clusters, determining a region in the one or more video sequences based on the positions in the cluster; and determining a second plurality of positions, each position indicating a position where an object track among multiple object tracks in the one or more video sequences is restored, clustering the second plurality of positions into a second plurality of clusters, and for each cluster in the second plurality of clusters, determining a region in the one or more video sequences based on the positions in the cluster.

[0045] Depending on the requirements of the object tracking application, any suitable clustering algorithm may be used. Examples of clustering algorithms include OPTICS (Order Points to Identify Cluster Structure), Mean Shift, and Density-Based Spatial Clustering of Applications with Noise (DBSCAN).

[0046] As shown above, mapping of occluded areas can have many uses in object detection or tracking applications.

[0047] In one embodiment, the method includes the steps of providing sensor data capturing a scene; tracking a third object in the sensor data to determine a third object trajectory; determining that the location of the third object in the sensor data is within a threshold distance from a first region in one or more video sequences; and determining a feature vector of the third object for use in object re-identification in an object tracking application. Advantageously, the method can provide low complexity and efficient selection criteria against which objects can be analyzed to determine feature vectors for object re-identification. By giving priority to areas in the scene before object trajectories are lost, objects that may be occluded in subsequent image frames can be prioritized for feature extraction. This method prioritizes objects that the tracking system may lose track of in subsequent image frames.

[0048] According to a second aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium having stored thereon instructions which, when executed on a device having processing capabilities, are used to implement the method according to the first aspect.

[0049] According to a third aspect of the invention, the above-mentioned purpose is achieved by a system for detecting one or more occluded areas of a scene analyzed by an object tracking system, the system comprising: one or more processors; and one or more non-temporary computer-readable media storing computer-executable instructions, which, when executed by one or more processors, cause the system to perform actions according to the first aspect.

[0050] The second and third aspects may generally have the same features and advantages as the first aspect. It is further noted that, unless explicitly stated otherwise, the present disclosure relates to all possible combinations of features. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and additional objects, features and advantages of the present invention will be better understood through an illustrative and non-limiting detailed description of embodiments of the present disclosure with reference to the accompanying drawings, in which like reference numerals will be used for similar elements, wherein:

[0052] Figure 1 illustrates a scene including multiple objects and multiple occluded areas according to an embodiment;

[0053] Figure 2 According to the embodiment, Figure 1 A visual representation of a map of occluded areas determined by the scene shown in;

[0054] Figure 3 According to other embodiments, Figure 1 A visual representation of a portion of a map of occluded areas determined by the scene shown in;

[0055] Figure 4 The third embodiment is shown Figure 1 A visual representation of a map of occluded areas determined by the scene shown in;

[0056] Figure 5 A method for detecting one or more occluded regions of a scene analyzed by an object tracking system according to an embodiment is shown;

[0057] Figure 6 Show a use such as using Figure 5 A method for determining freewheeling parameters by mapping the occluded area detected by the method;

[0058] Figure 7 A method for using the object tracking method in an object tracking application is shown. Figure 5 A method that uses a map of occluded areas detected by the method to prioritize areas in the scene for object re-identification. DETAILED DESCRIPTION

[0059] Occlusion poses a significant challenge in object tracking because it results in a temporary blockage of the target object from the perspective of the tracking system. During periods of occlusion, the lack of visual information introduces uncertainty into the tracking process. Furthermore, any sudden changes in the visible portion of the object can cause confusion to the tracking system, especially when visual features are heavily relied upon to identify and track the object. Additional difficulties can arise if the object changes its state (such as speed or direction) during occlusion. This can make predicting the reappearance of the object particularly challenging, complicating the process of re-identification.

[0060] Object tracking systems can use a variety of techniques to identify and keep track of occluded areas in the monitored scene. For example, using depth information from stereo cameras, lidar, or depth sensors, a tracking system can identify when an object moves behind another object, indicating potential occlusion. However, this technique depends on the availability and accuracy of depth information, which may not always be reliable or available. Other techniques include machine learning models that can be trained to predict occlusions based on patterns and features in the input data. These models can be difficult to implement and require large amounts of training data.

[0061] The inventors have realised that using re-identification techniques, which are an existing component in many object tracking systems, a map of occluded areas can be constructed and used to improve object tracking. This technique will now be combined with Figures 1 to 4 described and by Figure 5 This is supported by the flowchart presented in the paper.

[0062] Figure 1 A scene 100 analyzed by an object tracking system is shown. The scene includes three static objects 104, 106, 107 that form occluded regions. For example, a person 112 jogging within the scene 100 may temporarily disappear behind a tree 106. Similarly, another person 114 walking within the same scene 100 may momentarily disappear behind a playground building 107. A car 110 moving along a road 109 may become occluded by a building 104, and another pedestrian 116 walking near the road 109 may also be occluded by the same building 104. Depending on the size and shape of the occluding objects 104, 106, 107, the speed at which the objects 110, 112, 114, 116 are moving, and the orientation of these objects 110, 112, 114, 116 when they are occluded, all of which affect how long the moving objects 110, 112, 114, 116 are hidden from the perspective of the tracking system analyzing the scene 100.

[0063] As described above, when the object track is lost, a process called freewheeling can be used to try to recover the object track again. During the lost period, freewheeling continues to estimate the object's track, using, for example, a Kalman filter to maintain the assumed track. Over time, freewheeling brings the risk of cumulative errors, especially if the object changes its speed or direction when it is not detected. In addition, freewheeling is usually performed for a specific duration, referred to as a freewheeling period or freewheeling time in this article. During the freewheeling period, the system uses prior information and motion models to estimate the current state of the object. Once the freewheeling period has passed, if the object is not re-detected, the tracking system usually marks the track as lost. After the track is lost, a process called re-identification can be performed as described above and further described below. In other embodiments, for example, when freewheeling is not achieved or due to other reasons as described above, when the object tracker is no longer able to track the object of the object track, the track will be considered (determined to be) lost, as further described above, and re-identification can be performed.

[0064] Figure 1 The scene 100 shown in FIG. 1 is divided into a plurality of predetermined regions 102a to 102n, represented by dashed lines. For example, each such region corresponds to a pixel region of an image frame of a video sequence capturing the scene 100. These regions 102a to 102n can be used to identify occluded regions in the scene. Other ways of dividing the scene 100 into a plurality of regions can be adopted, for example, as described below in conjunction with Figure 4 described.

[0065] like Figure 5The method 500 for detecting one or more occluded regions of a scene 100 analyzed by an object tracking system may be performed by providing S502 a scene 100 (eg, Figure 1 The one or more video sequences of a scene 100 in FIG. 1 may include a plurality of tracked objects 110, 112, 114, 116. Therefore, a plurality of object tracks are determined S504, each object track being associated with a single object among the plurality of objects 110, 112, 114, 116.

[0066] It should be noted that a scene can include any number (1 to n) of occluded regions. Figure 1 In the example of FIG. 1 , the scene 100 includes three occluded regions.

[0067] If through Figure 5 As described in the method, a map of the occluded area can be constructed in S506. Figure 2 An example of such a map 200 according to some embodiments is shown. The map 200 includes Figure 1 The scene area corresponding to the areas 102a to 102n in Figure 2 ), but for ease of explanation, the regions in map 200 are represented by specific numbers (202 to 222).

[0068] When the object tracking system marks the tracked object as lost S508, such as for example Figure 1 When a person 116 is occluded by a building 104, a re-identification algorithm may be run S510. The re-identification algorithm may be run on the same video sequence as the video sequence in which the person 116 was last detected or on another video sequence capturing the scene 100. The purpose of the re-identification algorithm is to attempt to recover the object track associated with the person 116.

[0069] Re-identification algorithms aim to identify the same object across different spatial or temporal gaps. These algorithms can vary widely based on the specific requirements of the application, the type of object being tracked, and the type of features used for identification. Examples include matching based on Euclidean distance or matching based on cosine similarity to measure the similarity between feature vectors. More sophisticated methods such as deep learning-based re-identification can be used. For example, convolutional neural networks (CNNs) can be trained to extract feature vectors from images of objects and perform re-identification.

[0070] When the object trajectory is successfully recovered, the method 500 includes the step of determining S512 a first area of ​​the scene 100 where the object trajectory is lost among the multiple areas of the scene 100 and a second area of ​​the scene 100 where the first object trajectory is recovered among the multiple areas of the scene 100. Then, a connection between the first area and the second area of ​​the scene can be added S514 to the map of occluded areas, such that the map identifies that the object trajectory lost in the first area of ​​the scene is recovered in the second area of ​​the scene.

[0071] For example, Figure 2 As indicated by the mapping 200 in FIG. 1 , the object track associated with the person 116 may be lost at a first region 214 (also referred to as the start region) of the scene and recovered at a second region 216 (also referred to as the end region) of the scene. The connection between the first region 214 and the second region 216 is shown in FIG. Figure 2 Thus, mapping 200 identifies that object trajectories lost in first region 214 of the scene are recovered in second region 216 of the scene. In other words, mapping 200 indicates that there may be an occluded region between first region 214 and second region 216 of the scene.

[0072] In some cases, mapping 200 indicates that object tracks lost in one area of ​​a scene are recovered in a subset of areas of the scene, the subset including at least two areas of the scene among a plurality of areas of the scene. Figure 2 100 for an occluded region created by tree 106 in scene 100. According to mapping 200, object tracks lost in region 202 of the scene are recovered in multiple regions 204, 206, 208 of the scene. For example, an object track associated with jogger 112 can be re-identified in region 206 of the scene after being occluded by tree 106. Another tracked object ( Figure 1 ) can be re-identified in region 204 of the scene, while a third tracked object ( Figure 1 208) can be re-identified in the region 208 of the scene. Accordingly, building a map may include performing steps S512 and S514 of method 500 for a plurality of lost object tracks.

[0073] The map 200 further indicates two additional occluded regions, as the connection between the region 210 and the region 212 and the connection between the region 218 and the plurality of regions 220 , 222 are also included in the map.

[0074] Figure 3 FIG. 3 shows a map 300 of occluded areas according to some embodiments. The map 300 corresponds to Figure 2 , and therefore, another expression is used to illustrate Figure 1 A map 300 of occluded areas in the scene 100 shown in FIG. Figure 3 In , a mapping 300 is represented by a table including columns for a start region 302 and an end region 304, wherein each row represents a connection between a start region and an end region of a scene, between which the occluded region is located. For example, the table 300 identifies that an object track lost in the start region 214 of the scene is recovered in the end region 216 of the scene. The table further indicates a probability 306 that an object track lost in the start region 214 of the scene is recovered in the end region 216 of the scene. Figure 3 In the example of , the table indicates that this probability is 75%. In other words, 75% of the object tracks lost in area 214 of the scene are recovered in area 216 of the scene. Table 300 further includes an expression for the occluded areas generated by the trees 106 of the scene 100. Table 300 includes three rows associated with the occluded areas generated by the trees 106 of the scene 100. The first row represents the connection between the start area 202 and the end area 204 of the scene. The second row represents the connection between the start area 202 and the end area 206 of the scene. The third row represents the connection between the start area 202 and the end area 208 of the scene. Each of the three rows further includes an associated probability. In other words, table 300 indicates that an object trajectory lost in the first area 202 of the scene is recovered in a subset of areas 204, 206, 208 of the scene, the subset including at least two areas of the scene among multiple areas of the scene, wherein, for each area 204, 206, 208 in the subset, the mapping further indicates a probability 306 that the object trajectory lost in the first area 202 of the scene is recovered in an area in the subset.

[0075] Figures 1 to 3 An embodiment is shown in which the region of the scene used to identify the occluded region is predetermined. In other embodiments, the scene is iteratively divided into multiple regions of the scene. This embodiment is Figure 4 As shown in Figure 4 In the figure, for ease of explanation, objects forming occluded areas are also shown. In these embodiments, the scene is iteratively divided into multiple regions 401 to 411 of the scene based on the positions 412, 414 in the scene where the object track among the multiple object tracks is lost and recovered. This can be achieved by applying a clustering algorithm to each position 412, which indicates the position where the object track among the multiple object tracks in the scene is lost. Figure 4In , these positions are represented by filled circles. These positions 412 are clustered into 5 clusters, each cluster being used to determine a region 401, 402, 403, 404, 405 of the scene. Similarly, a clustering algorithm is applied to each position 414 indicating the position where an object track among a plurality of object tracks in the scene is recovered. Figure 4 In , these positions are represented by unfilled circles. These positions 414 are clustered into 6 clusters, each cluster is used to determine the region 406, 407, 408, 409, 410, 411 of the scene. Therefore, according to the following table, the mapping of the occluded area includes the connection between the regions ( Figure 4 not shown).

[0076]

[0077]

[0078] Table 1

[0079] The probabilities in Table 1 assume that all object tracks lost in the starting regions 401 to 405 are reliably determined to be recovered in any of the ending regions 406 to 411. However, it should be noted that lost object tracks are sometimes not recovered because the re-identification algorithm did not successfully find any objects that match the objects of the lost object tracks. For example, the re-identification algorithm may fail because it is not trained / configured for all edge cases. In other examples, the reason may be that the tracked object stays in an occluded area, such as entering a house. In these cases, the probability may be less than 100% for a certain starting area of ​​the scene.

[0080] Position 412 and position 414 may be clustered each time a new track is lost and / or recovered, causing regions 401 to 411 to change size and position, or causing a new region to be added or two regions to be combined into one region. If such changes are made to the regions, the corresponding connections (and optionally, probabilities) will change accordingly. For example, if an object track is Figure 4 In another example, if the object track is lost between region 405 and region 404, then in some cases, these regions may be merged into one region. Figure 4 If the vicinity of region 409 in the image is restored, region 409 can be divided into two independent regions.

[0081] Figures 1 to 4An embodiment of how to construct a map of occluded areas is shown. During the configuration phase, the map of occluded areas can be constructed offline using one or more video sequences that capture the scene. For example, when there are available computing resources for the tracking system to use, the tracker system can also build and adjust the map of occluded areas in real time while tracking objects in the scene.

[0082] In some embodiments, a map of occluded areas is not considered constructed until a threshold number of object tracks are lost and recovered using the techniques described herein. In these embodiments, a map of occluded areas may not be used to improve object tracking (as described below in conjunction with Figure 6 to Figure 7 described), until a threshold number of objects are lost and recovered.

[0083] In some embodiments, a connection between a first area and a second area of ​​a scene (which causes a mapping to identify object tracks lost in the first area of ​​the scene as recovered in the second area of ​​the scene) may not be considered verified until a threshold number of object tracks lost in the first area of ​​the scene are definitively recovered in the second area of ​​the scene. In these embodiments, the connection between the first area and the second area may not be used to improve object tracking (as described below in conjunction with Figure 6 to Figure 7 described) until the connection is verified.

[0084] As described above, mapping of occluded areas can be used to improve object tracking. Figure 6 to Figure 7 to describe an example of how to accomplish this.

[0085] Figure 6 The method 600 of improving freewheeling using a map of an occluded area is shown by way of example. The map of the occluded area is provided as described above, for example, as Figure 5 As shown in method 500 of Figures 1 to 4 exemplified in .

[0086] The method 600 comprises a step of providing S602 sensor data of capturing a scene. The sensor data may be video data, radar data or lidar data.

[0087] The sensor data is used to track the object to determine the object trajectory. For example, a second object may be tracked in the sensor data to determine the second object trajectory. If the second object becomes obscured by another object, moves out of the field of view, or the system fails to detect the object due to changes in lighting conditions, noise, or other factors, then it is determined S604 that the second object is not trackable because the tracking system is losing track of the second object. As described above, it may be necessary to perform a freewheel.

[0088] Using a mapping of occluded areas can improve freewheeling. More specifically, the position in the scene when the second object was last observed in the second object trajectory is determined S606. The position is compared to the position of the occluded area as specified in the mapping of the occluded area. If the position is inside or close to (close to within a threshold distance) one of the starting areas of the occluded area (referred to herein as the first area), then the connection between the starting area and other areas in the scene can be used to improve freewheeling. For example, using the mapping of occluded areas, an area in the scene where the object trajectory lost in the first area of ​​the scene is recovered can be identified S608. Therefore, the area identified S608 is the end area of ​​the relevant occluded area (which occludes the second object). Using Figure 2 For example, if the tracked object was last observed at Figure 2 If the area is inside or near area 214, the identified area is equal to area 216. When the area is identified S608, the distance between the first area of ​​the scene and the identified area of ​​the scene can be determined S610. The distance can be determined using, for example, a coordinate system of sensor data or by counting pixels between areas in a representation of the scene. In other embodiments, the map of occluded areas includes a distance for each connection in the map. Such distances can be determined during a configuration phase, such as using the field of view of a camera used to capture one or more video sequences to determine how far one area of ​​the scene is from another area of ​​the scene.

[0089] Using the determined S610 distance, the tracking system may estimate the size of the occluded area that currently hides the second object. Accordingly, the tracking system may set S614 a freewheeling period based on the estimated size such that a larger distance results in a relatively longer freewheeling period. Advantageously, the freewheeling period may be extended or shortened (compared to a predefined global freewheeling period) based on the evaluation such that it targets the size of the relevant occluded area in the scene to increase the likelihood of a successful freewheeling process. The freewheeling period may be set based on a predetermined mapping function between the determined distance and the freewheeling period.

[0090] In some embodiments, the freewheeling period is further defined by determining the speed of the second object; and the freewheeling period for the second trajectory is further set based on the speed of the second object, wherein a higher speed results in a relatively shorter freewheeling period. Therefore, the freewheeling period can be determined by dividing the determined distance by the speed (a margin can also be added) to estimate the time period that the second object will be obscured. In addition, in some embodiments, the position of the second object can be predicted S616 based on the determined speed. For example, if the trajectory has a position x0, y0 at time t and moves at a speed v (v_x, v_y), the position (x1, y1) of the object at time t+dt can be predicted as:

[0091] X1=x0+v_x×dt Equation 1

[0092] y1=y0+v_y×dt Equation 2

[0093] In some embodiments, the mapping indicates that object tracks lost in a first area of ​​the scene are recovered in a subset of areas of the scene, the subset comprising at least two areas of the scene among a plurality of areas of the scene. Figures 2 to 4 and are shown in Table 1, respectively for regions 202 and 401. In these cases, according to some embodiments, the region of the scene with the greatest probability indicated by the mapping may be selected from the subset.

[0094] In some embodiments, unless the probability of the identified region (ie, the ending region for the associated occluded region) satisfies the threshold probability, method 600 aborts without performing setting 614 the freewheeling period, etc.

[0095] In some embodiments, the estimated track of the second object when occluded is also determined based on the mapping of the occluded area. In these embodiments, the angle between the first area in the scene and the identified area can be determined S612. The position of the second object during the freewheeling can then be further predicted S616 based on the angle. Advantageously, historical data used to configure the mapping of the occluded area can indicate that objects that are occluded at a particular area of ​​the scene typically take a specific track when occluded, which may be different from the direction of the object when it enters the occluded area. Using the techniques described herein, this behavior can be taken into account when estimating / predicting the position of the second object during freewheeling. In this embodiment, the speed of the second object before being occluded may need to be mapped to the estimated track.

[0096] Mapping of occluded areas can be used for other purposes besides improving freewheeling. Figure 7 As shown in , mapping of occluded areas can be used to improve the chances of achieving a successful and efficient re-identification process. Figure 7 The method 700 of using the mapping of occluded areas to improve re-identification is shown by way of example. The mapping of the occluded areas is provided as described above, for example, Figure 5 As shown in method 500 and as Figures 1 to 4 As illustrated in . Method 700 further includes providing S702 sensor data of capturing a scene. The sensor data may include video data. Objects may be tracked in the sensor data. For example, a third object may be tracked S704 in the sensor data to determine a third object trajectory. When the third object gets closer and closer to the starting area of ​​the occluded area in the scene, that is, when it is determined S706 that the position of the third object is within a threshold distance from the first area of ​​the scene, the tracking system may prepare for the possibility that the third object will be occluded. The preparation may include determining S708 a feature vector of the third object for object re-identification in an object tracking application. Real-time processing requirements and the use of computationally intensive deep learning models for extracting features from objects may result in situations where available computing resources are insufficient to maintain a database of feature vectors of all tracked objects in the scene in preparation for re-identification, particularly when a large number of objects are detected in the scene. Computing resources may be saved by prioritizing areas in the scene where object trajectories were previously lost. In other embodiments, a mapping of occluded objects may be used to determine which areas are not prioritized for re-identification. Since freewheeling may be improved for occluded areas in the map as described above, re-recognition may be prioritized in other areas of the scene, such as areas that may be temporarily occluded by moving objects.

[0097] The above embodiments should be understood as illustrative examples of the present invention. Further embodiments of the present invention are contemplated. For example, the mapping of occluded areas can be further used to set a fixed privacy mask on an area in which an object is usually visible again after being occluded. In another example, the mapping of occluded areas can be used to guide a compression algorithm, for example, to compress the occluded area less or more (depending on the requirements). Other exemplary uses of the mapping of occluded areas include: in the sensor data corresponding to the area where the occluded object of the scene is estimated to reappear, when the object is estimated to reappear, changing the noise filter, increasing the gain parameter, adjusting the local tone mapping, etc. It should be understood that any feature described in any embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other embodiment or any combination of any other embodiment. In addition, equivalents and modifications not described above may also be adopted without departing from the scope of the present invention, which is defined in the appended claims.

Claims

1. A computer-implemented method for detecting one or more occluded regions in one or more video sequences analyzed by an object tracking system, the method comprising: providing one or more video sequences, wherein the one or more video sequences depict a same scene, the one or more video sequences comprising a plurality of objects; determining a plurality of object tracks in the one or more video sequences; Characterized in that the method comprises: Constructing a map of one or more occluded regions in the one or more video sequences, wherein constructing the map comprises: a) upon determining that a first object track among the plurality of object tracks is lost, executing a re-identification algorithm on at least one of the one or more video sequences to attempt to recover the first object track, the first object track corresponding to the first object among the plurality of objects; b) upon successfully recovering the first object trajectory: determining a first region where the first object trajectory is lost and a second region where the first object trajectory is recovered, wherein the first region is determined in the one or more video sequences among a plurality of regions in the one or more video sequences, wherein the second region is determined in the one or more video sequences among the plurality of regions in the one or more video sequences, wherein each of the plurality of regions in the one or more video sequences refers to a region of a 2D representation of a 3D region of the scene captured by the one or more video sequences, wherein the one or more video sequences are associated with a reference coordinate system to which objects and regions depicted in image frames of each video sequence are transformed; and A connection between the first region and the second region is added to the map of the one or more occluded regions such that the map identifies object tracks lost in the first region that are recovered in the second region.

2. The method according to claim 1, wherein: Constructing the mapping includes: Step a) and step b) are performed on a plurality of lost object trajectories among the plurality of object trajectories.

3. The method according to claim 2, wherein: The mapping of the one or more occluded regions further indicates a probability that an object track lost in the first region is recovered in the second region.

4. The method according to claim 3, wherein: The mapping indicates that object trajectories lost in the first region are recovered in a subset of regions in the one or more video sequences, the subset comprising at least two regions in the one or more video sequences among the multiple regions in the one or more video sequences, wherein the mapping further indicates, for each region in the subset, a probability that an object trajectory lost in the first region is recovered in the region in the subset.

5. The method according to claim 1, further comprising the steps of: providing sensor data capturing the scene; tracking a second object in the sensor data to determine a second object trajectory; and When it is determined that the second object is not trackable in the sensor data, determining a last observed position of the second object in the second object trajectory in the sensor data, and when it is determined that the last observed position of the second object in the second object trajectory is within a threshold distance from the first area: identifying, based on the mapping, an area in the one or more video sequences where object trajectories lost in the first area are recovered, and determining a distance between the first area and the identified area; and A freewheeling period is set for the second object trajectory based on the determined distance, wherein a greater distance results in a relatively longer freewheeling period.

6. The method according to claim 5, wherein: The mapping indicates that an object track lost in the first region is recovered in a subset of regions in the one or more video sequences, the subset comprising at least two regions in the one or more video sequences among the plurality of regions in the one or more video sequences, wherein the mapping further indicates, for each region in the subset, a probability that an object track lost in the first region is recovered in the region in the subset, The step of identifying, according to the mapping, an area in the one or more video sequences where object trajectories lost in the first area are restored comprises: The region in the one or more video sequences indicated by the mapping having the greatest probability is selected from the subset.

7. The method according to claim 5, wherein: Setting the coasting period includes: determining a velocity of the second object when the second object was last observed in the second object trajectory; and The freewheeling period for the second trajectory is set further based on the speed of the second object, wherein higher speeds result in relatively shorter freewheeling periods.

8. The method according to claim 7, wherein: When it is determined that the second object is not trackable in the sensor data, a position of the second object when it is not trackable in the sensor data is predicted based on the determined speed.

9. The method according to claim 8, further comprising: When it is determined that the second object is not trackable in the sensor data: determining an angle between the first region and the identified region in the one or more video sequences; The position of the second object when not trackable in the sensor data is further predicted based on the angle.

10. The method according to claim 4, wherein: The sensor data is one of radar data, lidar data and video data.

11. The method according to claim 1, wherein: The plurality of regions in the one or more video sequences include a plurality of predetermined regions in the one or more video sequences.

12. The method according to claim 1, wherein: The one or more video sequences are iteratively divided into the plurality of regions in the one or more video sequences based on locations where object trajectories among the plurality of object trajectories in the one or more video sequences are lost and restored.

13. The method according to claim 12, wherein: The step of iteratively dividing the one or more video sequences into the plurality of regions in the one or more video sequences comprises: determining a first plurality of locations, each location indicating a location in the one or more video sequences where an object track among the plurality of object tracks is lost, clustering the first plurality of locations into a first plurality of clusters, and for each cluster in the first plurality of clusters, determining a region in the one or more video sequences based on the location in the cluster; and Determine a second plurality of positions, each position indicating a position in the one or more video sequences where an object track among the plurality of object tracks is recovered, cluster the second plurality of positions into a second plurality of clusters, and for each cluster in the second plurality of clusters, determine a region in the one or more video sequences based on the position in the cluster.

14. The method according to claim 1, further comprising the steps of: providing sensor data capturing the scene; tracking a third object in the sensor data to determine a third object trajectory; determining that a location of the third object in the sensor data is within a threshold distance from the first area in the one or more video sequences; and A feature vector of the third object is determined for use in object re-identification in an object tracking application.

15. A non-transitory computer readable storage medium having instructions stored thereon, the instructions being operable to implement the method of claim 1 when executed on a device having processing capabilities.

Images