Neural network output stabilization and uncertainty estimation through multi-object-tracking

Abstract

According to an aspect of the present disclosure, there is provided a computer-implemented method, in particular for object tracking, the method comprising: receiving a stream of data items containing one or more data elements; associating a first data element of the one or more data elements with a unique identifier; determining at least one feature associated with the first data element; and for each feature: determining contextual information relating to the feature; and generating an output indicative of the feature based on the contextual information, wherein the output and said identifiers are useable for tracking an object represented by said first data element. In an embodiment, the output includes an average of values included in said contextual information. Determining an average enables generating a more stable output and reducing the amount of data that needs to be processed to generate the output. Thus, the method enables stabilizing the output of a neural network by tracking and averaging said values.

Description

[0001] Neural Network output stabilization and uncertainty estimation through multi-object- tracking

[0002] The present invention relates to a method of processing data that may be employed for object tracking, and in particular multiple object tracking.

[0003] Background

[0004] Neural network-based object detection allows to automatically detect objects in video streams. Video streams typically consist of video frames which are received one frame at a time. Typically, neural networks are applied to each frame individually and return detected features such as bounding boxes around observed objects or a classification into one of a plurality of predefined categories. Multi-object tracking is used to combine detections over multiple frames into so called tracks. Each track has a uniquely assigned identifier.

[0005] However, the outputs of neural networks are not stable and may vary depending on changes in lighting or perspective in the video stream, for example. Therefore, applying a frame-by- frame object detection may result in unstable results over time, resulting in an inconsistent experience for users of the neural network outputs.

[0006] In the paper “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?”" by Alex Kendall and Yarin Gal, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, there is described a method to predict an image multiple times with dropout in the neural network to produce an average of the predictions to estimate an uncertainty. However, predicting the same image multiple times is computationally expensive.

[0007] The present disclosure aims to address these and other drawbacks, and particularly to provide a computer implementable method that provides stable results, an estimation of an uncertainty with less processing and an improved user experience.

[0008] Summary of the Invention

[0009] The present invention is defined in the independent claims. The dependent claims recite selected optional features. In the following, each of the described methods, apparatuses, examples, and aspects, which do not fully correspond to the invention as defined in the claims is thus not according to the invention and is, as well as the whole following description, present for illustration purposes only or to highlight specific aspects or features of the claims.

[0010] According to an aspect of the present disclosure, there is provided a computer-implemented method, in particular for object tracking, the method comprising: receiving a stream of data items containing one or more data elements; associating a first data element of the one or more data elements with a unique identifier; determining at least one feature associated with the first data element; and for each feature: determining at least one contextual information relating to the feature; and generating an output indicative of the feature based on the contextual information, wherein, in particular, the output and said identifiers are useable for tracking the data element, and more particularly an object represented by said first data element.

[0011] The present method enables improving the tracking of data elements, or objects represented by data elements. In an illustrative example, the data items correspond to video frames of a stream of video data. The data elements correspond to data content of interest, e.g., images of an object to be tracked. More generally, the data elements may correspond to any kind of data content of interest, e.g., image content indicative of a (spatial) object, acoustic content indicative of an object, or other data content indicative of a certain signal (e g. pattern or characteristic). When reference is made in the present disclosure to the tracking of (spatial) objects, the teaching applies mutatis mutandis to the tracking of other data content of interest. The at least one feature associated with the data element may be an object class. The contextual information may include historical information indicating the object class in previous frames. The output may be generated based on the object class in the present and preceding frames. For example, the output may represent an average or a probability. Accordingly, the output can be used to reduce uncertainties or fluctuations when tracking an object or features thereof.

[0012] The present method can be implemented using a neural network. The output may represent the neural network output. Compared to conventional systems as described above, the neural network output is more stable and provides a better user experience. In an embodiment, the method further comprises changing the output to a new output if the contextual information represents a first value below a first threshold and a second value above a second threshold, or if the contextual information represents a first value above a first threshold and a second value below a second threshold.

[0013] The method enables stabilising the output of feature information relating to a tracked object. For example, the feature may be an object class or a bounding box of a tracked object. The output may be a visual representation of the object class or bounding box. The output is changed from a current one (e.g. current object class) to a new one (new object class) only if the contextual information indicates that the new output is considered “better”, e.g. more accurate, by a predetermined margin. The margin can be defined by setting the first and second thresholds.

[0014] This is not limited to features such object classes or bounding boxes but applicable to any other features that may be used to describe or identify the first data element. Examples include detected visual or acoustic features and descriptors, colours, textures, locational and motional parameters, etc.

[0015] The output may be a visual representation of the feature, e.g. a visual indication of the object class, bounding box or any other feature. However, the invention is not limited to visual outputs but also encompasses other outputs such as acoustic or haptic signals.

[0016] The first and second values may represent confidence or uncertainty values.

[0017] In one embodiment, the first and second values represent a confidence in the current and new outputs, respectively. Only if the confidence in the current value is below the first threshold (say 40%) and the confidence in the new output exceeds the second threshold (say 60%), the output is changed from the current to the new one.

[0018] In another embodiment, the first and second values represent uncertainties. In this case, if the uncertainty pertaining to the current output exceeds a first threshold (say 60%) and the uncertainty pertaining to the new output is below the second threshold (say 40%), the output is changed from the current to the new one. Either way, fluctuations between different outputs can be reduced. The larger the margin i.e. difference between the first and second thresholds, the more changes of the current output to the new output are inhibited. Conversely, by reducing the margin, more frequent changes are enabled.

[0019] The confidence and uncertainty values may be determined based on entropy, for example. This is as such known and therefore not described further.

[0020] In an embodiment, method further comprises adjusting, in particular dynamically adjusting the first and / or second thresholds while tracking the first data element.

[0021] This embodiment enables adjusting the sensitivity of reacting to changes of the contextual information relating to the feature.

[0022] In an embodiment, the contextual information includes historical information, and in particular historical information that indicates a status and / or presence of the feature and / or a value associated with the feature in one or more preceding data items. For example, in an embodiment where the data items correspond to image frames, the historical information may indicate whether a detected visual feature of an object represented by the data elements was also present in past image frames.

[0023] In an embodiment, the output includes an average of values included in said contextual information. Determining an average enables generating a more stable output and reducing the amount of data that needs to be processed to generate the output. Thus, the method enables stabilizing the output of a neural network by tracking and averaging said values. For example, a single average value may be determined to indicate if and to what extent a detected visual feature was present in a plurality of past image frames.

[0024] In an embodiment, the contextual information indicates a probability value associated with the feature. The probability value may indicate the probability that a feature has been (correctly) detected, and / or that a feature is present in a detected object. The probability may be derived from other contextual information, e.g. a detection of the same feature in the past (e.g., in preceding frames), or the detection of other, related features. In an embodiment, the output includes an average of the probability values over two or more of said data items and / or over a period of time. By averaging the probability values, a more stable output can be obtained, and the amount of data to be processed can be reduced compared to processing individual probabilities for all data items.

[0025] The method may further comprise tracking the first data element, in particular an object represented by said first data element, by tracking the average of the probability values. In this embodiment, the tracking of a data element / object may be performed or enhanced by tracking the average of the probability values. Accordingly, the method may be used to enhance existing tracking-by-detection systems, including multi-object tracking systems. The method enables tracking the probability values associated with one or more objects over time. The values may include probability values output from a classification step and indicate the probability that the detected object belongs to a certain object class.

[0026] In an embodiment, the feature represents a class of the first data element, and in particular a class of an object associated with the first data element. In particular, the probability value indicates the probability of the first data element / object belonging to one of a plurality of classes. The probability may relate to a classification of an object to be tracked. For example, it may indicate the probability that the object belongs to a given class. This probability may vary across different data elements, e.g., image frames. In this embodiment, the output is based on an average of the probabilities over a plurality of the data elements (image frames). In an illustrative example, the probabilities that the feature belongs to an object of a given class may be 0.2, 0.8 and 0.6 for three consecutive data items. The average may be determined to be >0.5. Accordingly, the output indicates that the classification is more likely than not correct.

[0027] The method may further comprise determining a class probability distribution in respect of the first data element / object and / or a confidence score that the first data element / object belongs to one or more of said classes. In this embodiment, the method enables keeping track of a classification probability distribution of detected objects. In particular, the method enables calculating an entropy -based uncertainty for the classification, and also stabilising the output of the classification. The method may further comprise re-allocating the first data element, in particular an object represented by the first data element, from a current one of said classes to a new one of said classes if an average probability that the first data element / object belongs to the current class is below a first threshold and if an average probability that the first data element / object belongs to the new class is above a second threshold. By setting thresholds for re-classifying data elements / objects based on average probabilities, the output is stabilised. Reclassifications between individual frames is inhibited unless the average probabilities reach certain thresholds. However, in such case, the method may also include indicating an uncertainty. Thus, the method enables decoupling the classification and the output of an uncertainty indicator, resulting in an improved experience for an operator of an object tracking system implementing the method.

[0028] In an embodiment, the data items correspond to image frames and the data elements correspond to images of an object to be tracked, wherein the feature represents a visual feature, in particular a bounding box. Preferably, the output includes an average of values included in the contextual information. The historical information may include positional and / or dimensional information, in particular the size and / or location of the visual feature (bounding box) in prior image frames.

[0029] In this embodiment, the method enables stabilizing visual features, i.e. features that may be displayed to a user, such as bounding boxes. The displayed features change less over time, thereby improving visual object tracking and operator experience. In other words, when applied to bounding boxes or other visual features, the method enables a more stable and easier-to-process visual representation.

[0030] The method may further comprise associating a weight with said contextual information, and in particular associating respective weights with one or more values included in said contextual information, wherein the output reflects said weights. In this embodiment, the contextual information is weighted to take into account the context it represents. For example, the contextual information may include historical information which indicates whether a feature was present in any preceding image frames. This information may be weighted depending on a distance, in terms of time, of the preceding frames from a current frame. More recent frames may be given more weight than earlier frames. Also, a current frame may be given more weight than past frames. The method may further comprise determining a first weight for a current data item; determining a second weight for one or more preceding data items, in particular an average in respect of the one or more preceding data items; and optionally, dynamically adjusting the first and / or second weights while tracking first data element, in particular an object represented by the first data element. For example, the data items correspond to image frames. In this embodiment, the method may include (separately) determining the weight for a current frame and the weight for past frames, in particular an average weight for past frames. This enables adjusting the stability and sensitivity of the output. For example, if a current data item (frame) is given relatively more weight, the output becomes more sensitive to changes. If, on the other hand, an average of past frames is given relatively more weight, the output becomes more stable and fluctuates less.

[0031] In an embodiment, the data items correspond to image frames captured by a camera and the data elements correspond to images of an object contained in the image frames, wherein said contextual information includes a camera parameter, in particular a zoom setting, and / or a distance of the object from the camera, the method further comprising associating weights with said contextual information based on the camera parameter, in particular the zoom setting, and / or the distance. In this embodiment, the weighting may be performed to take into account how close an object is to a device that implements the method. For example, if the information is easier to retrieve and therefore more trustworthy when the object is close, the information from images in which the object is closer may be given more weight than information from images in which the object is further away. Similarly, the weighting may be performed on the basis of a zoom setting of a camera used for capturing images of the object. Information from images in which the object is “zoomed in” may be given more weight.

[0032] In an embodiment, the stream of data items includes data obtained from video, audio, and / or radar signals associated with a detected object. Additionally, or alternatively, the unique identifier is determined based on one or more parameters or properties associated with the detected object, in particular the object’s position, movement, class, and / or visual appearance.

[0033] According to another aspect of the present disclosure, there is provided a data element tracking system, comprising: one or more sensors for generating sensor signals associated with one or more data elements, in particular an object represented by the one or more data elements; one or more storage devices; and one or more processors configured to: process the sensor signals to detect the data element; generate a plurality of data items associated with a detected data element; and perform any of the method steps described above.

[0034] According to yet another aspect of the present disclosure, there is provided a computer program product for loading into a memory of a computer, comprising instructions, that, when executed by a processor of the computer, cause the computer to execute any of the steps of the computer-implemented method described above.

[0035] Brief description of the drawings

[0036] The features, objects, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which:

[0037] Figure 1 schematically illustrates an environment in which a method according to an embodiment of the present disclosure can be implemented;

[0038] Figure 2 illustrates a flow chart of a method according to an embodiment of the present disclosure; and

[0039] Figure 3 illustrates a schematic block diagram of a device according to an embodiment of the present disclosure.

[0040] Detailed description of exemplary embodiments

[0041] Embodiments of the present disclosure provide methods that can be used to stabilize neural network outputs and improve operator experience in multi-object tracking systems. In particular, the methods may be used to enhance existing tracking-by-detection multi-object tracking systems. The methods add the capability to track values associated with detected objects over time. Such a value may be a probability value output from a classification step, the value indicating the probability of the detected object belonging to a certain object class. Tracking this value can be implemented by calculating the average based on the values determined for the same object in the present and previous frames, e g., calculating the following average: average x(t)= a average [x(t-l)] + (1-a) x(t) wherein a is a weight in the range [0,1], By increasing a, the output may be made more stable, i.e., the fluctuation of output values may be reduced. By decreasing a, it can be more sensitive.

[0042] The average may be a simple moving average, a weighted moving average, or an exponential moving average. This enables keeping track of a running average of any feature associated with a detected object, such as bounding box sizes, class probability distributions, confidence scores, etc.

[0043] The resulting values can then be used in different ways to stabilise the output of a system implementing the method. They also enable calculating uncertainties for them. As an example, the method keeps track of the classification probability distribution of observed objects. This enables on the one hand calculating an entropy-based uncertainty for the classification, and on the other hand stabilising the output of the classification. In particular, a transition threshold can be used to ensure that a determined class is only changed if the probability of a previously believed best estimate for the class and a new best guess surpass this threshold. Accordingly, in situations where the class cannot be determined with high confidence, the output remains stable, and a switching of classes between frames is prevented. However, in this case the system output may also indicate a high uncertainty. This effectively decouples the output of uncertainty and the best guess of the state / class, resulting in an improved experience for a system operator.

[0044] Figure 1 schematically illustrates an environment in which a method according to an embodiment of the present disclosure can be implemented. The method may be implemented in an unmanned aerial vehicle (UAV) 10 to detect and track objects. The UAV 10 may have one or more sensors, e.g., a video camera, to monitor an area corresponding to a field of view (FOV) 11 of the sensors. In the illustrated example, there are three objects within the FOV of the UAV 10, namely a car 12, a drone 13 and a house 14. Each of the objects 12, 13 and 14 belongs to one or more different object classes. The car 12 may belong to the object classes “moving object”, “ground vehicle” and “passenger car”. The drone 13 may belong to the object classes “moving object”, “aerial vehicle” and “drone”. The house 14 may belong to the object classes “stationary object”, “building” and “house”. These classes are illustrative only. Each object may belong to fewer or additional or different classes.

[0045] The drone 10 may be communicatively coupled to a remote server 15. The drone 10 may be configured to transmit some or all the sensor signals from its sensors to the remote server 15. The method according to the present disclosure may be partially or wholly implemented on the remote server 15.

[0046] The drone 10 is configured to process the sensor signals from its video camera to detect and track the objects 12, 13 and 14. The sensor signals correspond to a stream of video frames. Each video frame contains a representation of the objects 12, 13 and 14 visible within the FOV 11. The objects 12, 13 and 14 may not be visible all the time. For example, the drone 13 may leave and re-enter the FOV 11. The car 12, when moving relative to the UAV 10, may be temporarily obscured by other objects, e.g., trees. Also, the objects 12, 13 and 14 may be visible from different perspectives and angles at different points in time and thus have different appearances.

[0047] The drone 10, implementing the method according to an embodiment of the present disclosure, is configured to process data representing contextual information when tracking objects. Such contextual information may include historical information relating to the class of a tracked object in preceding image frames. For example, a current image frame may be processed to determine that the object 12 belongs to the class “passenger car”. The contextual information, however, may indicate that the object 12 has been classified as a “truck” in a plurality of preceding image frames. In this case, the object 12 is not re-classified as “passenger car”. Instead, the object 12 remains in the class “truck” until a sufficient number of subsequent frames may indicate otherwise.

[0048] Figure 2 illustrates a flow chart of a method 200 according to an embodiment of the present disclosure. In step 201, the method receives a data stream comprising a plurality of data items. For example, the data items correspond to video frames in a stream of video data. Each data item may contain one or more data elements representing objects in an environment. For example, the data elements may correspond to images of the objects 12, 13 and 14 illustrated in Figure 1. In step 202, the data items are processed to identify any objects represented by the data elements contained in the data items. The identification can be done using object identifiers which are assigned to objects when they are first detected. Each identifier represents a fingerprint that uniquely identifies the associated object. The identifiers may be based on or include positional information (e.g., information relating to the position, movement and pose of the object) and other information such as an object class or visual features.

[0049] In step 203, one or more features are determined for each of the data elements. For example, a given data element may represent the object 12 (Figure 1), and the feature corresponds to the object class “moving object”. In another example, the feature represents a bounding box of the detected object in the stream of image frames.

[0050] In step 204, contextual information relating to the features determined in step 203 is obtained. The contextual information may include historical information, for example, a classification of the object or the size and location of the bounding box in prior image frames.

[0051] In step 205, the contextual information is used to generate an output. For example, if in a current image frame, the bounding box corresponds to bounding boxes of the same or similar sizes and locations, the bounding box of the current image frame can be generated as an output with high confidence. The output may include a confidence value, which may be indicated to a user or used otherwise in subsequent actions, e.g., for engaging an object.

[0052] The method 200 represents a continuous, iterative process. There may be a continuous stream of incoming data which is processed as described above. The method 200 may be implemented by a neural network.

[0053] Figure 3 illustrates a schematic block diagram of a device 300 according to an embodiment of the present disclosure which may implement methods according to the present disclosure.

[0054] The device 300 comprises at least one processor 302 and a memory 304. The memory 304 can be any type of memory. The device 300 optionally also comprises a sensor 306, for example a video camera, for generating a stream of sensor data, and a receiver / transmitter 308 for communicating with a remote server 310 or other devices. The processor 302, memory 304, sensor (e.g., camera) 306 and receiver / transmitter 308 are communicatively coupled with one another. The memory 304 stores instructions that, when executed by the processor 302, result in the execution, at least in part, of the methods as described herein.

[0055] Generally, methods according to the present disclosure can be applied to any incoming data stream, not only image data. In an embodiment, the method includes the following steps:

[0056] Consecutively receiving a plurality of data items, i.e., a stream of data items. The data items represent specific data content of interest. The data content can include an object in image data, a sound in audio data, a frequency pulse within a radar signal, etc. Such data content is also referred to as “data elements” in the present disclosure.

[0057] Identifying the data content of interest within the received data items. This includes associating the data content with an unambiguous identifier.

[0058] For each of the received plurality of data items, determining at least one feature of the identified data content. Examples are an object class, a bounding box, a size of a bounding box, a mode of a radar signal, etc. The determination of the feature includes a determination of a respective (per data item) probability value associated with the feature.

[0059] Determining an average of the plurality of probability values associated with the (i.e., the same) feature.

[0060] Outputting an indication of the feature based on the determined average.

[0061] Accordingly, the determination of a feature, such as the determination of an object class (e.g., detected object = car), is based on a plurality of data items of the same data stream, e.g., via an average value. A change of determination of (the status of) a feature from one frame to another (i .e., due to a high uncertainty) does not necessarily result in a change of output of the indication of the feature. Thereby, the method enables calculating uncertainties for downstream systems, such as uncertainties in respect of tracked values, which in turn can be used by downstream systems to create a more accurate estimation of the state of detected objects. Also, the method enables improving an operator experience by stabilising the visualization of the results of object detectors. In particular, the operator can expect a more consistent visualization of the results that consists of stabilised bounding boxes that change less over time, less frequently changing classifications for objects, and the capability to inspect uncertainties of predictions independently of the output values.

Claims

Claims1. A computer-implemented method, in particular for object tracking, the method comprising: receiving a stream of data items containing one or more data elements; associating a first data element of the one or more data elements with a unique identifier; determining at least one feature associated with the first data element; and for each feature: determining contextual information relating to the feature; and generating an output indicative of the feature based on the contextual information, wherein, in particular, the output and said identifier are usable for tracking the first data element, and more particularly an object represented by said first data element.

2. The method of claim 1, further comprising: changing the output to a new output if the contextual information represents a first value below a first threshold and a second value above a second threshold, or if the contextual information represents a first value above a first threshold and a second value below a second threshold.

3. The method of claim 2, wherein the first and second values represent confidence or uncertainty values.

4. The method of claim 2 or 3, further comprising adjusting, in particular dynamically adjusting the first and / or second thresholds while tracking the first data element.

5. The method of any preceding claim, wherein said output includes an average of values included in said contextual information.

6. The method of any preceding claim, wherein the contextual information includes a probability value associated with the feature.

7. The method of claim 6, wherein the output includes an average of the probability values over two or more of said data items and / or over a period of time.

8. The method of claim 7, wherein the average is a moving average, in particular a simple moving average, an exponential moving average or a weighted moving average.

9. The method of claim 7 or 8, further comprising tracking said first data element, in particular an object represented by said first data element, by tracking the average of the probability values.

10. The method of any preceding claim, wherein the feature represents a class of the first data element, in particular a class of an object associated with the first data element.

11. The method of claim 10 and any of claims 6 to 9, wherein the probability value indicates the probability of the first data element belonging to one of a plurality of classes12. The method of claim 11, further comprising determining a class probability distribution in respect of the first data element and / or a confidence score that the first data element belongs to one or more of said classes.

13. The method of claim 11 or 12, further comprising re-allocating the data element from a current one of said classes to a new one of said classes if a probability or an average probability that the detected first data element belongs to the current class is below a first threshold and if a probability or an average probability that the detected first data element belongs to the new class is above a second threshold.

14. The method of any preceding claim, wherein the data items correspond to image frames and the data elements correspond to images of an object to be tracked, and wherein the feature represents a visual feature.

15. The method of any preceding claim, wherein the contextual information includes historical information.

16. The method of claim 15, wherein the historical information indicates a status and / or presence of the feature and / or a value associated with the feature in one or more preceding data items.

17. The method of claim 14 and claim 15 or 16, wherein the historical information includes positional and / or dimensional information associated with the visual feature, in particular the size and / or location of the visual feature in prior image frames.

18. The method of claim 14 and any of claims 15 to 17, wherein the visual feature is a bounding box.

19. The method of any preceding claim, further comprising: associating a weight with said contextual information.

20. The method of claim 19, further comprising: associating respective weights with one or more values included in said contextual information, wherein the output reflects said weights.

21. The method of claim 19 or 20, further comprising: determining a first weight for a current data item; determining a second weight for one or more preceding data items.

22. The method of claim 21, wherein determining the second weight comprises determining an average in respect of the one or more preceding data items.

23. The method of claim 21 or 22, further comprising adjusting, in particular dynamically adjusting the first and / or second weights while tracking the first data element.

24. The method of any of claims 19 to 23, wherein the data items correspond to image frames captured by a camera and the data elements correspond to images of an object contained in the image frames, and wherein said contextual information includes a camera parameter, in particular a zoom setting, and / or a distance of the object from the camera, the method further comprising:associating weights with said contextual information based on the camera parameter, in particular the zoom setting, and / or the distance.

25. The method of any preceding claim, wherein the stream of data items includes data obtained from video, audio, and / or radar signals associated with a detected data element, in particular an object.

26. The method of any preceding claim, wherein said unique identifier is determined based on one or more parameters or properties associated with the detected data element, in particular the object’s position, movement, class, and / or visual appearance.

27. A data element tracking system, comprising: one or more sensors for generating sensor signals associated with a data element, such as an object; one or more storage devices; and one or more processors configured to: process the sensor signals to detect the data element; generate a plurality of data items associated with a detected data element; and perform the method of any preceding claim.

28. A computer program product for loading into a memory of a computer, comprising instructions, that, when executed by a processor of the computer, cause the computer to execute the computer-implemented method of any of claims 1-26.