Compensating for lost data intended for object detection

Abstract

It is provided a method for compensating for lost data intended for object detection in image frames. The method is performed by a lost data compensator. The method comprises: detecting a missed image frame, by determining absence of an image frame of sufficient quality for object detection; predicting location data, per previously identified object in a previously received image frame, for a time corresponding to the missed image frame, wherein the predicting is based on a history database comprising previous location data for each object; and storing the predicted location data in the history database.

Description

COMPENSATING FOR LOST DATA INTENDED FOR OBJECT DETECTIONTECHNICAL FIELD

[0001] The present disclosure relates to the field of object detection based on image frames, and in particular to compensating for lost data intended for object detection in image frames.BACKGROUND

[0002] Object detection and tracking from images can be used for many purposes, e.g. in XR (extended reality) applications, automated vehicles, traffic surveillance, etc. The object detection and tracking processes are based on input image frames captured from the environment, to predict the location and type of objects in the scene depicted by the image frames. These processes depend highly on the data that is received, whereby any lack of data can seriously affect their performance. Sometimes, distributed object detection is applied, where the object detection occurs using several nodes. For instance, a mobile device node can be used to capture images and perform feature extraction, while a server node can be used to perform the object detection based on image frames and feature data provided from the mobile device node.

[0003] If network conditions are degraded or unstable, the image frame data might not be transferred correctly from the mobile device node to the server node, whereby the object detection in the server node fails, due to the data (image frame data) lost in the transfer.SUMMARY

[0004] One object is to compensate for lost data intended for object detection in image frames.

[0005] According to a first aspect, it is provided a method for compensating for lost data intended for object detection in image frames. The method is performed by a lost data compensator. The method comprises: detecting a missed image frame, by determining absence of an image frame of sufficient quality for object detection; predicting location data, per previously identified object in a previously received imageframe, for a time corresponding to the missed image frame, wherein the predicting is based on a history database comprising previous location data for each object; and storing the predicted location data in the history database.

[0006] The method may further comprise: receiving feature data corresponding to the missed image frame; and adjusting the predicted location data, per object, based on the feature data. In this case, the storing comprises storing the feature data in the history database.

[0007] The detecting a missed image frame may be based on failing to receive an image frame corresponding the received feature data.

[0008] The adjusting the predicted location data per object may comprise: obtaining a predicted bounding box based on the predicted location data; determining object features being features in the feature data in an image space that are associated with the object, and non-object features being features in the feature data in an image space that are not associated with the object; determining a minimal bounding box just encompassing the object features; determining a maximal bounding box encompassing the minimal bounding box and a maximum amount of surrounding free space without encompassing any non-object feature; and determining an adjusted bounding box by adjusting the predicted bounding box, with the smallest amount, to refrain from being inside the minimal bounding box and to refrain from being outside the maximal bounding box.

[0009] The method may further comprise: determining a preliminary bounding box based on the object features and object features for a previous iteration; and adjusting the adjusted bounding box based on the preliminary bounding box, yielding a final bounding box.

[0010] The detecting a missed image frame may be based on failing to receive an image frame in a time span within which an image frame was expected to be received.

[0011] The method may further comprise: performing object detection based on a received image frame, in which case the storing comprises storing results of the object detection corresponding to the image frame in the history database.

[0012] The predicting location data may comprise predicting a rate of change per object.

[0013] The lost data compensator may be a server in communication with a mobile device.

[0014] The lost data compensator may be a mobile device configured to perform object detection in image frames captured by an imaging device of the mobile device.

[0015] According to a second aspect, it is provided a lost data compensator for compensating for lost data intended for object detection in image frames. The lost data compensator comprises: processing circuitry; and memory circuitry storing instructions that, when executed by the processing circuitry, cause the lost data compensator to: detect a missed image frame, by determining absence of an image frame of sufficient quality for object detection; predict location data, per previously identified object in a previously received image frame, for a time corresponding to the missed image frame, wherein the predicting is based on a history database comprising previous location data for each object; and store the predicted location data in the history database.

[0016] The lost data compensator may further comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to: receive feature data corresponding to the missed image frame; and adjust the predicted location data, per object, based on the feature data; in which case the storing comprises storing the feature data in the history database.

[0017] The instructions to detect a missed image frame may comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to detect the missed image frame based on failing to receive an image frame corresponding the received feature data.

[0018] The instructions to adjust the predicted location data per object may comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to: obtain a predicted bounding box based on the predicted location data; determine object features being features in the feature data in an image space that are associated with the object, and non-object features being features in the featuredata in an image space that are not associated with the object; determine a minimal bounding box just encompassing the object features; determine a maximal bounding box encompassing the minimal bounding box and a maximum amount of surrounding free space without encompassing any non-object feature; and determine an adjusted bounding box by adjusting the predicted bounding box, with the smallest amount, to refrain from being inside the minimal bounding box and to refrain from being outside the maximal bounding box.

[0019] The lost data compensator may further comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to: determining a preliminary bounding box based on the object features and object features for a previous iteration; and adjusting the adjusted bounding box based on the preliminary bounding box, yielding a final bounding box.

[0020] The instructions to detect a missed image frame may comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to detect the missed image frame based on failing to receive an image frame in a time span within which an image frame was expected to be received.

[0021] The lost data compensator may further comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to: perform object detection based on a received image frame; in which case the instructions to store comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to store results of the object detection corresponding to the image frame in the history database.

[0022] The instructions to predict location data may comprise instructions that, when executed by the processing circuitry, cause the lost data compensator to predict a rate of change per object.

[0023] The lost data compensator may be a server configured to be in communication with a mobile device.

[0024] The lost data compensator may be a mobile device configured to perform object detection in image frames captured by an imaging device of the mobile device.

[0025] According to a third aspect, it is provided a computer program for compensating for lost data intended for object detection in image frames. The computer program comprises computer program code which, when executed on a lost data compensator causes the lost data compensator to: detect a missed image frame, by determining absence of an image frame of sufficient quality for object detection; predict location data, per previously identified object in a previously received image frame, for a time corresponding to the missed image frame, wherein the predicting is based on a history database comprising previous location data for each object; and store the predicted location data in the history database.

[0026] According to a fourth aspect, it is provided a computer program product comprising a computer program according to claim third aspect and a computer readable means comprising non-transitory memory in which the computer program is stored.

[0027] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, in which:

[0029] Fig 1 is a schematic diagram illustrating an environment in which embodiments presented herein can be applied;

[0030] Figs 2A-C are schematic diagrams illustrating embodiments of where the lost data compensator can be implemented;

[0031] Figs 3A-B are schematic diagrams illustrating how object detection and tracking occurs over time;

[0032] Fig 4 is a schematic diagram illustrating when there is a missing image frame;

[0033] Fig 5 is a schematic diagram illustrating how bounding boxes can be exploited by the lost data compensator to improve predicted location data for a particular object;

[0034] Fig 6 is a schematic diagram illustrating how data can be organised in a history database;

[0035] Figs 7A-C are flow charts illustrating embodiments of methods for compensating for lost data intended for object detection in image frames;

[0036] Fig 8 is a schematic diagram illustrating components of the lost data compensator of Figs 2A-C;

[0037] Fig 9 is a schematic diagram showing functional modules of the lost data compensator of Fig 8 according to one embodiment; and

[0038] Fig 10 shows one example of a computer program product 90 comprising computer readable means.DETAILED DESCRIPTION

[0039] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of invention to those skilled in the art. Like numbers refer to like elements throughout the description.

[0040] According to embodiments presented herein, missing image frame data is compensated for in real-time, based on exploiting historic data for object detection. The solution is intended for server-assisted object detection or distributed object detection,in which a mobile device captures image frames and image features, and a server processes the transferred data to perform object processing (e.g. object detection and / or object tracking).

[0041] It is here provided a way to perform real-time compensation for lost data, e.g. due to poor network conditions, to prevent failed object processing. As the mobile device transmits image frames and features extracted from the images, the server checks whether there is any data loss (e.g. missing or delayed data). If there is no data loss, the object processor in the server proceeds as usual to generate object results, including e.g. location, size, and type of object, and the object results are accumulated in a history database for possible future data loss occurrences. Otherwise, if data loss is detected (image frame is not received or is highly delayed), the object results are predicted based on the past accumulated data in the history database and possibly received data, such as feature data. Since feature data is much smaller in size than image frames, the feature data could potentially be successfully received while the image frame is corrupted. The history database is updated accordingly.

[0042] Fig 1 is a schematic diagram illustrating an environment in which embodiments presented herein can be applied. A user 5 carries a mobile device 2. The mobile device 2 can be a smartphone, mobile phone, wearable device (e.g. smart glasses or smart jewellery), etc. The mobile device comprises an imaging device 7, e.g. in the form of a camera, stereo camera, depth camera, optionally combined with lidar, radar, etc., that provides images. The imaging device 7 can provide a sequence of images (corresponding to a video sequence), where each image in the sequence is called an image frame. The images reflect the environment around the user 5 of the mobile device 2. For instance, in the example of Fig 1, the image can capture a tree 25a, a car 25b and / or a bicycle 25c. It is to be noted that the terms image, image frame and frame are all terms that are used interchangeably herein.

[0043] The mobile device 2 comprises an I / O interface for connecting to a wide area network 6, such as the Internet, e.g. via Wi-Fi or a cellular network, to allow communication with a server 3. The server 3 can be what is known as an edge server (located topologically close to the mobile device 2) or a cloud server (located topologically more centrally).

[0044] The mobile device 2 and the server 3 can cooperate to perform distributed object detection based on images captured by the imaging device 7. In this way, the mobile device 2 captures images and performs feature extraction. Image frames and feature data are then transmitted to the server 3, which determines object results, including location, size, and type of object, for each of any zero or more objects in an image frame.

[0045] Figs 2A-C are schematic diagrams illustrating embodiments of where the lost data compensator 1 can be implemented.

[0046] In Fig 2A, the lost data compensator 1 shown being implemented in the server 3. The server 3 is thus the host device for the lost data compensator 1 in this implementation. The server 3 is capable of communicating with the mobile device 2 to receive data in the form of captured images and features.

[0047] In Fig 2B, the lost data compensator 1 shown being implemented in the mobile device 2. The mobile device 2 is thus the host device for the lost data compensator 1 in this implementation. The mobile device is then configured to perform object detection in image frames captured by an imaging device 7 of the mobile device 2.

[0048] In Fig 2C, the lost data compensator 1 is shown being implemented as a stand-alone device. The lost data compensator 1 thus does not have a host device in this implementation.

[0049] It is to be noted that the lost data compensator 1 can optionally be implemented partly in the mobile device 2 and partly in the server 3.

[0050] Figs 3A-B are schematic diagrams illustrating how object detection and tracking occurs over time. Fig 3A shows an image 20 with two objects, a tree 25a and a car 25b. Fig 3B shows an image 20 captured at a time after the image of Fig 3A. Now, the car 25b has moved slightly to the right. In Fig 3B, the car 25b can be tracked from the previous image of Fig 3A. A first bounding box 22a can be applied for the tree 25a and a second bounding box 22b can be applied for the car 25a. As known in the art per se, a bounding box is a rectangle that encompasses the object, in this case the tree 25a or the car 25b.

[0051] Fig 4 is a schematic diagram illustrating when there is a missing image frame. There is here a sequence 21 of image frames, e.g. captured by the imaging device 7 of the mobile device 2. In the sequence 21, there is a first image frame 20a, a second image frame 20b. However, after the second image frame, when a third image frame image frame might have been expected, there is no image frame, i.e. there is a missing image frame 20’, which may be missing e.g. due to poor conditions for a network transfer. After the missing image frame 20’ the sequence continues with a fourth image frame 2od and a fifth image frame 2oe.

[0052] Fig 6 is a schematic diagram illustrating how data can be organised in a history database 29. The history database 29 can contain location data 27 as well as feature data 28. The location data 27 is stored for multiple objects 30. For each object, location data 32 is stored for a plurality of frames 31. The feature data 28 is stored as features 33 for each of a plurality of frames 31. The population (and purging) of the history database 29 is described in more detail below, with reference to the store step 48 of Figs 7A and 7B.

[0053] Figs 7A-C are flow charts illustrating embodiments of methods for compensating for lost data intended for object detection in image frames. The embodiments of the methods are performed by a lost data compensator. First, embodiments illustrated by Fig 7A will be described.

[0054] In a missed image frame step 42, the lost data compensator 1 detects a missed image frame 20’. This is performed by determining absence of an image frame of sufficient quality for object detection. For instance, the detecting a missed image frame is based on failing to receive an image frame corresponding to the feature data that have been correctly received. Alternatively or additionally, the detecting a missed image frame is based on failing to correctly receive an image frame in a time span within which an image frame was expected to be received.

[0055] Hence, the lost data compensator 1 checks to see what data it has obtained related to an image frame. There are four main scenarios presented here.

[0056] In a first scenario, both the image frame and image features are received, i.e. when the network connection is good, and no loss occurs. This scenario requires no compensation for lost data.

[0057] In a second scenario, only image features are correctly received. If image frame data is received, such data is corrupted in this scenario and the received image frame data is discarded. This scenario can occur when the network connection is slightly degraded (the image features constitute smaller data to be transmitted and are more likely to get to be correctly transferred compared to the larger amount of data of the image frame). The lost data compensator 1 notices this scenario when there is data missing because image features arrive, but image frame data does not.

[0058] In a third scenario, no image related data at all arrives (i.e. neither image frame data nor feature data) at the lost data compensator 1, which occurs when the network connection is significantly degraded. In this case, the lost data compensator 1 runs a specific check to detect that data is missing. For this situation, the system can exploit that the standard operation of an image capturing device providing image frames at a fixed frame rate. The lost data compensator 1 can detect the default frame rate of the sensor in an initialization phase by collecting the mode or median of the inter-image time. Thus, the lost data compensator 1 can detect that an image has been lost when the elapsed time is longer (by a margin of error) than expected. If the capturing device does not work at a fixed frame rate, a timeout can be employed to identify that an image frame should have been received for the well-functioning of the object processor. This timeout can be calculated using content information from the images such as motion or change rate.

[0059] In a fourth scenario, image frames and / or image features arrive but delayed. Image frames or image features are considered to be delayed when they arrive after they have been considered lost (by the procedures explained for the second and third scenario). However, due to network retransmissions, it can happen that the image frame and / or the image features eventually manage to get correctly transferred. By knowing the timestamp T of when the image frame was captured and the time t for the last predicted image, they can be compared according to: if T > t (the retransmitted data is still the newest data available), the process is started again by running the objectprocessor if the retransmission managed to send the image frame or the lost detection predictor if the image is not retransmitted; if T < t (the retransmitted data is older than the last prediction) the data is updated in the history database by correcting the prediction for time T and the subsequent predictions in the history of objects by reinitializing the history-based predictor with the new data to improve upcoming estimations.

[0060] In a predict location data step 46, the lost data compensator 1 predicts location data (for each previously identified object in a previously received image frame) for a time corresponding to the missed image frame. The predicting is based on a history database 29 comprising previous location data for each object 25a-c. In one embodiment, the predicting location data comprises predicting a rate of change (i.e. first-order parameter) and optionally rate of acceleration (i.e. second-order parameter) per object.

[0061] Hence, when an image frame is not received correctly, the accumulated history data in the history database 29 is used to predict the position of the objects for the missing image frame to prevent the network degradation from affecting the final user or application. This process can be performed with a state estimator or predictive method such as a Kalman filter, a particle filter or a support vector regressor.

[0062] There are two beneficial effects of this step. First, the current image frame is not needed in the prediction process, ensuring object results even with poor network conditions. Second, by using the history database 29, the prediction is only performed when lost data is detected, avoiding computation overhead for situations where the image frame is successfully transferred.

[0063] This step is now described in detail for an example where Kalman filters are employed.

[0064] With the accumulated object history in the history database 29, each object is assigned an instance of a Kalman filter that is initialized when the image frame loss is detected at a first time. Each Kalman filters estimates the new states of its associated object by modelling the motion of its object. The state of the object can vary for differentimplementations, being the state space can e.g. include the coordinates of the 2D centre (or other predetermined reference point) of the bounding box (the same size is applied to the prediction as the history size). This data is also accumulated over time in the object history of the history database 29 to obtain the history-based prediction. Each Kalman filter is initialized with a default transition matrix and a default observation matrix, as the motion models of objects are not known (different options are possible based on the available data, e.g. the objects can move linearly, non-linearly or following a motion model based on known priors).

[0065] The small state space design (only reference point coordinates [x, y]) reduces memory cost and computation of the Kalman filters, while keeping the quality of predicted results. Conversely, the state could optionally include more information, including first-order properties such as velocity, size change, and optionally also second- order properties such as acceleration etc. to obtain a more complete motion model which may lead to a better estimation but consumes more time and / or resources to be computed.

[0066] Next, for each Kalman filter, a corresponding measurement sequence is used from the earliest one to the last received one to update the motion model of the Kalman Filter in an expectation-maximization manner. In this process, a noise-aware filtering of the history measurements (that could contain noise) is achieved.

[0067] Then, each Kalman filter predicts the current lost detections using the step update function. Depending on the state design chosen for the implementation, the predicted results can be a subset of the object data (e.g. the object data contains object label, confidence, x, y coordinates and height and width and the state space can be only the x, y coordinates) that can be completed with the available data from the last predicted bounding box. The state space and predicted results can also be an extension of the object data (e.g. by adding first-order properties, or first-order and second-order properties). In this way, the history-based predictor outputs initial bounding boxes of the objects in the missing image frame which will then be refined with the feature data.

[0068] The Kalman filters for each object will not be reinitialized unless the history manager reinitializes the history for the corresponding object.

[0069] Snippet 1 presents pseudo code of an example of a history-based predictor using Kalman filters. Upon frame loss, the first operation is to ensure the Kalman Filters are instantiated, and use the function filter 0 to model the motion models. Notice that the call of filterf) is not always necessary, e.g. when two or more consecutive frames are lost. Then for each object, the Kalman Filters use the step_updateQ function to predict the lost results, which are first appended to the measurements and then output.Require: measurements of objects if Kalman Filters not initialized then for obj in objects do kf_ 0 bjs . append(KF. ini tQ ) end for end if predicted_locatians <-[] for obji, measurements in objects_measurements do if mean[pbji], cov[pbji] is None then# EM the motion model of ithobject's Kalman filter endmean, cov <- kf_objs[obji].step_update(mean[obji], cov[ bji]') predicted_locations.append((x = mean[o], y = mean[2])) end for for obji, location in predicted_locations do measurements[obji].append (jocation.x, location.y')) end for return predicted_locationsSnippet 1: pseudo code of a history-based predictor using Kalman filters

[0070] Depending on the use case, certain scenarios might tolerate longer computational time, but require higher precision of the predictions. In extreme cases where the frame loss continues for long periods, more complex probability algorithms could be applied in the approximation of the linear motion model of the Kalman filter toinclude the elapsed time since the last object detection results (from the object processor) were received. Other alternative motion modelling methods (including nonlinear Kalman filters and regression) also have the possibility to integrate similar algorithms, if more advanced estimation would be needed.

[0071] In a store step 48, the lost data compensator 1 stores the predicted location data 32 in the history database 29, that can be used for later location data predictions.

[0072] The history database 29 accumulates the transmitted features 33 per frame, e.g. in a data structure of a queue; where the queue does not require matching or resetting. All the features (belonging to the objects, background and unmatched ones) can be stored in the history database 29. All these features, along with the accumulated object results 27, allow the lost data compensator 1 to update the feature labels in future iterations.

[0073] As for the object results, the history database 29 accumulates the outputs from the object detection (when there is no image frame loss) and the final bounding boxes (when there is image frame loss) in the object history 27.

[0074] The object history 27 accumulates the data that is used for the history-based prediction. For instance, in the example above using x, y coordinates for the state space of the Kalman filter, the object history should store this same data.

[0075] The object history 27 is maintained in order to keep the correspondences between each object in the image frames. There are two embodiments to achieve this.

[0076] In a first embodiment, the object history is maintained in a simplified way only using the object detector or object tracking results. The object detector assigned an id to each detected object. The object tracker will reuse the id:s for the next frames. In this situation, the object results are appended directly to the queue. However, when a new object detection is performed or the number of tracked objects changes, the system will not be able to keep the correspondence of the object id:s. For this reason, in those situations, the accumulation of the object history is then restarted.

[0077] This embodiment allows low (or even zero) cost history management. However this embodiment is limited to the case where object tracking is used and needs to be reinitialized with every object detection.

[0078] In a second embodiment, a matching algorithm is used (e.g. Hungarian algorithm) to manage the object history queue to prevent the need to reset the queue when new detections or changes in the number of tracker object occur. As the objects in the current frame have different id:s than the history measurements, the Hungarian Algorithm matches the current ones with history measurements at t-1. The new objects in the current frame are then assigned an id, and the exiting objects are discarded. The rest matched objects would have longer history measurements than the new ones.

[0079] This embodiment is applicable for all object processing and could have longer history sequence that allows more accurate prediction. However the matching algorithm implies additional computational requirements.

[0080] Additionally, the two data sets of location data 27 and feature data 28 could optionally be set with a maximum length, to prevent the unnecessary storage of obsolete data. This can be maintained in a first -in-first-out manner. There are several ways in which the maximum length can be set, e.g. using the uncertainty (covariance for Kalman filters). We can have a scenario of good network quality and low uncertainty in which the maximum length is set to a predefined value. And a second scenario in which, as the uncertainty of the history-based prediction grows, the system can increase the maximum length in order to make use of more information in the estimation. In this way, we can tune the queue length to adapt to the uncertainty.

[0081] Looking now to Fig 7B, only new or modified steps compared to what is illustrated by Fig 7A will be described.

[0082] In an optional extract features step 38, the lost data compensator 1 extracts(one or more) features from an image frame. Each image frame is processed to extract relevant data from the image frame in the form of features. Features can be corners, edges, regions of interest points (e.g. local features such as SIFT (scale-invariant feature transform), ORB (orientated FAST (features from accelerated segment test) and robustBRIEF (binary robust independent elementary features)), etc.) or any other suitable sparse representation of the image. In this way, initial raw data can be reduced and more easily managed while continuing to describe the image.

[0083] In an optional send image frame and features step 39, the lost data compensator 1 sends the image frame and the features extracted in the extract features step 38 for further processing.

[0084] Collectively, the extract features step 38 and the send image frame and features step 39 can be termed data provision 51.

[0085] In an optional receive data step 40, the lost data compensator 1 receives feature data 11 corresponding to the missed image frame (e.g. provided by the send image frame and features step 39). When possible, the image frame is also received in this step. When the image frame is not correctly received in this step, the compensation for lost (image) data as described herein is applied.

[0086] In Fig 7B, the missed image frame step 42 of Fig 7A is represented as a conditional step 42. When the lost data compensator 1 determines that there is a missed image frame, the method proceeds to the predict location data step 46 (as in Fig 7A). Otherwise, there is no missed image frame, and the method proceeds to an optional perform object detection step 44.

[0087] In an optional perform object detection step 44, the lost data compensator 1 performs object detection based on a received image frame, e.g. based on YOLO (you only look once) or Faster R-CNN (region convolutional neural network) or object tracker (e.g. median flow or KCF (kernelized correlation filters) depending on the requirements of the final application to infer the objects in the image. The output of the object detection is the type of object, its location and size (e.g. using a bounding box) and in some cases also the confidence of that result.

[0088] When step 44 is performed, the subsequent store step 48 comprises storing the results of the object detection corresponding to the image frame in the history database 29.

[0089] In an optional adjust predicted location data step 47, the lost data compensator 1 adjusts the predicted location data, per object, based on feature data received in the receive data step 40. When network conditions are poor, feature data might still be successfully transferred when an image frame fails to transfer, since the feature data occupies significantly less data than an image frame. The feature data can be used to adjust and refine object detections in real time, even though the current image frame is not available. In this way, lost image frames can be compensated for such that the object data consumer (application / user) might not perceive the network degradation and is not affected by missing image frame(s). This allows more accurate results to be generated that compensates for fast environmental changes and less relevant history data due to long disconnection periods.

[0090] When the feature data 33 is available, the store step 48 comprises storing the feature data 33 in the history database 29.

[0091] In an optional provide result step 49, the lost data compensator 1 provides the results for the final user or application.

[0092] Collectively, the receive data step 40, the conditional missed image frame step 42, the perform object detection step 44, the predict location data step 46, the adjust predicted location data step 47, the store step 48 and the provide result step 49 can be termed lost data management 52.

[0093] It is to be noted that the data provision 51 and the lost data management 52 can be performed by the same device, e.g. when the lost data compensator 1 is provided in the mobile device 2. Alternatively, the data provision 51 can be performed in a different device (e.g. mobile device 2) than the device (e.g. server 3) that performs the lost data management 52.

[0094] Fig 7C illustrated optional sub-steps of the adjust predicted location data step 47 of Fig 7B. This is described with reference also to Fig 5 . This refinement involves first identifying the features that belong to each object and then, using that data, adjusting the bounding box of the object. This procedure takes as inputs the predictedbounding box from the history-based prediction, the features 11 received and the history data 12 from the previous iteration.

[0095] There are here three inputs: a first input 8a, a second input 8b and a third input 8c. The first input 8a comprises a predicted bounding box 10 that is determined based on predicted location data (see the predict location data step 46 described above). The second input 8b comprises a set of features 11. The features are the result of feature extraction from an image frame, e.g. performed by the mobile device 2 in the extract features step 38. The third input 8c comprises a previous features bounding box 12 that encompasses object features for a previous iteration for the particular object.

[0096] In an optional obtain predicted bounding box step 47a, the lost data compensator 1 obtains a predicted bounding box 10 (of the first input 8a based on the predicted location data (from the predict location data step 46).

[0097] In an optional classify features step 47b, the lost data compensator 1 determines 47b object features 14 (marked with x’s in Fig 5) and non-object features 13. The object features 14 are features in the feature data 11 in an image space that are associated with the object. The non-object features 13 are features in the feature data in an image space that are not associated with the object.

[0098] To identify the features that belong to each object, the features 11 received for the last image are matched against the history of features from the history database 29. This allows the lost data compensator 1 to estimate which features belong to the background, which features belong to each object and which features could not be matched. The features that could not be matched are discarded and the rest are stored for subsequent processing. The lost data compensator 1 masks the remaining features as object features 14 and non-object features 13.

[0099] In an optional determine minimal bounding box step 47c, the lost data compensator 1 determines a minimal bounding box 15 just encompassing the object features. In one embodiment, there needs to be at least ob_min features identified as belonging to the object, for the minimal bounding box to be generated. ob_min is theminimum number of features belonging to an object to be able to extract a reliable minimal bounding box.

[0100] In an optional determine maximal bounding box step 47d, the lost data compensator 1 determines a maximal bounding box 16 encompassing the minimal bounding box and a maximum amount of surrounding free space without encompassing any non-object feature. In one embodiment, there needs to be at least nob_min features identified as not belonging to that object, for the maximal bounding box to be generated. nob_min is the minimum number of features not belonging to the object in question to be able to extract a reliable maximal bounding box.

[0101] In an optional determine adjusted bounding box step 47e, the lost data compensator 1 determines an adjusted bounding box 17. The adjusted bounding box is determined by adjusting the predicted bounding box, with the smallest amount, to refrain from being inside the minimal bounding box and to refrain from being outside the maximal bounding box.

[0102] In other words, the object is inside the minimal and maximal bounding boxes. In this step, the adjusted bounding box, which is here represented by the top-left x_adj, y_adj coordinates, the width (w_adj) and height (h_adj) of the region containing the object, should meet the following constraints: x_min >= x_adj >= x_max y_min >= y_adj > = y_max w_max - (x_adj - x_max) >= w_adj >= w_min + ( x_min - x_adj) h_max - (y_adj - y_max) >= h_adj>= h_min + ( y_min - y_adj),

[0103] where x_min, y_min, h_min, w_min represent the parameters of the minimal bounding box and x_max, y_max, h_max, w_max are the parameters of the maximal bounding box. Considering these constraints, several techniques can be applied to select the exact x_adj, y_adj, w_adj and h_adj for the adjusted bounding box. For instance, an adjusted bounding box can be determined that meets the constraintsbut keeps as much as possible the size of the initial one, or that has the centroid as close as possible to the initial one.

[0104] If there are not a sufficient number of features detected to calculate the maximal or minimal bounding boxes, only the other one would be used to determine the adjusted bounding box. If neither the maximal nor minimal bounding boxes could be calculated, this step is skipped.

[0105] In an optional determine preliminary bounding box step 47!, the lost data compensator 1 determines a preliminary bounding box 18 based on the object features and object features 12 for a previous iteration. By analysing the relation between the bounding box and the features of the object in the previous frames, the preliminary bounding box can be extracted for the currently detected features. This can be performed using different techniques, for example comparing the relative position of the features and the corners of the bounding box in the previous frames to estimate their current relative position.

[0106] In an optional adjust adjusted bounding box step 47g, the lost data compensator 1 adjusts the adjusted bounding box 17 based on the preliminary bounding box, yielding a final bounding box 19.

[0107] The final bounding box 19 can be determined to be more similar to the preliminary bounding box 18 or to the adjusted bounding box 17. One way to do this is to weight both bounding boxes based on a decay. The decay is a temporally dependent variable [o, 1] that increases with a duration since the last object results that were added to the history database 29, and optionally also with a recorded movement for that object, such that object results for fast moving objects become obsolete faster. Then, the adjusted bounding box 17 and the preliminary bounding box can then be combined, with the decay being the weight for the preliminary bounding box (that is based more on history), and (1- decay) being the weight for the adjusted bounding box (that is based more on recent feature data). Another option to adjust the bounding boxes is to use the covariance (or other uncertainty metric) of the history-based prediction. In this way, the final bounding box can be adapted to increase the weight of the preliminary boundingbox when the uncertainty of the currently adjusted bounding box is high or the bounding boxes could even be enlarged to account for the increased uncertainty.

[0108] The final bounding box is then provided part of the results that are provided in the provide result step 49.

[0109] In embodiments illustrated by Fig 7C, having several sources of data and redundancy in the bounding box estimation increases accuracy even with low environmental changes and recent history data.

[0110] The usage of image features allows additional functionalities such as detection of appearing and disappearing objects. In the case of new objects appearing in the image, it can be identified by clusters of features that cannot be matched to neither the background features nor object features. Such clusters could give a hint on the possibility of a new object appearing. In the case of objects leaving the image, a combination of two situations gives a hint: firstly, by identifying that there were no image features matched for the object at time t and, secondly, by analysing the position of the features belonging to an object in a recent time interval ( t- 5, t) and how these features approached the border of the image.

[0111] According to embodiments presented herein, object results can be made available even when there is loss of image frames due to poor network conditions, effectively increasing the robustness of object detection. The lost data is compensated for in real time, as it does not need recovery of data transfer in order to start the prediction process, which would be needed e.g. if interpolation was used.

[0112] Efficiency is increased in terms of latency and power consumption, due to the accumulation of prior data in the history database, effectively avoiding the need to run the predictor for every frame, but only when necessary.

[0113] Fig 8 is a schematic diagram illustrating components of the lost data compensator 1 of Figs 2A-C. It is to be noted that when the lost data compensator 1 is implemented in a host device, one or more of the mentioned components can be shared with the host device. Processing circuitry 60 is provided using any combination of one or more of a suitable central processing unit (CPU), graphics processing unit (GPU),multiprocessor, neural processing unit (NPU), microcontroller, digital signal processor (DSP), etc., capable of executing software instructions 67 stored in memory circuitry 64, which can thus be a computer program product. The processing circuitry 60 could alternatively be implemented using an application specific integrated circuit (ASIC), field programmable gate array (FPGA), etc. The processing circuitry 60 can be configured to execute embodiments of methods described above with reference to Figs 7A-C.

[0114] The memory circuitry 64 can be any combination of random-access memory (RAM) and / or read-only memory (ROM). The memory circuitry 64 also comprises non- transitory persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid-state memory or even remotely mounted memory.

[0115] A data memory 66 is also provided for reading and / or storing data during execution of software instructions in the processing circuitry 60. The data memory 66 can be any combination of RAM and / or ROM.

[0116] The lost data compensator 1 further comprises an I / O interface 62 for communicating with external and / or internal entities. Optionally, the I / O interface 62 also includes a user interface.

[0117] Other components of the lost data compensator 1 are omitted in order not to obscure the concepts presented herein.

[0118] Fig 9 is a schematic diagram showing functional modules of the lost data compensator 1 of Fig 8 according to one embodiment. The modules are implemented using software instructions such as a computer program executing in the lost data compensator 1. Alternatively or additionally, the modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The modules correspond to the steps in the methods illustrated in Figs 7A-C.

[0119] A feature extractor 68 corresponds to step 38. A data sender 69 corresponds to step 39. A data receiver 70 corresponds to step 40. A missed image determiner 72corresponds to step 42. An object detector 74 corresponds to step 44. A location predictor 76 corresponds to step 46. A location adjuster 77 corresponds to step 47. A predicted bb (bounding box) obtainer 77a corresponds to step 47a. A classifier 77b corresponds to step 47b. A minimal bb determiner 77c corresponds to step 47c. A maximal bb determiner 7yd corresponds to step 47d. A bb adjuster 77c corresponds to step 47e. A preliminary bb determiner yyf corresponds to step 47k A storer 78 corresponds to step 48. A result provider 79 corresponds to step 49.

[0120] Fig 10 shows one example of a computer program product 90 comprising computer readable means. On this computer readable means, a computer program 91 can be stored in a non-transitory memory. The computer program can cause processing circuitry to execute a method according to embodiments described herein. In this example, the computer program product 90 is in the form of a removable solid-state memory, e.g. a Universal Serial Bus (USB) drive. As explained above, the computer program product could also be embodied in a memory of a device, such as the computer program product 64 of Fig 8. While the computer program 91 is here schematically shown as a section of the removable solid-state memory, the computer program can be stored in any way which is suitable for the computer program product, such as another type of removable solid-state memory, or an optical disc, such as a CD (compact disc), a DVD (digital versatile disc) or a Blu-Ray disc.

[0121] The aspects of the present disclosure have mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims. Thus, while various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMS1. A method for compensating for lost data intended for object detection in image frames, the method being performed by a lost data compensator (1), the method comprising: detecting (42) a missed image frame (20’), by determining absence of an image frame of sufficient quality for object detection; predicting (46) location data, per previously identified object in a previously received image frame, for a time corresponding to the missed image frame, wherein the predicting is based on a history database (29) comprising previous location data for each object (25a-c); and storing (48) the predicted location data (32) in the history database (29).

2. The method according to claim 1, further comprising: receiving (40) feature data (11) corresponding to the missed image frame; and adjusting (47) the predicted location data, per object, based on the feature data; wherein the storing (48) comprises storing the feature data (33) in the history database (29).

3. The method according to claim 2, wherein the detecting (42) a missed image frame is based on failing to receive an image frame corresponding the received feature data.

4. The method according to claim 2 or 3, wherein the adjusting (47) the predicted location data per object comprises: obtaining (47a) a predicted bounding box (10) based on the predicted location data; determining (47b) object features (14) being features in the feature data in an image space that are associated with the object, and non-object features (13) being features in the feature data in an image space that are not associated with the object; determining (47c) a minimal bounding box (15) just encompassing the object features; determining (47d) a maximal bounding box (16) encompassing the minimal bounding box and a maximum amount of surrounding free space without encompassing any non-object feature; anddetermining (47c) an adjusted bounding box (17) by adjusting the predicted bounding box, with the smallest amount, to refrain from being inside the minimal bounding box and to refrain from being outside the maximal bounding box.

5. The method according to claim 4, further comprising: determining (47F) a preliminary bounding box (18) based on the object features and object features (12) for a previous iteration; and adjusting (47g) the adjusted bounding box (17) based on the preliminary bounding box, yielding a final bounding box (19).

6. The method according to any one of claims 2 to 5, wherein the detecting (42) a missed image frame is based on failing to receive an image frame in a time span within which an image frame was expected to be received.

7. The method according to any one of the preceding claims, further comprising: performing (44) object detection based on a received image frame; and wherein the storing (48) comprises storing results of the object detection corresponding to the image frame in the history database (29).

8. The method according to any one of the preceding claims, wherein the predicting (46) location data comprises predicting a rate of change per object.

9. The method according to any one of the preceding claims, wherein the lost data compensator (1) is a server (3) in communication with a mobile device (2).

10. The method according to any one of claims 1 to 8, wherein the lost data compensator (1) is a mobile device (2) configured to perform object detection in image frames captured by an imaging device (7) of the mobile device (2).

11. A lost data compensator (1) for compensating for lost data intended for object detection in image frames, the lost data compensator (1) comprising: processing circuitry (60); and memory circuitry (64) storing instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to: detect a missed image frame (20’), by determining absence of an image frame ofsufficient quality for object detection; predict location data, per previously identified object in a previously received image frame, for a time corresponding to the missed image frame, wherein the predicting is based on a history database (29) comprising previous location data for each object (25a-c); and store the predicted location data (32) in the history database (29).

12. The lost data compensator (1) according to claim 11, further comprising instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to: receive feature data (11) corresponding to the missed image frame; and adjust the predicted location data, per object, based on the feature data; wherein the storing (48) comprises storing the feature data (33) in the history database (29).

13. The lost data compensator (1) according to claim 12, wherein the instructions to detect a missed image frame comprise instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to detect the missed image frame based on failing to receive an image frame corresponding the received feature data.

14. The lost data compensator (1) according to claim 12 or 13, wherein the instructions to adjust the predicted location data per object comprise instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to: obtain a predicted bounding box (10) based on the predicted location data; determine object features (14) being features in the feature data in an image space that are associated with the object, and non-object features (13) being features in the feature data in an image space that are not associated with the object; determine a minimal bounding box (15) just encompassing the object features; determine a maximal bounding box (16) encompassing the minimal bounding box and a maximum amount of surrounding free space without encompassing any nonobject feature; and determine an adjusted bounding box (17) by adjusting the predicted bounding box,with the smallest amount, to refrain from being inside the minimal bounding box and to refrain from being outside the maximal bounding box.

15. The lost data compensator (1) according to claim 14, further comprising instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to: determining (47F) a preliminary bounding box (18) based on the object features and object features (12) for a previous iteration; and adjusting (47g) the adjusted bounding box (17) based on the preliminary bounding box, yielding a final bounding box (19).

16. The lost data compensator (1) according to any one of claims 12 to 15, wherein the instructions to detect a missed image frame comprise instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to detect the missed image frame based on failing to receive an image frame in a time span within which an image frame was expected to be received.

17. The lost data compensator (1) according to any one of claims 11 to 16, further comprising instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to: perform object detection based on a received image frame; and wherein the instructions to store comprise instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to store results of the object detection corresponding to the image frame in the history database (29).

18. The lost data compensator (1) according to any one of claims 11 to 17, wherein the instructions to predict location data comprise instructions (67) that, when executed by the processing circuitry, cause the lost data compensator (1) to predict a rate of change per object.

19. The lost data compensator (1) according to any one of claims 11 to 18, wherein the lost data compensator (1) is a server (3) configured to be in communication with a mobile device (2).

20. The lost data compensator (1) according to any one of claims 11 to 18, wherein the lost data compensator (1) is a mobile device (2) configured to perform object detection in image frames captured by an imaging device (7) of the mobile device (2).

21. A computer program (67, 91) for compensating for lost data intended for object detection in image frames, the computer program comprising computer program code which, when executed on a lost data compensator (1) causes the lost data compensator (1) to: detect a missed image frame (20’), by determining absence of an image frame of sufficient quality for object detection; predict location data, per previously identified object in a previously received image frame, for a time corresponding to the missed image frame, wherein the predicting is based on a history database (29) comprising previous location data for each object (25a-c); and store the predicted location data (32) in the history database (29).

22. A computer program product (64, 90) comprising a computer program according to claim 21 and a computer readable means comprising non-transitory memory in which the computer program is stored.

Images