Systems, methods, and computer programs for retraining pre-trained object classifiers

Abstract

To provide a mechanism for retraining a pre-trained object classifier.SOLUTION: A method comprises obtaining a stream of image frames of a scene. Each of the image frames depicts an instance of a tracked object. The method comprises classifying, with a level of confidence, each instance of the tracked object so as to belong to an object class. The method comprises verifying that the level of confidence for at least one of the instances of the tracked object is higher than a threshold confidence value. The method comprises, when it is verified being higher than the threshold confidence value, annotating all instances of the tracked object in the stream of image frames as belonging to the object class, and yielding annotated instances of the tracked object. The method comprises retraining a pre-trained object classifier with at least some of the annotated instances of the tracked object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION The embodiments presented herein relate to methods, systems, computer programs, and computer program products for retraining a pre-trained object classifier. [Background technology]

[0002] Object recognition is a general term used to describe a collection of related computer vision tasks involving identifying objects within an image frame. One such task is object classification, which involves predicting the class of an object within an image frame. Another such task is object localization, which refers to identifying the location of one or more objects within an image frame and, optionally, providing a bounding box surrounding the objects thus localized. Object detection can be considered a combination of these two tasks, thus localizing and classifying one or more objects within an image frame.

[0003] One way to perform object classification is to compare an object to be classified with a set of template objects representing different object classes, and then classify the object into the object class of the template object that is most similar to the object according to a metric. Classification can be based on a model trained on a dataset of annotated objects. Models for object classification can be developed using deep learning (DL) techniques. The resulting model can be called an object classification DL model, or simply a model. Such a model can be trained based on a dataset that is continuously improved. The trained and updated model needs to be sent to systems that use older versions of the model during a firmware update or similar update process.

[0004] Generally speaking, a model is only as good as it has been trained. Thus, a certain amount of further training of the model may result in improved classification. In other words, the more training data there is, the better an object classifier may perform (in terms of classification accuracy, computational speed at which the classification can be performed, etc.). To capture as much training data as possible and be as general as possible, the model is a traditional static, pre-trained, general-purpose model that is applied to a set of training data regardless of the conditions under which the image frames in the training data were captured. This generally requires annotating the training data. However, the cost of annotation, either in terms of manual labor or computation, and privacy regulations may limit the possibility of annotating large sets of training data, as well as the acquisition of actual training data itself.

[0005] US Patent Application Publication No. 2017 / 039455 relates to a method for protecting the environment.

[0006] "Robust people counting using sparse representation and random projection," by Foroughi Homa et al., Pattern Recognition, Vol. 48, No. 10, October 1, 2015, pp. 3038-3052, relates to a method for estimating the number of people present in an image for practical applications including visual surveillance and public resource management.

[0007] U.S. Patent Application Publication No. 2021 / 042530 relates to aspects of artificial intelligence (AI)-based ground truth generation for object detection and tracking in image sequences.

[0008] Therefore, there remains a need for improved training of object classifiers. Summary of the Invention

[0009] The purpose of the embodiments herein is to address the above problems.

[0010] Generally speaking, in accordance with the inventive concepts disclosed herein, improved training of an image classifier is achieved by retraining a pre-trained object classifier, which is then achieved in accordance with the inventive concepts disclosed herein by annotating instances of the tracked object to belong to an object class based on the fact that another instance of the same tracked object has already been verified to belong to the same object class.

[0011] According to a first aspect, the inventive concept is defined by a method for retraining a pre-trained object classifier. The method is performed by a system including a processing circuit. The method includes acquiring a stream of image frames of a scene, each of the image frames depicting an instance of a tracked object. The tracked object is the same object that is tracked as it moves through the scene. The method includes classifying each instance of the tracked object with a level of confidence as belonging to an object class. The method includes verifying that the level of confidence of at least one and only one of the instances of the tracked object is higher than a threshold confidence value. This may ensure that at least one of the instances of the tracked object is classified with high confidence into only one object class. If the threshold confidence value is verified, the method includes annotating all instances of the tracked object in the stream of image frames as belonging with high confidence to only one object class (i.e., the object class for which the level of confidence of at least one of the instances of the tracked object is higher than the threshold confidence value), resulting in annotated instances of the tracked object. The method includes retraining a pre-trained object classifier using at least some of the annotated instances of the tracked object.

[0012] According to a second aspect, the inventive concept is defined by a system for retraining a pre-trained object classifier. The system includes a processing circuit configured to cause the system to acquire a stream of image frames of a scene, each of the image frames depicting an instance of a tracked object, the tracked object being the same object that is tracked as it moves through the scene. The processing circuit is configured to cause the system to classify each instance of the tracked object with a level of confidence as belonging to an object class. The processing circuit is configured to cause the system to verify that the level of confidence of at least one of the instances of the tracked object and of exactly one object class is higher than a threshold confidence value. This can ensure that at least one instance of the tracked object is classified into exactly one object class with high confidence. If the threshold confidence value is verified to be higher, the processing circuitry is configured to cause the system to annotate all instances of the tracked object in the stream of image frames as belonging with high confidence to only one object class (i.e., the object class for which at least one of the instances of the tracked object has a confidence level higher than the threshold confidence value), resulting in annotated instances of the tracked object. The processing circuitry is configured to cause the system to retrain a pre-trained object classifier using at least some of the annotated instances of the tracked object.

[0013] According to a third aspect, the inventive concept is defined by a system for retraining a pre-trained object classifier. The system comprises an acquirer module configured to acquire a stream of image frames of a scene, each of which depicts an instance of a tracked object, the tracked object being the same object that is tracked as it moves through the scene. The system comprises a classifier module configured to classify each instance of the tracked object with a level of confidence as belonging to an object class. The system comprises a verifier module configured to verify that the confidence level of at least one of the instances of the tracked object, and of only one object class, is higher than a threshold confidence value. This may ensure that at least one of the instances of the tracked object is classified into only one object class with high confidence. The system comprises a processing circuit configured to annotate all instances of the tracked object in the stream of image frames as belonging with high confidence to only one object class (i.e., an object class for which at least one of the instances of the tracked object has a confidence level higher than a threshold confidence value), resulting in annotated instances of the tracked object. The system comprises a retrainer module configured to retrain a pre-trained object classifier on at least some of the annotated instances of the tracked object.

[0014] According to a fourth aspect, the inventive concept is defined by a computer program for retraining a pre-trained object classifier, the computer program comprising computer program code which, when executed on a system, causes the system to perform the method according to the first aspect.

[0015] According to a fifth aspect, the inventive concept is defined by a computer program product comprising a computer program according to the fourth aspect, and a computer-readable storage medium on which the computer program is stored. The computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0016] Advantageously, these aspects provide for improved training of pre-trained object classifiers.

[0017] The annotated instances of tracked objects can also be used to train or retrain object classifiers other than the pre-trained object classifiers described in the above aspects. Thus, advantageously, these aspects provide a means for automatically generating large sets of annotated training data from sets having only partially annotated training data.

[0018] Other objects, features, and advantages of the accompanying embodiments will become apparent from the following detailed disclosure, from the attached dependent claims, and from the drawings.

[0019] In general, all terms used in the claims should be interpreted according to their ordinary meaning in the art unless otherwise expressly defined herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." should be interpreted broadly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless expressly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0020] The inventive concept will now be described, by way of example, with reference to the accompanying drawings. [Brief explanation of the drawings]

[0021] [Figure 1] 1 illustrates a schematic diagram of a system according to an embodiment; [Figure 2] FIG. 1 is a block diagram of a system according to an embodiment. [Figure 3] 1 is a flowchart of a method according to an embodiment. [Figure 4] 1 illustrates schematically the annotation of a stream of image frames according to an embodiment; [Figure 5] FIG. 2 is a schematic diagram illustrating functional units of a system according to one embodiment. [Figure 6] 1 illustrates an example of a computer program product comprising a computer-readable storage medium according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] The inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concepts are shown. However, the inventive concepts may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concepts to those skilled in the art. Like numbers refer to like elements throughout the description. Any steps or features shown with dashed lines should be considered optional.

[0023] As noted above, there remains a need for improved training of object classifiers.

[0024] Generally speaking, as realized by the inventors of the inventive concepts disclosed herein, the classification results of an object classifier for an object may depend on the angle at which the image frames of the object are captured, the lighting conditions under which the image frames of the object are captured, the image resolution of the image frames, etc. Accordingly, some angles, lighting conditions, image resolutions, etc. will yield a higher confidence level than other angles, lighting conditions, image resolutions, etc.

[0025] This means that as a tracked object moves along a path through a scene, during some parts of the path the tracked object is likely to be relatively easy to classify by the object classifier used, and during some other parts of the path the object is likely to be relatively difficult to classify using the same object classifier due to changes in angle, lighting conditions, etc. However, this fact cannot be exploited because the object tracker and object classifier typically operate independently of each other.

[0026] Tracked objects that are relatively easy to classify generally result in a high confidence value, while tracked objects that are relatively difficult to classify generally result in a low confidence value for the object in that particular image frame. Because it is unlikely that an object will change from one object class to another, e.g., from “person” to “truck,” while moving through a scene, this means that if an object in any one instance is classified as a “person” with a confidence level, e.g., a high confidence, the underlying physical object that the tracked object represents will remain a “person” even if it is more difficult to classify the tracked object at a later stage in its path, and therefore in a later image frame; e.g., the tracked object will still be classified as a “person” with a low confidence. This realization is exploited by the inventive concepts disclosed herein. Note that, herein, it is the task of the object classifier to classify tracked objects with a certain level of confidence as belonging to any given object class. The embodiments disclosed herein are based on the assumption that the object classifier is operating correctly and does not provide false positives in this regard, thus representing the normal behavior of any object classifier.

[0027] In particular, in accordance with the inventive concepts disclosed herein, this implementation is utilized to generate new classified training data for a pre-trained object classifier. Accordingly, at least some of the embodiments disclosed herein generally relate to the automated accumulation and annotation of training data for refined object classification or for a particular camera installation. The embodiments disclosed herein particularly relate to mechanisms for retraining a pre-trained object classifier. To achieve such mechanisms, there are provided systems, methods executed by the systems, e.g., computer program products including code in the form of a computer program that, when executed on the system, causes the system to perform the methods.

[0028] FIG. 1 is a schematic diagram illustrating a system 110 to which embodiments presented herein can be applied. The system 110 includes at least one camera device 120a, 120b. Each camera device 120a, 120b is configured to capture image frames depicting a scene 160 within a respective field of view 150a, 150b. According to the illustrative example of FIG. 1, the scene 160 includes a walking man. The walking man represents a tracked object 170, as surrounded by a bounding box. The tracked object 170 is a target object being tracked using object tracking. Generally speaking, object tracking refers to a technique for estimating or predicting the position (and possibly other characteristics) of a target object in each successive frame within a video segment, defined by a stream composed of a limited number of image frames, once an initial position of the target object is defined. Object tracking can be considered a deep learning process in which the movement of the target object is tracked in this manner. Object tracking typically involves object detection, where an algorithm detects, classifies, and annotates the target object. Each detected object may be assigned a (unique) identification. The detected objects are then tracked as they move through the image frames (while storing associated information). Object detection will only work if the target object is visible in the image frames. However, as disclosed further below, even if the target object is partially obscured in some of the image frames, once the target object is detected, it may still be possible to track the target object both forward and backward in the stream of image frames. The position of the tracked target object in two or more image frames may be considered to represent the tracking of the tracked object 170. Thus, a trajectory may be constructed by tracking the position of the tracked object 170 from image frame to image frame. However, because the tracked object 170 may be partially obscured in some of the image frames, the confidence level may change from image frame to image frame.

[0029] In some examples, each camera device 120a, 120b is a digital camera device and / or is capable of pan, tilt, and zoom (PTZ) and therefore may be considered a (digital) PTZ camera device. System 110 is configured to communicate with a user interface 130 for displaying captured image frames. Furthermore, system 110 is configured to encode images so that they can be decoded using any known video encoding standard, such as any of High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2, Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10, H.266, MPEG-I Part 3, and Versatile Video Coding (VVC), also known as Future Video Coding (FVC), VP9, ​​VP10, and AOMedia Video 1 (AV1), to name just a few. In this regard, the encoding may be performed in direct cooperation with the camera devices 120a, 120b that capture the image frames, or may be performed by a separate entity and then, at least temporarily, stored in the database 116. The system 110 further comprises a first entity 114a and a second entity 114b. Each of the first entity 114a and the second entity 114b may be implemented on either the same computing device or separate computing devices. The first entity 114a may be configured, for example, to obtain a stream of image frames of the scene 160 and provide instances of the tracked object 170 to the second entity 114b. Further details of the first entity 114a are disclosed with reference to FIG. 2. The second entity 114b may be an object classifier 114b. The object classifier 114b is assumed to be pre-trained and is therefore hereinafter referred to as the pre-trained object classifier 114b.System 110 may include additional entities, functions, nodes, and devices, as represented by device 118. Examples of such additional entities, functions, nodes, and devices are computing devices, communication devices, servers, sensors, etc.

[0030] In some aspects, the first entity 114a, the second entity 114b, and the database 116 define a video management system (VMS). Accordingly, the first entity 114a, the second entity 114b, and the database 116 are considered part of the system 110. The first entity 114a, the second entity 114b, and the database 116 are operably connected to the camera devices 120a, 120b via a network 112. The network 112 may be wired, wireless, or partially wired and partially wireless. In some examples, the system 110 includes a communication interface 520 (as in FIG. 5 ) configured to communicate with a user interface 130 to display image frames. Reference numeral 140 indicates an exemplary connection between the system 110 and the user interface 130. The connection 140 may be wired, wireless, or partially wired and partially wireless. A user 180 may interact with the user interface 130. It will be appreciated that user interface 130 is at least partially a visual user interface, as it is configured to display image frames captured by camera devices 120a, 120b.

[0031] An embodiment for retraining a pre-trained object classifier 114b will now be disclosed with parallel reference to Figures 2 and 3. Figure 2 is a block diagram of a system 110 according to an embodiment. Figure 3 is a flow chart illustrating an embodiment of a method for retraining a pre-trained object classifier 114b. The method is performed by the system 110. The method is advantageously provided as a computer program 620 (see Figure 6).

[0032] S102: A stream of image frames of the scene 160 is acquired. The stream of image frames may be acquired by an acquirer module 410. Each of the image frames depicts an instance of a tracked object 170. The tracked object 170 is the same object 170 that is tracked as it moves within the scene 160. The stream of image frames may be acquired by the acquirer module 410 from at least one camera device 120a, 120b, as in FIG. 2 .

[0033] S104: Each instance of the tracked object 170 is classified as belonging to an object class with a level of confidence. The classification may be performed by the classifier module 420.

[0034] S106: It is verified that the confidence level of at least one of the instances of the tracked object 170 and of only one object class is higher than a threshold confidence value. The verification may be performed by the verifier module 430. This may ensure that at least one of the instances of the tracked object 170 is classified into only one object class with high confidence. Therefore, the confidence level of the instance of the tracked object 170 used as a reference must be higher than the threshold confidence value of only one object class. This threshold confidence value may be adjusted upward or downward as needed to change the accuracy of retraining the pre-trained object classifier 114b. For example, the accuracy may be increased by setting a very high threshold confidence value (e.g., 0.95 on a scale of 0.0 to 1.0). Furthermore, if it is observed that the amount of false indications produced is higher than desired, the threshold confidence value may be adjusted upward to increase the accuracy of retraining the pre-trained object classifier 114b.

[0035] S118: All instances of the tracked object 170 in the stream of image frames are annotated with high confidence as belonging to only one object class (i.e., the object class for which the confidence level of at least one of the instances of the tracked object 170 is higher than a threshold confidence value). This annotation results in an annotated instance of the tracked object 170. The annotation may be performed by the annotator module 440.

[0036] If it can be successfully verified that the confidence level of at least one of the instances of the tracked object 170 is higher than the threshold confidence value, one or more instances of the same object in other image frames that were only classified with a medium or low confidence level can be extracted and annotated as if they were classified with a high confidence level.

[0037] Thus, at this point, S118 is entered only if it can be successfully verified that the confidence level of at least one of the instances of tracked object 170 is higher than the threshold confidence value in S106. As mentioned above, S106 can ensure that tracked object 170 is classified with high confidence into only one object class before entering S118.

[0038] S122: The pre-trained object classifier 114b is retrained on at least some of the annotated instances of the tracked object 170. The retraining may be performed by the retrainer module 460.

[0039] Thus, the pre-trained object classifier 114b can be retrained with additional annotated instances of the tracked object 170. If this is done to the object classification model, then the object classification model will be better able over time to provide improved classification of objects in areas of the scene 160 where classification was originally difficult, i.e., areas where the classification was given low or medium confidence. An example of this is provided below with reference to FIG. 4.

[0040] Therefore, if it is verified that the tracked object 170 in S106 is classified as belonging to only one object class with a high degree of confidence in at least one image frame, the same tracked object 170 from other image frames will be used to retrain the pre-trained object classifier 114b. Similarly, if the tracked object 170 is not classified as belonging to one object class with a high degree of confidence, the tracked object 170 in other image frames will not be used to retrain the pre-trained object classifier 114b.

[0041] Next, embodiments relating to further details of the retraining of the pre-trained object classifier 114b performed by the system 110 will be disclosed.

[0042] 2, classification and retraining may be performed on separate devices. That is, in some embodiments, classification is performed on a first entity 114a and retraining is performed on a second entity 114b that is physically separate from the first entity 114a. The second entity may be a (pre-trained) object classifier 114b.

[0043] Further validation may be performed before all instances of tracked object 170 in the stream of image frames are annotated as belonging to the object class, and thus before entering S118. Different embodiments in this regard will now be described.

[0044] It may be possible that some tracked objects 170 are classified as belonging to more than one object class, with each classification having its own level of confidence. Thus, in some aspects, it is verified that only one of these object classes has a high level of confidence. Thus, in some embodiments, at least some of the instances of tracked objects 170 are classified as belonging to additional object classes at additional levels of confidence, and the method includes: S108: verifying that the further confidence level is lower than a threshold confidence value for at least some of the instances of the tracked object 170. The verification may be performed by the verifier module 430.

[0045] Thus, if the tracked object 170 is classified with high confidence into two or more different object classes, the tracked object 170 will not be used to retrain the pre-trained object classifier 114b. The same is true, i.e., if the tracked object 170 is not classified with high confidence into any object class, the tracked object 170 will not be used for retraining.

[0046] In some aspects, the path along which the tracked object 170 moves from one image frame to the next is also tracked. If the tracked object 170 has been classified at least once with high confidence as belonging to only one object class, the path can be used to retrain the pre-trained object classifier 114b. Thus, in some embodiments, the tracked object 170 moves along a path in the stream of image frames, and the path is tracked as the tracked object 170 is tracked.

[0047] In some aspects, it is verified that the path itself can be tracked with a high degree of accuracy, i.e., in some embodiments, the path is tracked with a level of accuracy and the method comprises: S110: Verifying that the level of certainty is higher than a threshold certainty value. The verification may be performed by the verifier module 430.

[0048] In some aspects, it is verified that the path has not split or merged. If the path has split and / or merged at least once, this may indicate that the level of confidence with which the path was tracked is below a threshold confidence value. This may be the case if the path is suspected to have merged from two or more other paths or split into two or more other paths. If so, the path will be determined to have low confidence and will not be used to retrain the pre-trained object classifier 114b. Thus, in some embodiments, the method includes: S112: It is further included to verify that the path is not split into at least two paths and is not joined from at least two paths in the stream of image frames. In other words, it is verified that the path does not have any splits or joins and therefore constitutes one single path. The verification may be performed by the verifier module 430.

[0049] The same principle can also be applied if the tracked object 170 is suspected to have strange size behavior that may be suspected to be caused by shadowing, mirror effects, or the like. If so, the tracked object 170 will be assumed to be classified with low confidence and will not be used to retrain the pre-trained object classifier 114b. Thus, in some embodiments, the tracked object 170 has a size in the image frame and the method S114: Verifying that the size of the tracked object 170 does not change by more than a threshold size value within the stream of image frames. The verification may be performed by the verifier module 430.

[0050] Because the apparent size of the tracked object 170 depends on the distance between the tracked object 170 and the camera devices 120a, 120b, compensation of the size of the tracked object 170 for this distance may be performed as part of the verification of S114. In particular, in some embodiments, the size of the tracked object 170 is adjusted by a distance-dependent compensation factor determined as a function of the distance between the tracked object 170 and the camera devices 120a, 120b that captured the stream of image frames of the scene 160 upon verifying that the size of the tracked object 170 does not change by more than a threshold size value within the stream of image frames 210a:210c.

[0051] As shown in and disclosed with reference to FIG. 1, the scene 160 may be captured by one or more camera devices 120a, 120b. If the scene 160 is captured by two or more camera devices 120a, 120b, then the stream of image frames in which the tracked object 170 is present may also be captured by two or more different camera devices 120a, 120b. Thus, in some embodiments, the stream of image frames originates from image frames captured by at least two camera devices 120a, 120b. If tracking of the tracked object 170 is performed locally on each of the camera devices 120a, 120b, this may require that information about the tracked object 170 be communicated between the at least two camera devices 120a, 120b. In other examples, tracking of the tracked object 170 is performed centrally on image frames received from all of the at least two camera devices 120a, 120b. The latter does not require any information about the tracked object 170 to be exchanged between the at least two camera devices 120a, 120b.

[0052] In some aspects, for example, when the stream of image frames originates from image frames captured by at least two camera devices 120a, 120b, there is a risk that tracking of the tracked object 170 may be lost and / or the classification of the tracked object 170 may change from image frame to image frame, among other examples. Thus, in some embodiments, the method includes: S116: Verifying that the object class of the instance of the tracked object 170 does not change within the stream of image frames. The verification may be performed by the verifier module 430.

[0053] In certain aspects, to avoid training bias (e.g., machine learning bias, algorithm bias, or artificial intelligence bias), the pre-trained object classifier 114b is not re-trained using tracked objects 170 that have already been classified with a high confidence level. There are various ways to achieve this. A first way is to explicitly exclude tracked objects 170 that have already been classified with a high confidence level from re-training. In particular, in some embodiments, the pre-trained object classifier 114b is re-trained only with annotated instances of tracked objects 170 that have been verified to have a confidence level no higher than a threshold confidence value. A second way is to set a low weighting value for tracked objects 170 that have already been classified with a high confidence level during re-training. In this way, tracked objects 170 that have already been classified with a high confidence level can be implicitly excluded from re-training. That is, in some embodiments, each annotated instance of the tracked object 170 is assigned a respective weighting value according to which the annotated instances of the tracked object 170 are weighted when the pre-trained object classifier 114b is retrained, with the weighting value of an annotated instance of the tracked object 170 verified to have a level of confidence higher than the threshold confidence value being lower than the weighting value of an annotated instance of the tracked object 170 verified to have a level of confidence not higher than the threshold confidence value. The weighting values ​​thereby make the object classifier 114b less influenced during retraining by tracked objects 170 that have already been classified with a high level of confidence.

[0054] There may be further different uses of the annotated instances of the tracked objects 170 other than retraining the pre-trained object classifier 114b. In some aspects, the annotated instances of the tracked objects 170 may be collected in the database 116 and / or provided to the further device 118. Thus, in some embodiments, the method includes: S120: The annotated instance of the tracked object 170 is provided to the database 116 and / or the further device 118. The annotated instance may be provided to the database 116 and / or the further device 118 by the provider module 450.

[0055] This also allows other pre-trained object classifiers to benefit from the annotated instances of the tracked object 170.

[0056] Reference is now made to FIG. 4, which schematically illustrates two streams 200, 200′ of image frames 210a:210c. Each stream 200, 200′ consists of a limited sequence of image frames. Each such stream may correspond to a captured video segment of a scene 160 in which one or more objects 170 have been tracked. For illustrative purposes, each stream 200, 200′ is shown in FIG. 4 to consist of three image frames 210a:210c. In particular, FIG. 4 illustrates the first stream 200 of image frames 210a:210c, in which the tracked object 170 has been classified with a level of confidence higher than a threshold confidence value only in image frame 210c. In FIG. 4, it is assumed, for illustrative purposes, that the tracked object 170 has been classified with a level of confidence higher than a threshold confidence value as belonging to the object class “man walking” only in image frame 210c. This is because in image frames 210a, 210b, the bottom half of tracked object 170 is obscured by vegetation, making it difficult for the object classifier to determine whether a person is walking. FIG. 4 also shows a second stream 200′ of the same image frames 210a:210c, but after application of embodiments disclosed herein, as indicated by the arrow labeled “annotation.” By applying embodiments disclosed herein, instances of tracked image 170 in image frames 210a, 210b are also annotated as belonging to the same object class as instances of tracked image 170 in image frame 210c. This is indicated by arrows 220a, 220b in FIG. 4. Thus, instances of tracked object 170 in image frames 210a, 210b are also annotated as belonging to the object class “man walking” with a level of confidence higher than the threshold confidence value, despite the presence of partially obscuring vegetation. In other words, the instances of tracked object 170 in image frames 210a, 210b inherit the annotations of the instance of tracked object 170 in image frame 210c.The instances of the tracked object 170 in image frames 210a, 210b can then be used to retrain the pre-trained object classifier 114b. Thus, for this method to work, it is only necessary that the classification of the object 170 in one image frame 210c of stream 200 be higher than a threshold (as shown in FIG. 4 as "High Confidence"). As a result of performing the method, the object 170 will also be annotated in image frames 210a and 210b in stream 200'. Now, because the object 170 is also annotated in image frames 210a and 210b in stream 200', this means that retraining the pre-trained object classifier 114b on stream 200' will actually improve the pre-trained object classifier 114b. More precisely, when the pre-trained object classifier 114b is run on a new stream of image frames with partially obscured objects belonging to the object class "man walking", such objects can also be tracked and annotated as being classified with a high level of confidence.

[0057] 5 illustrates components of system 110 in terms of several functional units according to one embodiment. Processing circuitry 510 is provided using any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers, digital signal processors (DSPs), etc. capable of executing software instructions stored, for example, in a computer program product 610 (as in FIG. 6) in the form of a storage medium 530. Processing circuitry 510 may further be provided as at least one application specific integrated circuit (ASIC) or field programmable gate array (FPGA).

[0058] In particular, processing circuitry 510 is configured to cause system 110 to perform a set of operations or steps, as described above. For example, storage medium 530 may store a set of operations, and processing circuitry 510 may be configured to retrieve the set of operations from storage medium 530 to cause system 110 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0059] Thus, the processing circuitry 510 is configured to thereby perform the methods disclosed herein. The storage medium 530 may also comprise persistent storage, which may be, for example, any single or combination of magnetic memory, optical memory, solid-state memory, or even remotely mounted memory. The system 110 may further comprise a communication interface 520 configured at least to communicate with additional devices, functions, nodes, and devices. Thus, the communication interface 520 may comprise one or more transmitters and receivers comprising analog and digital components. The processing circuitry 510 controls the general operation of the system 110, for example, by sending data and control signals to the communication interface 520 and the storage medium 530, receiving data and reports from the communication interface 520, and retrieving data and instructions from the storage medium 530. Other components and associated functions of the system 110 have been omitted so as not to obscure the concepts presented herein.

[0060] 6 illustrates an example of a computer program product 610 comprising a computer-readable storage medium 630. The computer-readable storage medium 630 may store a computer program 620 that may cause the processing circuitry 210 and entities and devices operatively coupled thereto, such as the communications interface 220 and the storage medium 230, to perform methods according to embodiments described herein. Thus, the computer program 620 and / or the computer program product 610 may provide means for performing any of the steps disclosed herein.

[0061] 6, computer program product 610 is shown as an optical disc, such as a CD (compact disc), DVD (digital versatile disc), or Blu-ray disc. Computer program product 610 may also be embodied as memory, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), or more specifically as a non-volatile storage medium of the device in an external memory, such as a USB (universal serial bus) memory or flash memory, such as a compact flash memory. Thus, although computer program 620 is shown schematically herein as tracks on a depicted optical disc, computer program 620 may be stored in any manner suitable for computer program product 610.

[0062] The inventive concept has been described above primarily with reference to certain embodiments. However, as will be readily appreciated by those skilled in the art, other embodiments besides those disclosed above are equally possible within the scope of the inventive concept as defined by the appended claims.

Claims

1. 1. A method for retraining a pre-trained object classifier (114b), the method being performed by a system (110), the system (110) comprising a processing circuit (510), the method comprising: acquiring (S102) a stream (200, 200') of image frames (210a:210c) of a scene (160), each of the image frames (210a:210c) depicting an instance of a tracked object (170), the tracked object (170) being the same object (170) that is tracked as it moves through the scene (160); Classifying (S104) each instance of the tracked object (170) with a level of confidence that it belongs to an object class; verifying (S106) that the confidence level of at least one of the instances of the tracked object (170) and of only one object class is higher than a threshold confidence value, thereby ensuring that at least one of the instances of the tracked object (170) is classified into the only one object class with high confidence; and if the confidence level is verified to be higher than the threshold confidence value, annotating (S118) all instances of the tracked object (170) in the stream (200, 200') of image frames (210a:210c) as belonging with high confidence to the single object class, resulting in annotated instances of the tracked object (170); retraining (S122) the pre-trained object classifier (114b) using at least some of the annotated instances of the tracked object (170); A method comprising:

2. At least some of the instances of the tracked object (170) are classified as belonging to additional object classes with additional levels of confidence, the method comprising: verifying (S108) that the level of further confidence is lower than the threshold confidence value for at least some of the instances of the tracked object (170); The method of claim 1 further comprising:

3. The method comprises: Verifying (S116) that the object class of the instance of the tracked object (170) does not change within the stream (200, 200') of image frames (210a: 210c). The method of claim 1 or 2, further comprising:

4. 3. The method of claim 1, wherein the tracked object (170) moves along a path in the stream (200, 200') of image frames (210a:210c), the path being tracked when the tracked object (170) is tracked.

5. The path is tracked with a level of accuracy, the method comprising: Verifying that the level of certainty is greater than a threshold certainty value (S110). The method of claim 4 further comprising:

6. The method comprises: Verifying (S112) that the path has not split into at least two paths and has not merged from at least two paths in the stream (200, 200') of image frames (210a: 210c). The method of claim 4 further comprising:

7. The tracked object (170) has a size within the image frame (210a:210c), and the method comprises: Verifying (S114) that the size of the tracked object (170) does not change by more than a threshold size value within the stream (200, 200') of image frames (210a: 210c). The method of claim 1 or 2, further comprising:

8. 8. The method of claim 7, wherein the size of the tracked object (170) is adjusted by a distance-dependent compensation factor determined as a function of a distance between the tracked object (170) and a camera device (120a, 120b) that captured the stream of image frames (210a:210c) of the scene (160) upon verifying that the size of the tracked object (170) does not change by more than the threshold size value within the stream of image frames (210a:210c).

9. 3. The method of claim 1 or 2, wherein the pre-trained object classifier (114b) is retrained only with the annotated instances of the tracked object (170) for which the confidence level is verified to be not higher than the threshold confidence value.

10. 3. The method of claim 1, wherein each of the annotated instances of the tracked object (170) is assigned a respective weighting value according to which the annotated instances of the tracked object (170) are weighted when the pre-trained object classifier (114b) is retrained, and the weighting value of the annotated instances of the tracked object (170) whose confidence level is verified to be higher than the threshold confidence value is lower than the weighting value of the annotated instances of the tracked object (170) whose confidence level is verified to be not higher than the threshold confidence value.

11. The method comprises: providing (S120) the annotated instance of the tracked object (170) to a database (116) and / or a further device (118); The method of claim 1 or 2, further comprising:

12. The method of claim 1 or 2, wherein the stream (200, 200') of image frames (210a:210c) originates from the image frames (210a:210c) captured by at least two camera devices (120a, 120b).

13. 3. The method of claim 1, wherein the classifying is performed at a first entity (114a) and the retraining is performed at a second entity (114b) that is physically separated from the first entity (114a).

14. 1. A system (110) for retraining a pre-trained object classifier (114b), the system (110) comprising a processing circuit (510), the processing circuit causing the system (110) to: acquiring a stream (200, 200') of image frames (210a:210c) of a scene (160), each of the image frames (210a:210c) depicting an instance of a tracked object (170), the tracked object (170) being the same object (170) that is tracked as it moves within the scene (160); classifying each instance of the tracked object (170) with a level of confidence that it belongs to an object class; verifying that the confidence level of at least one of the instances of the tracked object (170) and of only one object class is higher than a threshold confidence value, thereby ensuring that at least one of the instances of the tracked object (170) is classified into the only one object class with high confidence; and if the confidence level is verified to be higher than the threshold confidence value, annotating all instances of the tracked object (170) in the stream (200, 200') of image frames (210a:210c) as belonging with high confidence to the single object class, resulting in annotated instances of the tracked object (170); retraining the pre-trained object classifier (114b) using at least some of the annotated instances of the tracked object (170); The system (110) is configured to:

15. A non-transitory computer-readable storage medium (630) having stored thereon a computer program (620) for retraining a pre-trained object classifier (114b), the computer program comprising computer code that, when executed on a processing circuit (510) of a system (110), causes the system (110) to: acquiring (S102) a stream (200, 200') of image frames (210a:210c) of a scene (160), each of the image frames (210a:210c) depicting an instance of a tracked object (170), the tracked object (170) being the same object (170) that is tracked as it moves through the scene (160); Classifying (S104) each instance of the tracked object (170) with a level of confidence that it belongs to an object class; verifying (S106) that the confidence level of at least one of the instances of the tracked object (170) and of only one object class is higher than a threshold confidence value, thereby ensuring that at least one of the instances of the tracked object (170) is classified into the only one object class with high confidence; and if the confidence level is verified to be higher than the threshold confidence value, annotating (S118) all instances of the tracked object (170) in the stream (200, 200') of image frames (210a:210c) as belonging with high confidence to the single object class, resulting in annotated instances of the tracked object (170); retraining (S122) the pre-trained object classifier (114b) using at least some of the annotated instances of the tracked object (170); A non-transitory computer-readable storage medium (630).

Images