Method and system for detecting objects of a first object type in a video sequence

Abstract

The present disclosure relates to a method and system for detecting objects of a first object type in a video sequence. A first algorithm is used to detect regions or objects with an uncertain object type status in a scene captured in a video stream. A second algorithm is used to provide a background model of the video sequence. The background model is used to check, for a region or object with an uncertain object type status, whether the region or object is considered to be part of the background or foreground in the video sequence. If the region or object is determined to belong to the foreground, the region or object is classified as the first object type. If the region or object is determined not to belong to the foreground, the region or object is not classified as the first object type.

Description

Technical Field

[0001] The present invention relates to object detection, and more particularly to a method, system, and non-transitory computer-readable storage medium for detecting an object of a first object type in a first image frame of a plurality of image frames in a video sequence of a captured scene. Background Technology

[0002] Object detection in video surveillance systems has evolved over the years into a crucial component of various applications such as security, traffic monitoring, retail analytics, and smart cities. The primary objective of these systems is to identify and track objects of interest, such as people, vehicles, or animals, within the video stream.

[0003] Object classification is typically an integral part of object detection systems. Object detection involves identifying objects of interest within an image or video frame and assigning them to a specific category or classification. The combination of object localization (finding the location of objects) and object classification (identifying the classification of objects) ensures a complete object detection system.

[0004] One problem with object detection systems is the misclassification of objects, i.e., false positives or false negatives. Such classification can negatively impact user experience, for example. In systems where human operators review or interact with video output, misclassified objects can lead to confusion, fatigue, and frustration. This can reduce user trust in the system and may cause operators to miss objects of genuine interest due to an increased number of false alarms.

[0005] Therefore, improvements are needed in this area. Summary of the Invention

[0006] In view of the foregoing, as stated in the independent claims, it would be beneficial to resolve or at least reduce one or more of the aforementioned disadvantages.

[0007] According to a first aspect of the invention, a method is provided for detecting an object of a first object type in a first image frame of a plurality of image frames in a video sequence capturing a scene, the method comprising: for each of the plurality of image frames: analyzing the image frame using a first algorithm to identify an object of the first object type in the image frame, the first algorithm calculating for each of a plurality of regions in the scene a probability that image data corresponding to a region is part of an object of the first object type in the image frame, the first algorithm having a probability threshold to determine whether the image data is part of an object of the first object type, and storing an indicator of the difference between the probability that the image data is part of an object of the first object type and the probability threshold in a list of indicators associated with the region.

[0008] The method further includes detecting an object of a first object type in a first image frame, the first image frame being the last image frame among a plurality of image frames, wherein the detection includes: providing a background model of a video sequence, the background model defining whether image data corresponding to a region in the first image frame belongs to the background or the foreground for each region in a plurality of regions in the scene, the background model defining this by applying a second algorithm different from the first algorithm to at least some of the image frames in the plurality of image frames; for each region of the scene, identifying whether the image data corresponding to the region has an uncertain object type state in the first image frame based on a list of associated indicators; when an uncertain object type state is identified in the image data: if the background model defines the image data as belonging to the background in the first image, determining that the image data is not part of an object of the first object type; if the background model defines the image data as belonging to the foreground in the first image, determining that the image data is part of an object of the first object type.

[0009] In object classification applications designed to detect specific object types, problems can arise when object classification alternates between frames. Some objects may have an appearance similar to the target object type, causing fluctuations in the probability of the classification algorithm. These probabilities may exceed a threshold, classifying the object as the target type in one frame, and then drop below the threshold, classifying the object as a different type in the next frame. Factors contributing to these variations may include the object's viewpoint, image sensor noise, minor changes in segmentation mask determination, or variations in the object's appearance.

[0010] Such inconsistent categorization can confuse operators of monitoring outputs and may negatively impact other issues such as system performance, decision-making, and privacy.

[0011] Advantageously, using two different algorithms to determine object types increases the robustness of this method. The second algorithm, which provides a background model of the video sequence, can be seen as a backup in the case where the first algorithm outputs different evaluations (probabilities above and below a probability threshold) for multiple image frames of the corresponding image content. Such fluctuating outputs between multiple image frames of a specific region of the scene may result in a region of the scene, as depicted in the currently processed image frame (referred to as the first image frame, the latest image frame in the video sequence), acquiring an "uncertain object type state." By using previous frames in the video sequence to determine whether a region of the scene, as depicted in the first frame (i.e., the image data depicting the region in the first frame), should acquire an uncertain object type state, prediction of future states is avoided, resulting in a low-complexity and robust method.

[0012] It is important to clarify that the term "first image frame" as used herein does not refer to the order of frames in a video sequence. Rather, it is merely a means of distinguishing one frame from another for the purposes of describing or claiming protection of the invention. The order of frames in a sequence is determined by their temporal relationship to each other, with the first frame being captured or displayed before the second, and so on, until the last frame is reached.

[0013] In the context of this specification, the term "uncertain object type state" is understood to mean that, when analyzing multiple image frames, the output from the first algorithm results in a mixture of classification results, for example, a region of a scene depicted in some image frames is classified as not being part of an object of the first object type, while in other image frames, the region is classified as being part of an object of the first object type.

[0014] The degree of mixing that results in uncertain object type states (e.g., the ratio between positive and negative classifications, the difference between the probability that image data is part of an object of the first object type and the probability threshold) depends on the use case and application requirements.

[0015] When an uncertain object type state is determined for specific image data of the first image, the background model is used to determine whether an object of the first object type is detected in the image data.

[0016] Advantageously, because the background model can determine that image data depicting static objects in the first image frame belongs to the background, this method can avoid false positives for static objects that are similar in appearance to objects of the first object type. Furthermore, the background model can prevent false negatives for moving objects belonging to the first object type, even when their appearance in one or more image frames does not match the evaluation criteria of the first algorithm. This is because the background model can determine that image data representing moving objects in the first image frame belongs to the foreground, thus avoiding false negatives.

[0017] Additionally, as new image frames are added to the video sequence, the list of indicators can potentially be continuously updated using a first-in, first-out (FIFO) method. As a result, this method can effectively adapt to new objects or other scene changes captured within the video sequence.

[0018] In some embodiments, the method further includes the following steps: when image data is identified as not having an uncertain object type state: when the first algorithm determines that the image data is part of an object of a first object type, determine that the image data is part of an object of a first object type; when the first algorithm determines that the image data is not part of an object of a first object type, determine that the image data is not part of an object of a first object type.

[0019] In other words, if the image data in the first image is not identified as having an uncertain object type state, then the output of the first algorithm for the image data is reliable, and if the first algorithm has already made an assessment that the image data is part of an object of the first object type, then the image data is determined to be part of an object of the first object type. Advantageously, this relies on consistent assessments from the first algorithm and can avoid false positives and false negatives.

[0020] In some examples, the step of identifying whether image data has an uncertain object type state in a first image includes: determining, based on a list of associated indicators, whether the absolute value of the difference between the probability that the image data is part of an object of a first object type and a probability threshold exceeds a threshold difference; if the difference exceeds the threshold difference, determining that the image data does not have an uncertain object type state; if the absolute value of the difference does not exceed a threshold distance, determining a distribution measurement between indicators indicating positive differences in the list of associated indicators and indicators indicating negative differences in the list of associated indicators; if the distribution measurement indicates a mixture of positive and negative differences included in the uncertainty range, determining that the image data has an uncertain object type state, and otherwise determining that the data does not have an uncertain object type state.

[0021] Advantageously, the evaluation is relied upon when the first algorithm is sufficiently certain in its assessment of the image data of the first image (the difference between the probability that the image data in the first image is part of an object of the first object type and a probability threshold exceeds a threshold difference), and the image data is not considered to have an uncertain object type state. This embodiment can reduce the complexity of the method because if the probability that the first algorithm assesses the image data of the first image is part of an object of the first object type is sufficiently high or low, there is no need to investigate the remaining list of indicators (i.e., the evaluation performed on previous frames) and check the background model. Moreover, false positives and false negatives can be avoided.

[0022] The threshold difference can depend on the requirements of the application of the implementation method and / or the reliability metric of the first algorithm. The threshold difference can, for example, be set to 0.3 on a scale from 0 to 1. If the probability threshold is 0.5, this means that a probability of 0.8 or higher or 0.2 or lower is considered "sufficiently high" or "sufficiently low," respectively. Other threshold differences are also possible, depending on the use case.

[0023] If the first algorithm is insufficiently deterministic, the list of indicators is analyzed to determine whether the image data is considered to have an uncertain object type state. Any appropriate statistical methods are used to analyze the distribution between positive and negative errors, such as determining the ratio between negative and positive values, the mean of the list, the median of the list, the sum of the values ​​in the list, etc. If the distribution measurement is included within the uncertainty range, the image data is considered to have an uncertain object type state; otherwise, the image data is not considered to have an uncertain object type state.

[0024] Because the list indicates positive and negative differences, and the first algorithm can classify image data into two categories (first object type or not first object type), the uncertainty range typically includes both low and high thresholds. For example, with a defined ratio, the uncertainty range could indicate a mixture between a 20 / 80 distribution and an 80 / 20 distribution for indicators indicating positive and negative differences. This means that if the ratio is between 20 / 80 and 80 / 20 (such as 20 / 80, 30 / 70, 50 / 50, 60 / 40, 80 / 20, but not 100 / 0, 0 / 100, 10 / 90, or 95 / 5), the image data is considered uncertain. Any other suitable ratio, such as 10 / 90, 25 / 75, etc., can be used as a threshold for this range. When using an average value, the range of uncertainty can include lower values ​​of -0.2 and higher values ​​of 0.3, making the average value between -0.2 and +0.3 considered uncertain, but values ​​outside this range (such as -0.3, +0.35, etc.) are not considered uncertain.

[0025] In some embodiments, the step of identifying whether image data has an uncertain object type state in a first image includes: determining a distribution measurement between an indicator indicating positive error in a list of associated indicators and an indicator indicating negative error in a list of associated indicators; determining that the image data has an uncertain object type state when the distribution measurement indicates that a mixture of positive and negative errors is included in the uncertainty range; and otherwise determining that the image data does not have an uncertain object type state.

[0026] In this embodiment, the list of indicators is always analyzed, which provides a more reliable evaluation of the image data. Furthermore, this embodiment allows the list of indicators to be a list of binary values, where positive differences are indicated by a first binary value and negative differences by a second binary value. This reduces the complexity of the method in both storing the list of indicators and determining the distribution measurement of the list.

[0027] In some embodiments, the first algorithm includes a feature extraction process to extract features of objects within an image and represent them as numerical vectors. This means that the first algorithm may include an artificial intelligence (AI) or machine learning (ML) algorithm trained to detect objects of a first object type in an image. AI / ML is a suitable technique for classifying objects in images and can be trained relatively easily with large datasets of images labeled with objects of interest. Suitable AI / ML algorithms include Haar cascades, histogram of oriented gradients (HOG), histogram of local binary patterns (LBPH), convolutional neural networks (CNN), and Transformer-type models (which take sequential data such as video streams as input).

[0028] In some embodiments, the second algorithm is a motion-based background detection algorithm. Examples of such algorithms include those using a time-averaging filter, frame differencing methods, a mean filter, running a Gaussian average, and a background mixture model (e.g., a Gaussian mixture model, GMM). Other possible algorithms include adaptive multi-band binary (AMBB) algorithms and optical flow methods. In cases where the first algorithm, as discussed above, produces uncertain results, using a motion-based background detection algorithm can provide a good alternative.

[0029] In some embodiments, the list of indicators associated with a region is a FIFO list with 5 to 15 values. In other embodiments, the FIFO list may have more values, such as 20, 30, etc. Using a FIFO data structure can provide a simple, efficient, and effective way to compute distribution measurements, as discussed herein, especially in real-time applications where new data points are constantly being added.

[0030] In some embodiments, the method further includes masking or highlighting image data identified as part of an object of a first object type in a first image. When processing these types of applications, the operator may be frustrated if an object is masked or highlighted in one frame but not in the next. This inconsistency can be irritating and draw unnecessary attention. Additionally, in masking applications, there are potential privacy concerns if objects are only occasionally obscured. This method can address or at least reduce one or more of these drawbacks.

[0031] In some embodiments, the method is implemented in a camera, wherein the video sequence is part of a real-time video stream captured by the camera.

[0032] According to a second aspect of the invention, a method is provided for detecting an object of a first object type in a first image frame of a plurality of image frames in a video sequence capturing a scene, the scene including a plurality of objects tracked in the plurality of image frames, the method comprising: for each of the plurality of image frames: analyzing the image frame using a first algorithm to identify an object of the first object type in the image frame, the first algorithm calculating a probability that an object is an object of the first object type for each of the plurality of objects tracked in the image frame, the first algorithm having a probability threshold to determine whether an object is an object of the first object type, and storing an indicator of the difference between the probability that an object is an object of the first object type and the probability threshold in a list of indicators associated with the object.

[0033] The method further includes detecting an object of a first object type in a first image frame, the first image frame being the last image frame among a plurality of image frames, wherein the detection includes: providing a background model of a video sequence, the background model defining for each region in a plurality of regions in the scene whether the image data corresponding to the region in the first image frame belongs to the background or the foreground, the background model being defined by applying a second algorithm different from the first algorithm to at least some of the image frames among the plurality of image frames.

[0034] The method further includes, for each of the multiple objects tracked in the first image, identifying whether the object has an uncertain object type state based on a list of indicators associated with the object; when an object is identified to have an uncertain object type state: determining image data corresponding to the object in the first image frame; determining that the object is not an object of the first object type when the background model limits the image data to the background in the first image frame; and determining that the object is an object of the first object type when the background model limits the image data to the foreground in the first image frame.

[0035] The second aspect of the method is similar to the first concept and achieves the same or corresponding advantages. The difference lies in that, in the second aspect, the evaluation from the first algorithm is associated with objects detected in the video stream, rather than with regions of the scene. Advantageously, the second aspect can handle both static and moving objects. Similar to the first aspect, a background model is used to evaluate whether a tracked object with an uncertain object type state is considered background or foreground in the first image frame. If the image data corresponding to the object is to be considered foreground, the object is determined to be a first object type; otherwise, the object is determined not to be a first object type.

[0036] In some embodiments, each of the tracked objects is located outside a threshold distance from the camera capturing the video sequence of the scene. For example, the first algorithm may have an effective range or detection range within which it can accurately classify objects based on features or characteristics of the objects in the image. This range is typically determined by the camera's resolution and quality, as well as the performance of the object classification algorithm. For objects outside the detection range, classification accuracy may decrease due to factors such as image quality degradation, occlusion, and changes in lighting conditions. In embodiments, the threshold distance may be the same as or close to the detection range of the first algorithm. Using this embodiment, objects sometimes classified as a first object type but not considered background can be determined as belonging to the first object type. This reduces false negatives. This may be particularly important in scenarios where objects of the first object type should be occluded, for example, for privacy reasons.

[0037] The second aspect can generally have the same characteristics and advantages as the first aspect.

[0038] According to a third aspect of the invention, a non-transitory computer-readable storage medium is provided having instructions stored on the non-transitory computer-readable medium, which, when executed on a processing device, are used to implement the method according to the first or second aspect.

[0039] According to a fourth aspect of the invention, a system is provided for detecting an object of a first object type in a first image frame of a plurality of image frames in a video sequence of a captured scene, the system comprising: one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the system to perform the method of the first aspect or the second aspect.

[0040] The third and fourth aspects can usually have the same characteristics and advantages as the first aspect.

[0041] Other objects, features and advantages of the invention will become apparent from the following detailed disclosure, dependent claims and drawings.

[0042] In general, all terms used in the claims should be interpreted according to their conventional meaning in the technical field, unless otherwise expressly defined herein. Unless otherwise expressly stated, all references to “a / the [element, device, component, manner, step, etc.]” ​​should be openly interpreted as referring to at least one instance of an element, device, component, manner, step, etc.

[0043] Unless explicitly stated otherwise, the steps of any method disclosed herein need not be performed in the exact order disclosed. Attached Figure Description

[0044] The above and other objects, features and advantages of the invention will be better understood from the following illustrative and non-limiting detailed description of embodiments of the invention with reference to the accompanying drawings, wherein like reference numerals will be used for similar elements, wherein:

[0045] Figure 1 A video sequence comprising multiple images is shown according to a first embodiment, wherein an object of a first object type is identified by a first algorithm;

[0046] Figure 2 A video sequence comprising multiple images is shown according to a second embodiment, wherein an object of a first object type is identified by a first algorithm;

[0047] Figure 3 It shows Figure 1 Background model of mid-length video sequences;

[0048] Figure 4 It shows Figure 2 Background model of mid-length video sequences;

[0049] Figure 5 A list of indicators according to an embodiment is shown;

[0050] Figure 6 A list of indicators according to an embodiment is shown;

[0051] Figure 7 A flowchart of a method for detecting an object of a first object type in a first image frame among a plurality of image frames, according to an embodiment, is shown;

[0052] Figure 8 A flowchart of a method for detecting an object of a first object type in a first image frame among a plurality of image frames, according to an embodiment, is shown. Detailed Implementation

[0053] Object classification systems may produce false positives or false negatives for various reasons, including imbalanced training data, similar object appearances, occlusion, lighting differences, etc. For example, objects with similar visual features or appearances may confuse the classification system, leading to false positives or false negatives. The model may have difficulty distinguishing objects with similar shapes, textures, or colors, especially under certain lighting conditions or from certain angles.

[0054] In many object classification systems, especially those where output is monitored by operators, consistent false positives or false negatives may go unnoticed and receive little attention. However, inconsistent false positives or false negatives are more difficult to ignore and can lead to user frustration, decreased trust in the system, and reduced adoption of the technology. Furthermore, in real-time applications such as monitoring systems, inconsistent false positives or false negatives can result in unstable decision-making as the system constantly reassesses the presence or absence of objects of a certain type. This can lead to inefficient or potentially dangerous decision-making.

[0055] Figure 1 An example of a video sequence in which the object classification algorithm outputs fluctuating classification for objects is shown. Figure 1 Three images 100a to 100c from a video sequence of a captured scene are shown. The scene includes four objects: a tree 108, a dog 104, a vehicle 106 driving on a road, and a person 102. Images 100a to 100c are further enhanced with highlighted features 110, 112, 114, and 116, which indicate the image regions (image data) in which a first algorithm has identified an object of a first object type in each of image frames 100a to 100c. Figure 1 In the example, the first algorithm is configured to detect faces. This detection can be used for various purposes, such as monitoring when an individual enters a restricted area and alerting the operator, which may also trigger an alarm. Additionally, face detection can be used for privacy purposes, enabling the obscuring or blurring of faces to protect personal identity. It should be noted that, depending on the object classification system being used, the first algorithm can be configured to detect other object types such as vehicles, animals, etc.

[0056] The first algorithm can be configured to identify objects of a first object type in image frames 100a-100c by calculating, for each of multiple regions in the scene captured by image frames 100a-100c, the probability that the image data corresponding to that region is part of an object of a first object type in image frames 100a-100c. The first algorithm may have a probability threshold to determine whether image data is part of an object of a first object type. The first algorithm can be a machine learning or artificial intelligence model, offering various technical advantages such as improved accuracy, adaptability to new data, ability to handle complex relationships, and scalability. Typically, such an algorithm or other suitable algorithm classifies the object type of an object with a determined probability (e.g., 0.7 or 0.2 on a scale from 0 to 1). If the determined probability exceeds the probability threshold, for example, 0.5, the object is determined to be of the corresponding object type; otherwise, it is determined not to be of the corresponding object type. The probability threshold can be fixed for all images or can vary based on factors such as the number of objects in the image, the image quality, and the image's lighting conditions.

[0057] The first algorithm may include a feature extraction process to extract features of objects within image frames 100a to 100c and represent them as digital vectors. These digital vectors (feature vectors) can then be used to determine the probability that the image data of an object is part of an object of a first object type within image frames 100a to 100c. The location of the object can be mapped to a region of the scene. The camera capturing the image is a still camera (e.g., a static camera). Figure 1 In the example shown, each region of the scene can be represented by the same pixel index or a range of pixel indices in each image frame 100a to 100c. For motion cameras, in addition to pixel indices, motion data can also be used to determine the corresponding pixel regions between image frames 100a to 100c. When image frames 100a to 100c are captured at different resolutions, scaling parameters can be additionally used to determine the corresponding pixel regions between image frames 100a to 100c.

[0058] However, no matter how well the first algorithm is trained, the model may struggle to distinguish objects with similar shapes, textures, or colors, especially under certain lighting conditions or from certain angles. This is in Figure 1 For example, image data 114, which depicts a portion of the leaves of tree 108, is interpreted as a face in the last image 100c of the tree images 100a-100c. Apart from this misclassification, the first algorithm correctly interprets other areas of the scene depicted in the three images 100a-100c, and only classifies image data 110, 112, and 116 corresponding to the area where person 102 in the scene is located as part of an object of the first object type.

[0059] As previously discussed, if the output from the first algorithm is the only data source for detecting objects of the first object type, the resulting inconsistent false positives related to tree 108 could lead to several problems. The area of ​​the scene corresponding to the leaves (i.e., image data 114 depicting such an area in the third image) has an uncertain object type state in the final image 100c (which will be combined below). Figures 5 to 6 (Further discussion is needed) because it is interpreted as a face in this image 100c, but not as a face in other images 100a, 100b. However, as disclosed herein, using a background model to make the final decision on such regions can reduce the problem of inconsistent false positives, and in this case, false positives are removed overall from the third image frame 100c.

[0060] Figure 3 A background model 300 is shown. Figure 3 In the example, background model 300 defines whether the image data corresponding to each of the multiple regions in the scene belongs to the background or the foreground, based on the region in the final image frame 100c. Figure 1 In the scene, only dog ​​104 and vehicle 106 move between image frames 100a and 100c. Figure 3 The background model in the diagram will schematically be related to Figure 1 In the final image 100c, regions 304 and 306 corresponding to the positions of vehicle 106 and dog 104 are defined as foreground, and the remaining regions are defined as background 302; however, this is only for ease of explanation. The background model can be determined by applying a second algorithm (different from the first algorithm) to at least some of the multiple images. The background model can therefore depend on the specific application of the second algorithm. The complexity of the second algorithm can range from a simple frame differencing algorithm to a Gaussian mixture model that considers color and depth information. When determining the background model using a scaling of the scene different from image frames 100a to 100c, the scaling parameter can be used to determine the corresponding region between image frames 100a to 100c and background model 300.

[0061] If the background model 300 is applied to image data 114 with an uncertain object type state as disclosed herein, the misclassification of the image data provided by the first algorithm can be recovered, and the image data can be correctly classified as part of an object that is not of the first object type (i.e., not part of a face). This is because image data 114 is considered part of the background (by...). Figure 3 The dashed area 308 in the diagram indicates that it spatially corresponds to, for example, Figure 1The image data shown (114) is therefore determined to be not part of an object of the first object type.

[0062] For image data in the final image 100c that were not identified as having an uncertain object type state (such as...) Figure 3 Regarding the image data 116 in the final image 100c or the image data depicting the dog 104, the output of the first algorithm is relied upon. Thus, image data 116 is determined to be part of an object of the first object type (i.e., part of a human face), while the image data depicting the dog is determined not to be part of an object of the first object type.

[0063] Figure 2 This illustrates another aspect of the technology described herein, which in many ways relates to the combination of Figure 1 The descriptions are similar. The second aspect differs from the first in that the evaluation made by the first algorithm is associated with objects detected and tracked within the video stream, rather than with specific regions of the scene. In other words, the first algorithm focuses on analyzing and classifying individual objects within the video, rather than regions or parts of the scene. Figure 2 An example of a video sequence in which the object classification algorithm outputs wave classification for several objects is shown. Figure 2 Three images 200a to 200c from a video sequence of a captured scene are shown. The scene includes five objects: a tree 222, a dog 218, a vehicle 220, and two people 202 and 204. Objects are tracked across multiple image frames 200a to 200c. Images 200a to 200c are further enhanced with highlight features 206, 208, 210, 212, and 214, which indicate that a first algorithm has identified an object of a first object type in each of image frames 200a to 200c. Figure 2 In the example, the first algorithm is configured to detect people.

[0064] As previously discussed, if the output from the first algorithm is the only data source for detecting objects of the first object type, the resulting inconsistent false positives related to tree 222 and inconsistent false negatives related to individuals 202 and 204 could lead to several problems. These three objects 202, 204, and 220 have an uncertain object type state in the final image 200c (as will be combined below). Figures 5 to 6 (To be discussed further), because the object types of these objects have been evaluated differently throughout the video stream. However, as disclosed herein, using a background model to make the final decision on such objects can reduce the problem of inconsistent false positives and false negatives, and in this case, remove false positives and false negatives overall from the third image frame 200c.

[0065] Figure 4 A background model 400 is shown, which is similar to Figure 3 Background model 300. In Figure 4 In the example, background model 400 defines for each of the multiple regions in the scene whether the image data corresponding to that region in the last image frame 200c belongs to the background or the foreground. Figure 2 In the scene, dog 218, vehicle 220, and two people 202 and 204 move between image frames 200a and 200c. Figure 4 The background model in the diagram will schematically be related to Figure 2 In the final image 200c, the regions 406, 408, 404, and 410 corresponding to the positions of the two people 202, 204, the dog 218, and the vehicle 220 are defined as the foreground, and the remaining regions are defined as the background 402.

[0066] If the background model 400 is applied to tree 222, which has an uncertain object type state as disclosed herein, the misclassification of objects provided by the first algorithm can be recovered, and objects can be correctly classified as objects that are not of the first object type (i.e., excluding faces). This is because the image data corresponding to tree 222 is considered part of the background in the final image 200c (by...). Figure 4 The dashed area 412 in the diagram indicates that it corresponds spatially to, for example, Figure 2 The location and extent of tree 222 shown are thus determined to be not part of an object of the first object type.

[0067] If the background model is applied to individuals 202 and 204, each with an uncertain object type state as disclosed herein, then any omissions in the classification of object 202 provided by the first algorithm for the final image 200c can be corrected, and the object can be correctly classified as an object of the first object type (i.e., including faces). This is because the image data corresponding to each individual 202 and 204 is considered part of the foreground in the final image 200c (by...). Figure 4 The dashed areas 406 and 408 in the diagram indicate that they spatially correspond to... Figure 2 The positions and extents of people 202 and 202 shown are thus identified as part of an object of the first object type.

[0068] and, Figure 2 The dashed line 216 represents the effective range or detection range within which the first algorithm can accurately classify objects based on their features or characteristics in the image. In some embodiments, the above is combined with... Figure 2 The described technique (the evaluation performed by the first algorithm is associated with objects detected within the video stream, rather than with, for example, combined with...) Figure 1The specific region of the scene discussed (associated with this) is primarily applied to objects outside a threshold distance from the camera capturing the video sequence of the scene. This threshold distance can be set to be the same as or similar to the effective range of the first algorithm. Because the moving people 202 and 204 are outside this threshold, i.e. Figure 2 Outside of the dashed line 216, the first algorithm may be unreliable for correctly classifying moving objects of the first object type. Therefore, this embodiment can reduce false negatives as described above.

[0069] In some examples, combined Figure 1 and Figure 2 The two aspects of the described object detection technique can be combined. For example, a region-based technique can be used for image content closer to the effective range of the first algorithm, while an object-based technique can be used for objects detected outside the effective range of the first algorithm.

[0070] Figures 5 to 6 Two embodiments of lists 502 and 602 of indicators generated by the first algorithm are illustrated by way of example, when analyzing a specific region of a captured scene in a video sequence or when analyzing a specific object tracked in a video sequence. As discussed above, in the region-based method, the first algorithm can be configured to store, for each region in each image, an indicator representing the difference between the probability that image data (depicting the region in the currently analyzed image) is part of an object of a first object type and a probability threshold, in a list of indicators associated with the region. In the object-based method, the first algorithm can be configured to, for each tracked object in each image, store, an indicator representing the difference between the probability that the object is an object of a first object type and a probability threshold, in a list of indicators associated with the object.

[0071] Independent of the method used, each region or each object can be associated with, for example... Figure 5 or Figure 6 Lists 502 and 602 shown are related. Each cell in each list 502 and 602 represents the output from the first algorithm when analyzing images from multiple images. Figures 5 to 6 Lists 502 and 602 in the table include X = 8 cells, indicating that 8 images have been analyzed. In some embodiments, the list of indicators associated with a region or object is a FIFO list with 5 to 15 values. In other embodiments, the list has 15 to 30 values. Depending on the use case, such as based on the frame rate of the camera capturing the video sequence, any suitable number of values ​​can be used. When a new image is analyzed, the oldest value in the list of indicators (the leftmost one in these examples) is removed, and the new value is inserted at the end of the list. Therefore, the list can always contain values ​​from the X most recently analyzed images in the video stream.

[0072] Figure 5 List 502 is shown, where each indicator indicates how far the estimated probability of a region / object being the first object type is from a probability threshold. Figure 1 In this scale, a positive number indicates an evaluation probability higher than the threshold (meaning the first algorithm evaluates the region / object as the first object type). A negative number indicates an evaluation probability lower than the threshold (meaning the first algorithm evaluates the region / object as not the first object type). Zero can be considered positive. The probability range is between 0 and 1, and the probability threshold is 0.5, meaning the cell value can be between -0.5 and +0.5. Other scales and thresholds can be applied.

[0073] The list of indicators 502 can be used to determine whether a region / object in the last of the multiple images represented by the list of indicators has an uncertain object type state. In other words, for a specific image, the result from that specific image is combined with the results from the previous X-1 images in the video stream to determine whether a region / object in the specific image has an uncertain object type state.

[0074] In one embodiment, the process of determining the uncertain object type state for a region / object in the last image frame includes: when evaluating whether a region / object (as depicted in the last image frame) is a first object type, first checking the degree of determination of a first algorithm. If the first algorithm is sufficiently deterministic, the result from the first algorithm is relied upon. Figure 5 In the table, cell 504 corresponding to the last image frame has a value of +0.15. This means the evaluated probability is 0.65. If this is considered sufficiently certain (the absolute value of the difference between the probability that the image data is part of an object of the first object type and a probability threshold exceeds the threshold difference), then the object / region associated with list 502 is indeed of the first object type. If the determination is insufficient, or if an embodiment that first checks the probability of the last image frame is not used, then further analysis of list 502 is required.

[0075] Analysis of the list may include determining a distribution measure between indicators indicating positive differences in the list of associated indicators and indicators indicating negative differences in the list of associated indicators. Distribution measures may include, for example, the mean ( Figure 3In the example, this could be 0.13, the median (0.2), or any suitable statistical method for the percentage distribution between positive and negative values ​​(75 / 25). If the distribution measurement indicates a mixture of positive and negative errors included in the uncertainty range, then the region / object in the final image is determined to have an uncertain object type state; otherwise, the region / object in the final image is determined not to have an uncertain object type state. As discussed above, because the list includes both positive and negative numbers, the uncertainty range is typically included on both the negative and positive sides by a threshold. The first algorithm can therefore be considered to provide a sufficiently consistent assessment for both negative and positive evaluations. Thus, in Figure 5 In such cases, if the mean and median are used, the uncertainty range is between negative and positive (e.g., a mean or median between -0.2 and +0.2 results in a region / object in the first image data being identified as having an uncertain object type state). For distributions (regardless of the magnitude of the values), the uncertainty range can, for example, be a mixture between a 20 / 80 distribution indicating positive and a 80 / 20 distribution indicating negative errors. Because... Figure 5 The distribution of the list falls within this range (75 / 25), which means that the region / object associated with list 502 may be considered to have an uncertain object type state.

[0076] In some embodiments, the list of indicators is a list of binary values, where positive differences are indicated by a first binary value and negative differences by a second binary value. This is as follows: Figure 6 The middle corresponds to Figure 5 Listing 502 shows the distribution, but in binary format. The same distribution measures, such as the mean (0.6) and the percentage distribution between positive and negative values ​​(75 / 25), can be applied. Figure 5 Compared to the format shown, the range of uncertainty may need to be adjusted according to the binary format.

[0077] Figures 7 to 8 A flowchart is shown illustrating two methods for detecting an object of a first object type in a first image frame out of multiple image frames, according to the first and second aspects discussed herein. Figure 7 Corresponding to the region-based aspect, and Figure 8 This corresponds to the object-based aspect.

[0078] Figure 7A method 700 for detecting an object of a first object type in a first image frame of a video sequence containing multiple image frames of a captured scene is illustrated. The method includes analyzing each region of the scene in each image frame of the video sequence S702 using a first algorithm configured to detect the first object type. In other words, method 700 may include, for each image frame in the multiple image frames: analyzing the image frame using the first algorithm to identify an object of the first object type in the image frame, the first algorithm calculating for each region in the multiple regions of the scene a probability that image data corresponding to that region is part of an object of the first object type in the image frame, the first algorithm having a probability threshold to determine whether the image data is part of an object of the first object type. An indicator of the difference between the probability that the image data is part of an object of the first object type and the probability threshold is then stored in a list of indicators associated with the region.

[0079] The method further includes providing a background model for the S704 video sequence using a second algorithm when an object of the first object type is detected in the last image frame among multiple image frames. The background model can define whether the image data corresponding to the region in the first image frame belongs to the background or the foreground for each region in multiple regions of the scene.

[0080] The method further includes, for each region of the scene, analyzing and identifying, based on the first algorithm, whether the image data corresponding to the region in S706 has an uncertain object type state. The uncertain object type state can be based on a list of associated indicators.

[0081] The method further includes checking the background model in step S708 to determine the object type state of the region of the first image if the region (i.e., the image data corresponding to the region) has an uncertain object type state. The background model can be used such that if the background model defines the image data as belonging to the background in the first image, it is determined that the image data is not part of an object of the first object type, and if the background model defines the image data as belonging to the foreground in the first image, it is determined that the image data is part of an object of the first object type.

[0082] In some examples, if the region (i.e., the image data corresponding to the region) does not have an uncertain object type state, the analysis of the region of the first image by the first algorithm in S710 is relied upon. This could mean that the image data is determined to be part of an object of a first object type when the first algorithm determines that the image data is part of an object of a first object type, and the image data is determined not to be part of an object of a first object type when the first algorithm determines that the image data is not part of an object of a first object type.

[0083] In some examples, the method includes masking or highlighting image data in the first image that S712 is determined to be part of an object of a first object type.

[0084] Figure 8 A method 800 is shown for detecting an object of a first object type in a first image frame of a video sequence of multiple image frames in which a scene is captured, the scene including multiple objects tracked in the multiple image frames.

[0085] Method 800 includes analyzing tracked objects in a scene within each frame of the S802 video sequence using a first algorithm configured to detect a first object type. In other words, method 800 may include: for each of a plurality of image frames, using the first algorithm to analyze the image frames to identify objects of the first object type within the image frames; the first algorithm calculating a probability that an object is an object of the first object type for each of the plurality of objects tracked in the image frames; and the first algorithm having a probability threshold to determine whether an object is an object of the first object type. An indicator of the difference between the probability that the image data is part of an object of the first object type and the probability threshold is then stored in a list of indicators associated with the object.

[0086] The method further includes providing a background model for the S804 video sequence using a second algorithm when an object of the first object type is detected in the last image frame among multiple image frames. The background model can define whether the image data corresponding to the region in the first image frame belongs to the background or the foreground for each region in multiple regions of the scene.

[0087] The method further includes, for each tracked object in the scene, analyzing and identifying, based on the first algorithm, whether the tracked object S806 has an uncertain object type state. The uncertain object type state can be based on a list of associated indicators.

[0088] The method further includes: if the object has an uncertain object type state, checking the S808 background model to determine the object type state of the tracked object in the first image. The background model can be used such that if the background model defines the image data corresponding to the object in the last image of the video sequence as belonging to the background in the first image, it is determined that the object is not an object of the first object type; and if the background model defines the image data as belonging to the foreground in the first image, it is determined that the object is of the first object type.

[0089] In some examples, if an object does not have an uncertain object type state, the analysis of the object in the first image by the first algorithm in S810 is relied upon. This could mean that the object is determined to be of the first object type when the first algorithm determines that the object is of the first object type, and the object is determined not to be of the first object type when the first algorithm determines that the object is not of the first object type.

[0090] In some examples, the method includes masking or highlighting S812 in the first image as an object of the first object type.

[0091] Figures 7 to 8 The methods illustrated herein, as well as any other methods or functions described herein, can be stored as instructions on a non-transitory computer-readable storage medium such that these methods are implemented when executed on a processing-capable device or system. Such a device or system may include one or more processors. Therefore, the features and methods described herein can be advantageously implemented in one or more computer programs that execute on a programmable system, which may include at least one programmable processor, at least one input device such as a camera for capturing image frames / video sequences, and at least one output device such as a display for displaying, possibly masking, or highlighting images as described herein. The at least one programmable processor is coupled to receive and send data and instructions to a data storage system. Suitable processors for executing the program's instructions include, for example, both general-purpose processors and special-purpose microprocessors, as well as one of a single processor or multiple processors or cores of any type of computer. The processor may be supplemented or incorporated into an ASIC (Application-Specific Integrated Circuit).

[0092] The above embodiments should be understood as illustrative examples of the present invention. Further embodiments of the present invention are anticipated. For example, the techniques described herein can be used in any suitable object classification system, such as for autonomous vehicles, motion analysis, surveillance and security, and weather forecasting. It should be understood that any feature described with respect to any embodiment can be used alone or in combination with other described features, and can also be combined with one or more features of any other embodiment or any combination of any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the claims.

[0093] List of Examples

[0094] A. A method for detecting an object of a first object type in a first image frame of a video sequence of multiple image frames in a captured scene, the method comprising:

[0095] For each of the multiple image frames: the image frame is analyzed using a first algorithm to identify an object of a first object type in the image frame. The first algorithm calculates the probability that the image data corresponding to the region is part of an object of the first object type in the image frame for each region in the scene. The first algorithm has a probability threshold to determine whether the image data is part of an object of the first object type, and stores an indicator of the difference between the probability that the image data is part of an object of the first object type and the probability threshold in a list of indicators associated with the region.

[0096] Detecting objects of a first object type in a first image frame, where the first image frame is the last image frame among multiple image frames, the detection includes:

[0097] A background model for a video sequence is provided. The background model defines whether the image data corresponding to the region in the first image frame belongs to the background or the foreground for each region in multiple regions of the scene. The background model is defined by applying a second algorithm, different from the first algorithm, to at least some of the multiple image frames.

[0098] For each region of the scene, based on a list of associated indicators, identify whether the image data in the first image corresponding to the region has an uncertain object type state;

[0099] When image data is identified as having an uncertain object type state.

[0100] When the background model limits the image data to belonging to the background in the first image, it is determined that the image data is not part of an object of the first object type.

[0101] When the background model limits the image data to belong to the foreground in the first image, it is determined that the image data is part of an object of the first object type.

[0102] B. The method according to embodiment A further includes the following steps:

[0103] When image data is identified as not having an uncertain object type:

[0104] When the first algorithm determines that the image data is part of an object of a first object type, the image data is determined to be part of an object of a first object type.

[0105] When the first algorithm determines that the image data is not part of an object of the first object type, the image data is determined to be not part of an object of the first object type.

[0106] C. The method according to any one of embodiments A to B, wherein the step of identifying whether image data in the first image has an uncertain object type state includes:

[0107] Based on a list of associated indicators, determine whether the absolute value of the difference between the probability that the image data is part of an object of a first object type and a probability threshold exceeds the threshold difference.

[0108] When the absolute value of the difference exceeds the threshold difference, it is determined that the image data does not have an uncertain object type state;

[0109] When the difference does not exceed a threshold distance, determine the distribution measurement between the indicators indicating positive differences in the list of associated indicators and the indicators indicating negative differences in the list of associated indicators;

[0110] When the distribution measurement indication includes a mixture of positive and negative errors within the uncertainty range, the image data is determined to have an uncertain object type state; otherwise, the image data is determined not to have an uncertain object type state.

[0111] D. The method according to any one of embodiments A to B, wherein the step of identifying whether image data in the first image has an uncertain object type state includes:

[0112] Determine the distribution of measurements between indicators indicating positive deviations in the list of associated indicators and indicators indicating negative deviations in the list of associated indicators.

[0113] When the distribution measurement indication includes a mixture of positive and negative errors within the uncertainty range, the image data is determined to have an uncertain object type state; otherwise, the image data is determined not to have an uncertain object type state.

[0114] E. The method according to embodiment D, wherein the list of indicators is a list of binary values, wherein positive differences are indicated by a first value of the binary values ​​and negative differences are indicated by a second value of the binary values.

[0115] F. The method according to any one of embodiments C to E, wherein the uncertainty range indicates a mixture between a 20 / 80 distribution and an 80 / 20 distribution of indicators indicating positive and negative differences.

[0116] G. The method according to any one of embodiments A to E, wherein the first algorithm includes a feature extraction process to extract features of objects within an image and represent the features as a digital vector.

[0117] H. The method according to any one of embodiments A to G, wherein the second algorithm is a motion-based background detection algorithm.

[0118] I. The method according to any one of embodiments A to H, wherein the list of indicators associated with the region is a FIFO list having 5 to 15 values.

[0119] J. The method according to any one of embodiments A to I further includes: masking or highlighting image data in the first image that is determined to be part of an object of a first object type.

[0120] K. The method according to any one of embodiments A to J is implemented in a camera, wherein the video sequence is part of a real-time video stream captured by the camera.

[0121] L. A method for detecting an object of a first object type in a first image frame of a video sequence of multiple image frames capturing a scene, the scene including multiple objects tracked in the multiple image frames, the method comprising:

[0122] For each of the multiple image frames: the image frame is analyzed using a first algorithm to identify an object of a first object type in the image frame. The first algorithm calculates the probability that an object is an object of the first object type for each of the multiple objects tracked in the image frame. The first algorithm has a probability threshold to determine whether an object is an object of the first object type, and stores an indicator of the difference between the probability that an object is an object of the first object type and the probability threshold in a list of indicators associated with the object.

[0123] Detecting objects of a first object type in a first image frame, where the first image frame is the last image frame among multiple image frames, the detection includes:

[0124] A background model for a video sequence is provided. The background model defines whether the image data corresponding to the region in the first image frame belongs to the background or the foreground for each region in multiple regions of the scene. The background model is defined by applying a second algorithm, different from the first algorithm, to at least some of the multiple image frames.

[0125] For each of the multiple objects tracked in the first image, the object is identified as having an uncertain object type state based on a list of indicators associated with the object;

[0126] When an object is identified as having an uncertain object type state.

[0127] Determine the image data corresponding to the object in the first image;

[0128] When the background model limits image data to belonging to the background in the first image, it determines that the object is not an object of the first object type.

[0129] When the background model limits the image data to belong to the foreground in the first image, the object is determined to be an object of the first object type.

[0130] M, the method according to embodiment L, wherein each of the tracked objects is located outside a threshold distance from the camera capturing the video sequence of the scene.

[0131] N. The method according to any one of embodiments L to M further includes the following steps:

[0132] When an object is identified as not having an uncertain object type, if the first algorithm determines that the object is an object of the first object type, then the object is determined to be an object of the first object type; if the first algorithm does not determine that the object is an object of the first object type, then the object is determined not to be an object of the first object type.

[0133] O. According to any one of embodiments L to N, the step of identifying whether an object has an uncertain object type state includes:

[0134] Based on a list of associated indicators, determine whether the absolute value of the difference between the probability that an object is an object of a first object type in a first image and a probability threshold exceeds the threshold difference.

[0135] When the absolute value of the difference exceeds the threshold difference, the determined object does not have an uncertain object type state;

[0136] When the difference does not exceed a threshold distance, determine the distribution measurement between the indicators indicating positive differences in the list of associated indicators and the indicators indicating negative differences in the list of associated indicators;

[0137] When the distribution measurement indication includes a mixture of positive and negative deviations within the uncertainty range, the object is determined to have an uncertain object type state; otherwise, the object is determined not to have an uncertain object type state.

[0138] P. According to any one of embodiments L to N, the step of identifying whether an object has an uncertain object type state includes:

[0139] Determine the distribution of measurements between indicators indicating positive differences in the list of associated indicators and indicators indicating negative differences in the list of associated indicators;

[0140] When the distribution measurement indication includes a mixture of positive and negative deviations within the uncertainty range, the object is determined to have an uncertain object type state; otherwise, the object is determined not to have an uncertain object type state.

[0141] Q. According to the method of embodiment P, the list of indicators is a list of binary values, wherein positive differences are indicated by a first value of the binary values ​​and negative differences are indicated by a second value of the binary values.

[0142] R. According to the method of any one of embodiments O to Q, the uncertainty range indicates a mixture between a 20 / 80 distribution and an 80 / 20 distribution of indicators indicating positive and negative differences.

[0143] S. A method according to any one of embodiments L to R, wherein the first algorithm includes a feature extraction process to extract features of objects within an image and represent the features as a digital vector.

[0144] T, the method according to any one of embodiments L to S, wherein the second algorithm is a motion-based background detection algorithm.

[0145] U. According to any one of embodiments L to T, the list of indicators associated with the object is a FIFO list with 5 to 15 values.

[0146] V. The method according to any one of embodiments L to U further includes masking or highlighting objects identified as being of the first object type in the first image.

[0147] X. A method according to any one of embodiments L to V, wherein the method is implemented in a camera, and the video sequence is part of a real-time video stream captured by the camera.

[0148] Y. A non-transitory computer-readable storage medium storing instructions that, when executed on a processing device, are used to implement a method according to any one of embodiments A to K or a method according to any one of embodiments L to X.

[0149] Z. A system for detecting an object of a first object type in a first image frame of a video sequence of multiple image frames in a captured scene, the system comprising:

[0150] One or more processors; and

[0151] One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by one or more processors, cause a system to perform the method of any one of embodiments A to K or the method of any one of embodiments L to X.

Claims

1. A method for detecting an object of a first object type in a first image frame of a plurality of image frames in a video sequence capturing a scene, the method comprising: For each of the plurality of image frames: the image frame is analyzed using a first algorithm to identify an object of the first object type in the image frame, the first algorithm calculates for each region in the plurality of regions in the scene the probability that image data corresponding to the region is part of an object of the first object type in the image frame, the first algorithm has a probability threshold to determine whether image data is part of an object of the first object type, and stores an indicator of the difference between the probability that the image data is part of an object of the first object type and the probability threshold in a list of indicators associated with the region; Detecting an object of the first object type in the first image frame, wherein the first image frame is the last image frame among the plurality of image frames, wherein the detection includes: A background model is provided for the video sequence, wherein the background model defines whether the image data corresponding to the region in the first image frame belongs to the background or the foreground for each region in the plurality of regions in the scene, and the background model is defined by applying a second algorithm different from the first algorithm to at least some of the plurality of image frames; For each region of the scene, based on multiple indicators in the list of associated indicators, it is identified whether the image data corresponding to the region in the first image frame has an uncertain object type state, and the list of indicators includes indicators different for each of the multiple image frames; When the image data is identified as having an uncertain object type: When the background model defines the image data as belonging to the background in the first image frame, it is determined that the image data is not part of an object of the first object type; When the background model defines the image data as belonging to the foreground in the first image frame, it is determined that the image data is part of an object of the first object type.

2. The method according to claim 1, further comprising the following steps: When image data is identified as not having an uncertain object type: When the first algorithm determines that the image data is part of an object of the first object type, the image data is determined to be part of an object of the first object type. When the first algorithm determines that the image data is not part of an object of the first object type, the image data is determined to be not part of an object of the first object type.

3. The method according to claim 1, wherein the step of identifying whether the image data has an uncertain object type state in the first image frame includes: Using the indicators from the list of associated indicators indicating the difference between the probability that the image data is part of an object of the first object type and the probability threshold, determine whether the absolute value of the difference between the probability that the image data is part of an object of the first object type and the probability threshold exceeds a threshold difference; When the absolute value of the difference exceeds the threshold difference, it is determined that the image data does not have an uncertain object type state; When the difference does not exceed a threshold distance, the distribution measurement between the indicators indicating positive differences in the list of associated indicators and the indicators indicating negative differences in the list of associated indicators is determined by applying statistical methods; When the distribution measurement indication includes a mixture of positive and negative errors within the uncertainty range, the image data is determined to have an uncertain object type state; otherwise, the image data is determined not to have an uncertain object type state. The uncertainty range includes a low threshold and a high threshold, which are defined by statistical measurements based on the statistical method.

4. The method according to claim 1, wherein the step of identifying whether the image data has an uncertain object type state in the first image frame includes: Determine the distribution measurement between the indicators indicating positive differences in the list of associated indicators and the indicators indicating negative differences in the list of associated indicators. When the distribution measurement indication includes a mixture of positive and negative errors within the uncertainty range, the image data is determined to have an uncertain object type state; otherwise, the image data is determined not to have an uncertain object type state.

5. The method of claim 4, wherein the list of indicators is a list of binary values, wherein positive differences are indicated by a first value of the binary values ​​and negative differences are indicated by a second value of the binary values.

6. The method of claim 3, wherein the uncertainty range indicates a mixture between a 20 / 80 distribution and an 80 / 20 distribution of the indicator indicating positive error and the indicator indicating negative error.

7. The method of claim 1, wherein the first algorithm includes a feature extraction process to extract features of objects within an image and represent the features as a digital vector.

8. The method according to claim 1, wherein the second algorithm is a motion-based background detection algorithm.

9. The method of claim 1, wherein the list of indicators associated with the region is a FIFO list having 5 to 15 values.

10. The method of claim 1, further comprising masking or highlighting image data identified as part of an object of the first object type in the first image frame.

11. The method of claim 1, wherein the method is implemented in a camera, and wherein the video sequence is part of a real-time video stream captured by the camera.

12. A method for detecting an object of a first object type in a first image frame of a plurality of image frames in a video sequence capturing a scene, the scene comprising a plurality of objects tracked in the plurality of image frames, the method comprising: For each of the plurality of image frames: the image frame is analyzed using a first algorithm to identify an object of the first object type in the image frame, the first algorithm calculates the probability that the object is an object of the first object type for each of the plurality of objects tracked in the image frame, the first algorithm has a probability threshold to determine whether the object is an object of the first object type, and stores an indicator of the difference between the probability that the object is an object of the first object type and the probability threshold in a list of indicators associated with the object; Detecting an object of the first object type in the first image frame, wherein the first image frame is the last image frame among the plurality of image frames, wherein the detection includes: A background model is provided for the video sequence, wherein the background model defines whether the image data corresponding to the region in the first image frame belongs to the background or the foreground for each region in a plurality of regions in the scene, and the background model is defined by applying a second algorithm different from the first algorithm to at least some of the plurality of image frames; For each of the plurality of objects tracked in the first image frame, the object is identified as having an uncertain object type state based on a plurality of indicators in a list of indicators associated with the object, the list of indicators including indicators different for each of the plurality of image frames; When the object is identified as having an uncertain object type state: Determine the image data corresponding to the object in the first image frame; When the background model defines the image data as belonging to the background, it is determined that the object is not an object of the first object type; When the background model defines the image data as belonging to the foreground, the object is determined to be an object of the first object type.

13. The method of claim 12, wherein each of the tracked objects is located outside a threshold distance from the camera that captures the video sequence of the scene.

14. A non-transitory computer-readable storage medium having instructions stored on the non-transitory computer-readable storage medium, which, when executed on a processing device, are used to perform the method according to claim 1 or the method according to claim 12.

15. A system for detecting an object of a first object type in a first image frame of a plurality of image frames in a video sequence capturing a scene, the system comprising: One or more processors; and One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the system to perform the method of claim 1 or the method of claim 12.

Images