Method and system for detecting threshold calibration in video object detection

Abstract

Methods and systems for detection threshold calibration in video object detection are disclosed. The method includes capturing a video of a scene, segmenting a first image into at least first and second scene regions, the first and second scene regions having attributes such that object detection conditions between desired scene regions differ. And setting a first object detection threshold value for the first scene area. A first object in a first scene area in the first image is detected, and a first confidence value of the first object is detected to be higher than a first object detection threshold. In a later plurality of images, the first object is tracked while moving to the second scene area. A second confidence value for detecting the first object of the second scene area is determined. And setting a second object detection threshold value for the second scene area, so that a second confidence value for detecting the first object is higher than the second object detection threshold value. In a subsequent image, an object of the first scene area is detected using a first object detection threshold, and an object of the second scene area is detected using a second object detection threshold.

Description

Technical Field

[0001] The present invention relates to the field of video object detection, and more specifically, to how to set a detection threshold in video object detection. Background Art

[0002] It is known to detect various objects in videos, such as in monitoring or surveillance scenarios. Object detection can be used for different use cases, such as for privacy masking or for people counting.

[0003] In privacy masking, it is important not to miss any objects that should be masked, that is, to avoid false negatives during object detection. For example, a person's face or entire body may need to be masked so that they cannot be identified. Another example is a license plate, which may need to be masked to prevent indirect identification of the vehicle owner. In this case, false positives may be more tolerable than false negatives. However, if too many unnecessary objects are masked, false positives may make the monitored scene difficult to understand.

[0004] On the other hand, in people counting, it can be important not to overcount—that is, to avoid false positives during object detection. For example, if visitor data falsely indicates a store is crowded, overcounting could deter potential customers from visiting or lead to unnecessarily high power consumption in occupancy-controlled HVAC systems. Avoiding false negatives is also important in certain counting scenarios. For example, if people counting is used to ensure that a room or building is not overly occupied by people to comply with fire safety regulations, then undercounting is crucial.

[0005] Object detectors or object detector algorithms are well known and can be implemented in different ways. Today, many object detection algorithms use machine learning or deep learning and may employ neural networks, for example. Other object detection algorithms are based on feature detection. Examples of such feature detection-based algorithms are the Viola-Jones object detection framework, which is based on Haar features, the Scale-Invariant Feature Transform (SIFT) algorithm, and the Histogram of Oriented Gradients (HOG) feature algorithm.

[0006] To balance the risks of false negatives and false positives, object detection algorithms often employ thresholds. When a neural network object detector finds object candidates in an image, each of these candidates is associated with a confidence value that indicates the certainty or likelihood of the detection. Confidence values can be applied on a per-pixel basis, or on a per-region or bounding box basis, depending on the object detector being used. Detection thresholds are set so that only detection candidates with confidence values above the threshold are output as object detections. In scenarios where object detection is used for privacy masking, these thresholds can also be used to determine which detected objects should be occluded. Detection thresholds can be used for both object detection, which is the task of detecting whether there are any objects in an image or at a certain location in an image, and classification, which is the task of detecting whether there are objects of a specific object class in an image or at a certain location in an image.

[0007] Setting detection thresholds is not an easy task. They need to be set high enough to limit the number of false positives, while being low enough to limit the number of false negatives. Furthermore, different thresholds may need to be set for different parts of the scene, as detection conditions may vary from area to area, making it more difficult to confidently detect objects in certain areas.

[0008] Therefore, there is a need for methods and systems that make it easier to set a detection threshold for object detection. Summary of the Invention

[0009] An object of the present invention is to provide an object detection method that reduces the difficulty of setting a detection threshold. Another object of the present invention is to provide an object detection system that can conveniently and efficiently set a detection threshold for object detection. A further object of the present invention is to provide a camera and a computer-readable storage medium that can perform object detection by simplifying the setting of the object detection threshold.

[0010] The invention is defined in the independent claim, embodiments are defined in the dependent claims.

[0011] Thus, according to a first aspect, the above-mentioned object is achieved in whole or in part by a method for detecting an object in a scene captured by a camera, the method comprising: capturing a video of the scene; segmenting a first image in the video into at least a first scene region and a second scene region, the first scene region and the second scene region having properties such that object detection conditions are expected to be different between the first scene region and the second scene region; setting a first object detection threshold for the first scene region; detecting a first object in the first scene region in the first image, wherein a first confidence value for detecting the first object is above the first object detection threshold; tracking the first object in a plurality of images subsequent to the first image as the first object moves into the second scene region; determining a second confidence value for detecting the first object in the second scene region; setting a second object detection threshold for the second scene region, wherein the second confidence value for detecting the first object is above the second object detection threshold; and, in subsequent images, using the first object detection threshold for detecting the object in the first scene region and using the second object detection threshold for detecting the object in the second scene region. In this way, the object detection thresholds can be adapted to conditions in the scene. Detection conditions in a scene can vary from scene region to scene region, so a uniform detection threshold will result in too many false positives in some scene regions if set too low, and too many false negatives if set too high. Trying to strike a balance, setting an intermediate detection threshold can result in both too many false positives in some regions and too many false negatives in other regions. By tracking a first object as it moves from a first scene region to a second scene region, the detection threshold used in the second scene region can be adjusted to ensure detection in the second scene region even when detection conditions are more difficult in the second scene region. For example, if the same detection threshold is used in the second scene region as in the first scene region, the first object detection threshold may be too high for the second scene region because detection conditions there are less favorable. Unfavorable object detection conditions can be caused, for example, by reduced visibility, such as if the second scene region is in shadow or if there is glare. Reduced visibility can also be caused by fog, rain, or frosted glass, or by objects partially submerged in water or partially obscured by foliage. Because the first object is tracked in the second scene region, the object is known to be there, even if the object detection threshold is set too high for the scene as a whole. Therefore, a second object detection threshold can be set for the second scene area so that the first object can also be detected there. Therefore, the second object detection threshold is set so that the second confidence value for detecting the first object is higher than the second object detection threshold. In this way, appropriate object detection thresholds can be set for different scene portions with different object detection conditions, ensuring acceptable false positive and false negative rates.

[0012] As used herein, the term "object detection" refers to both the general detection of objects, i.e., determining whether an object is present in an image, and more specific object detection or classification, such as person detection or vehicle detection, i.e., determining whether an object of a particular object class is present in an image. Specific object detectors or classifiers can operate according to different principles. Some classifiers operate according to the principle of "detection first, classification later," where an object is first detected and only then the object class of the detected object is determined. The output of such classifiers is typically a single object class for each detected object. Other classifiers classify possible objects directly, outputting a probability for each of multiple object classes for each pixel or object. For example, such a "direct classifier" might output a result such that a particular pixel has an X% probability of being part of a person and a Y% probability of being part of a vehicle. Depending on the thresholds used for the different object classes in a given scene, the user can be presented with results indicating that there are either people, vehicles, or no objects of interest at all.

[0013] In line with this, the term "confidence value" is used in this article to refer to the confidence value of a general object detector as well as the confidence value of a classifier. In other words, the confidence value of an object detection can indicate the likelihood that an object exists at a certain location in an image, or the likelihood that an object of a specific object class exists.

[0014] The first object can be associated with a first object class. Setting a first object detection threshold for the first scene area can include setting the first object detection threshold for objects of the first object class in the first scene area, and setting a second object detection threshold can include setting a second object detection threshold for objects of the first object class in the second scene area. Thus, the first object detection threshold and the second object detection threshold can be applicable to objects of the first object class.

[0015] The method may also include setting a third object detection threshold for detecting objects of the second object class in the first scene area; detecting a second object in the first scene area in a second image, wherein the second object is associated with the second object class and wherein a third confidence value for detecting the second object is greater than the third object detection threshold; tracking the second object in a plurality of images subsequent to the first image as the second object moves into the second scene area; determining a fourth confidence value for detecting the second object in the second scene area; setting a fourth object detection threshold for the second scene area such that the fourth confidence value for detecting the second object is greater than the fourth object detection threshold; and, in subsequent images, using the first object detection threshold to detect objects of the first object class in the first scene area and the second object detection threshold to detect objects of the first object class in the second scene area, and using the third object detection threshold to detect objects of the second object class in the first scene area and the fourth object detection threshold to detect objects of the second object class in the second scene area. In this manner, appropriate object detection thresholds may be set for objects of different object classes. It should be noted that even if objects of all object classes are in the same scene area, their object detection conditions do not necessarily need to be the same. For example, a boat may be easily detected on water, while a person swimming in the same scene area may be more difficult to detect than a person walking on land.

[0016] In some variations, the method further includes performing new segmentations on subsequent images following the first image based on a triggering event. This may be useful if the detection conditions in the scene change. Such changes may be predictable, such as when a scene is darkened by the shadow of a building at approximately the same time each day. The changes in detection conditions may also be more unpredictable, such as when an area of the scene becomes saturated with steam suddenly released from a machine. Some changes may be more gradual and long-term in a natural manner, such as sparse leaves growing in the spring and falling in the fall. The triggering event may be based on an automatic detection, such as the detection of lighting conditions in the scene. In other cases, the triggering event may be based on a timer or user input, such that new segmentations are performed at regular or irregular intervals to ensure that the detection conditions in different scene areas do not change to the extent that the object detection thresholds become obsolete.

[0017] Similarly, the method may further include re-tracking a subsequent object in an image subsequent to the first image based on the triggering event; determining an updated second confidence value for detecting the subsequent object in a second scene area; setting an updated second object detection threshold for the second scene area such that the updated second confidence value for detecting the subsequent object is higher than the updated second object detection threshold; and, in an image subsequent to the subsequent image, using the first object detection threshold for detecting the object in the first scene area and using the updated second object detection threshold for detecting the object in the second scene area. This method can adapt the object detection thresholds to changing detection conditions in the scene.

[0018] The triggering event can be one of a user input, an elapsed time since the second object detection threshold was set, a frequency at which object detections in the first or second scene regions deviate from a historical range, a time of day, a time of year, and a change in lighting conditions in the scene. This allows segmentation and object detection thresholds to adapt to various reasons for changes in detection conditions. For example, if the number of object detections in a scene region drops significantly compared to historical conditions, this may indicate that object detection conditions have changed, requiring new segmentation and the need to set a new object detection threshold.

[0019] The segmentation step can be performed using a semantic segmentation algorithm or a neural network. This can be an effective way to segment the scene into different scene regions.

[0020] According to a second aspect, the above-mentioned object is achieved in whole or in part by an object detection system for detecting an object in a scene captured by a camera, the system comprising circuitry configured to perform: a capture function configured to capture a video of a scene; a segmentation function configured to segment a first image in the video into at least a first scene region and a second scene region, the first scene region and the second scene region having properties such that object detection conditions are expected to differ between the first scene region and the second scene region; a threshold setting function configured to set a threshold for object detection in the images in the video; an object detection function configured to detect the object in the images in the video; a confidence value determination function configured to determine a confidence value for object detection; and an object tracking function configured to track the object in the images in the video. The threshold setting function is configured to set a first object detection threshold for the first scene region. The object detection function is configured to detect a first object in the first scene region in the first image, wherein a first confidence value for detecting the first object is greater than the first object detection threshold. The object tracking function is configured to track the first object in a plurality of images subsequent to the first image as the first object moves into the second scene region. The confidence value determination function is configured to determine a second confidence value for detecting the first object in the second scene area. The threshold setting function is configured to set a second object detection threshold for the second scene area such that the second confidence value for detecting the first object is greater than the second object detection threshold. The object detection function is configured to, in subsequent images, use the first object detection threshold for detecting the object in the first scene area and use the second object detection threshold for detecting the object in the second scene area.

[0021] In some embodiments of the system, the object detection functionality is configured to determine an object class of the detected object, the first object being associated with the first object class. In these embodiments, the threshold detection functionality is configured to set a first object detection threshold for objects of the first object class in the first scene area and to set a second object detection threshold for objects of the first object class in the second scene area.

[0022] According to a third aspect, the above objects are fully or at least partially achieved by a camera comprising the object detection system according to the second aspect. Such a camera allows for convenient object detection by simplifying the setting of a suitable detection threshold.

[0023] According to a fourth aspect, the above objects are fully or at least partially achieved by a computer-readable storage medium comprising instructions which, when executed by a device having processing capabilities, cause the device to perform the method of the first aspect.

[0024] Further scope of applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that while indicating preferred embodiments of the present invention, the detailed description and specific examples are given by way of illustration only, as various changes and modifications within the scope of the present invention will become apparent to those skilled in the art based on this detailed description.

[0025] Therefore, it is to be understood that the present invention is not limited to the specific component parts of the described devices or the steps of the described methods, as such devices and methods may vary. It is also to be understood that the terminology used herein is used only to describe specific embodiments and is not intended to be limiting. It must be noted that the articles "a", "an", "the" and "said" used in the specification and the appended claims are intended to indicate the presence of one or more elements, unless the context clearly dictates otherwise. Thus, for example, a reference to "an object" or "the object" may include a plurality of objects, etc. Furthermore, the word "comprising" does not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will now be described in more detail by way of example and with reference to the accompanying drawings, in which:

[0027] Figure 1 is an illustration of the image of the scene,

[0028] Figure 2a yes Figure 1 Illustration of a first image of a scene in FIG, indicating a detected person on a beach,

[0029] Figure 2b yes Figure 1 Illustration of a second image of the scene in which the detected person has moved into the water,

[0030] Figure 3a yes Figure 1 Illustration of the first image of the scene in FIG, indicating a detected car on the beach,

[0031] Figure 3b yes Figure 1 Illustration of a second image of the scene in which the detected car has moved into the water,

[0032] Figure 4 is an illustration of an image of another scene,

[0033] Figure 5 is a flowchart of the object detection method,

[0034] Figure 6 is a block diagram of an object detection system, and

[0035] Figure 7 is a block diagram of a camera comprising an object detection system. DETAILED DESCRIPTION

[0036] Figure 1 An image 1 of a scene 2 is shown. This scene 2 comprises a beach, which has a land area, an ocean area, and some buildings in the distance. There are people on the beach, as well as other objects such as balls and parasols. There is a need to detect people and other objects in the scene. This need may be related to privacy issues, such as the desire or obligation to cover up identifiable or recognizable people. The need to detect objects in the scene may also be related to security or safety issues. For example, it may be meaningful to detect whether people or other objects enter parts of the scene that are not allowed to enter or where there is a risk of injury. For example, sometimes the current in the sea is too strong to be safe for bathing or swimming, making it very important to detect people in the sea in order to warn them or rescue them. As another example, cars may not be allowed to drive on the beach, making it very important to detect cars in order to stop them or warn people on the beach.

[0037] Figure 1 The image 1 in is segmented into three different scene regions, namely the land region A1, the ocean region A2 and the sky region A3. Figure 1 In the figure, scene areas A1, A2, and A3 are separated by dotted lines. Segmentation can be performed using a semantic segmentation neural network or algorithm (such as a Gaussian mixture model). Segmentation is performed while taking into account the expected differences in object detection conditions. Different scene areas A1, A2, and A3 have different properties, and these differences in properties result in different levels of difficulty in detecting objects. For example, an object detector may have been trained to detect people and can easily detect people on land, such as people in land area A1, where they are generally clearly visible. However, people in water, such as people in ocean area A2, may be more difficult to detect because only the upper part of the body is clearly visible above the water surface, while the lower part of the body is partially obscured or distorted by the water, even if the water is clear. In sky area A3, people may be easily detected, but it is unlikely that people are actually there. Depending on the reason for detecting people or other objects, it may be important to avoid false positives or false negatives in unlikely detection areas such as sky area A3.

[0038] Using the same detection threshold for the entire scene 2 creates the risk of false positives, false negatives, or both. If the object detection threshold is set appropriately for land area A1, it may be too high for ocean area A2, preventing many people in the water from being detected. On the other hand, if the object detection threshold is set appropriately for ocean area A2, it may be too low for land area A1, leading to false positives—image regions that don't contain people being identified as containing them. For example, a toy or parasol may be mistakenly detected as a person. Regardless of whether the object detection threshold is optimized for land area A1 or ocean area A2, it may be incorrect for the sky area. In sky area A3, it may be desirable to ignore all but the most confidently detected objects, as the object of interest is unlikely to be there, and false alarms caused by birds should be avoided. In other scenarios, detecting whether an object is present in this unlikely region may be crucial, so it may be desirable to detect all objects of a certain class, such as people, even if the confidence level of the object detection is very low.

[0039] To meet these conflicting requirements, it would be useful to set different object detection thresholds for different scene areas A1, A2, and A3. For unlikely detection areas, such as the sky area A3, this can be very simple. If it is important not to miss any objects in the sky area, the detection threshold can be set low enough to ensure that even clouds or birds that are vaguely similar to the object of interest are detected. If there is no interest in objects in the sky at all, the object detection threshold can be set high enough so that only detections with a confidence value of 100% are output as actual detections. For the land area A1 and the ocean area A2, it may be more difficult to set a suitable object detection threshold. One approach is trial and error, where the user supervises object detection over a long period of time. This would be tedious and expensive. The inventors have proposed a method that makes setting the object detection threshold more convenient and can be performed automatically.

[0040] Now refer to Figure 2a and 2b Explains an improved method for setting the detection threshold for object detection. Figure 2a Shown with Figure 1 The same scene 2 is shown as the first image I1. The first image I1 is the same as the Figure 1The same method as described above has been used to segment the scene into three areas A1, A2, and A3. Object detection can be performed using any suitable object detector, for example a neural network object detector based on a convolutional neural network or a transformer. A first object detection threshold Th1A1 is set for the first scene area A1 (i.e., the land area A1). This first object detection threshold Th1A1 can be a default detection threshold and can be set automatically or manually. After a short period of supervised use of the object detection system, the first object detection threshold may have been adjusted from the default object detection threshold. A first object 3 in the form of a person is detected in the land area A1, which can be considered a scene area with good object detection conditions. The detection of person 3 has a first confidence value that is higher than the first object detection threshold Th1A1 for the first scene area, which means that the object detection algorithm is confident enough that this is a person and can output it as an object detection. Further images of the scene 2 are captured after the first image I1. Figure 2b A second image I2 is shown, which is a multiple image captured after the first image I1. It can be seen here that the detected person 3 has moved into the water, i.e., into the second scene area or ocean area A2. In the images following the first image I1, person 3 is tracked as they move into ocean area A2. Tracking can be performed using any suitable tracking algorithm, such as a Kalman tracker. As an alternative to such a video analysis-based tracker, additional sensors, such as radar sensors or lidar sensors, can be used for tracking. Figure 2bThe tracking in is represented by a wavy line 4, indicating how person 3 moves from the position in first image I1 to second image I2 through subsequent images (not shown). Through this tracking, even if the object detection threshold for ocean area A2 may initially be set too high, preventing person 3 from being detected, tracking in images subsequent to the first image I1 where person 3 was detected can confirm that person 3 is present in ocean area A2 in second image I2. A second object detection threshold Th2A2 for ocean area A2 can now be set by examining the second confidence value C2 for the attempted object detection in person 3's position in second image I2. Because person 3 is tracked while moving from a position detected in land area A1 in first image I1 to ocean area A2, even if the second confidence value C2 is lower than the first object detection threshold Th1A1 set for land areas or any other object detection threshold initially set for ocean area A2, it can be confirmed that person 3 is present in ocean area A2 in second image I2. Therefore, tracking can be considered ground truth for object detection. The second object detection threshold Th2A2 can now be set to be lower than the second confidence value C2. Therefore, the person will be detected by the object detector and output as an object detection. Taking into account whether avoiding false positives or false negatives is more important for the scenario, the second object detection threshold Th2A2 for ocean area A2 can be set to a smaller or larger margin below the second confidence value C2. Once the second object detection threshold Th2A2 has been set for the ocean area, object detection is performed in images subsequent to the second image I2 using the first object detection threshold Th1A1 for detecting objects in land area A1 and the second object detection threshold Th2A2 for detecting objects in ocean area A2.

[0041] The object detection thresholds Th1A1, Th2A2 set for the first scene area A1 and the second scene area A2 can be universal in that they apply to all objects of interest to be detected. However, in many cases, it may be desirable to use different object detection thresholds for objects of different object classes. For example, for each scene area A1, A2, A3, one object detection threshold can be set for detecting people and another object detection threshold can be set for detecting vehicles. Figure 2a 、 Figure 2b 、 Figure 3a and Figure 3bThis is explained. In this variant of the object detection method, objects of interest are associated with object classes. For example, it may be useful to detect people and vehicles. In this case, the object classes may be "people" and "vehicles". Possibly, an object class called "other" can also be used to label detection objects that are neither people nor vehicles. In other cases, it may be useful to detect different types of vehicles. In this case, the object classes may be, for example, "cars", "trucks", "motorcycles", and "bicycles". Again referring to Figure 2a and Figure 2b In this variation, the first object detection threshold Th1A1 set for the land area A1 is set only for objects of the first object class (the object class "person" in this example), and the second object detection threshold Th2A2 set for the ocean area A2 is also set only for objects of the object class "person". Figure 3a and Figure 3b , the same principle is used to set the object detection threshold for detecting objects of the second object class (the object class "vehicle" in this example). Therefore, the third object detection threshold Th3A1 is set for detecting objects of the object class "vehicle" in the land area A1. Figure 3a As shown in FIG, a car 5 can be found in the land area A1. The confidence value C3 for detecting the car 5 is higher than the third object detection threshold Th3A1 of “vehicle” in the land area A1. The car 5 is tracked in subsequent images, and as shown in FIG. Figure 3b As shown in FIG, the car has entered the water, that is, into the ocean area A2. Figure 3b denoted by the wavy line 6 in FIG. A fourth confidence value C4 is determined for attempting object detection at the location of the car 5 in the ocean area A2, and a fourth object detection threshold Th4A2 for "vehicle" in the ocean area A2 is set such that the fourth object detection threshold Th4A2 is lower than the fourth confidence value C4. When different object detection thresholds are set, object detection is performed in subsequent images using the corresponding object detection thresholds for different scene areas A1 and A2 and different object classes.

[0042] It may be noted that the object detection conditions do not need to vary in the same way between different scene regions for objects of all object classes. For example, it may be easier to detect a person on land than in water, but it may be easier to detect a boat in water than stranded or towed on land. This difference may be due to the training of the object detectors. In many cases, an object detector for detecting people may have been trained on people on land, while an object detector for detecting boats may desirably have been trained on boats on water. Therefore, it may be useful to start the object detection threshold setting process for objects of object class "person" by setting the object detection threshold for land area A1, and then track the detected person as it moves into ocean area A2; but it may be better to start the object detection threshold setting process for object class "boat" by setting the object detection threshold for ocean area A2, and then track the detected boat as it moves into land area A1.

[0043] Image segmentation can be performed on the first image and then applied to all subsequent images. However, in many scenes, detection conditions are not constant, so it may be advisable to perform new segmentations from time to time. A new segmentation may be useful if the scene changes. Such changes can be somewhat predictable and regular, such as in an indoor scene where lights are on during work hours and off outside, when leaves on shrubs grow in spring and fall in autumn, or when the tide shifts the waterline on a beach. Or they can be more unpredictable and irregular, such as when a large container is temporarily placed in the scene, casting a large shadow, or when the light changes due to weather conditions. Therefore, it may be useful to perform a new segmentation based on a trigger event. For example, the trigger event could be the elapsed time since the last segmentation. If the user deems a new segmentation necessary, the user can manually trigger the trigger event by interacting with the user interface. In some cases, the trigger event can be based on automatic detection. For example, if automatic detection of the scene's lighting conditions is used, a change in lighting conditions can be the trigger event. It is possible to make a new segmentation for each image, but it is generally more efficient to do it less frequently, and then use the same segmentation for multiple images until there is a motivation to make a new segmentation.

[0044] If object detection conditions in at least a portion of the scene have changed, whether or not a new segmentation has been performed, it may be useful to recalibrate the object detection threshold so that the false alarm and false negative rates do not deteriorate unacceptably. The method for recalibration is the same as that used to initially set the object detection threshold. As with a new segmentation, recalibration of the object detection threshold can be initiated by a trigger event. The trigger event can be manually activated, such as by interacting with a user interface. Alternatively, the trigger event can be automatic, such as the elapsed time since the second object detection threshold was set. The trigger event may recur regularly, initiating recalibration at a certain time of day or year. Changes in lighting conditions in the scene can also be automatically detected and generate a trigger event. In some cases, historical data can be used to generate a trigger event when the frequency of object detection in a first or second scene area deviates from a historical range. If the number of object detections in a scene area decreases or increases significantly compared to historical conditions, this can be due to an actual change in the number of objects present in the scene, but it can also be due to changes in object detection conditions. If a new segmentation is performed, recalibration of the object detection threshold is generally also required. However, even if the segmentation remains unchanged, object detection conditions may have changed to the point where threshold recalibration is necessary. Such changes in object detection conditions can apply to the entire scene, such as when the overall lighting conditions change, or they can apply only to one or a few scene areas.

[0045] by Figure 1 As an example, in scene 2 in Figure 1, a subsequent object (not shown) is detected in the land area A1 and tracked in a series of images in the same way as the first object 3 was detected from the first image I1 to the second image I2. This new tracking will be used as the ground truth for detecting the subsequent object in the ocean area A2. An updated second confidence value C is determined. 2U For object detection attempts in the location of subsequent objects in ocean region A2. Now set the updated second object detection threshold Th2 for ocean region A2 U A2, making it lower than the updated second confidence value C for the ocean area A2 2U In subsequent images, the updated second object detection threshold Th2 U A2 is used to detect objects in the ocean area A2.

[0046] The present invention has been described above with reference to an example scene in the form of a beach. Those skilled in the art will appreciate that there are many other types of scenes where the object detection conditions differ depending on the scene area. Figure 4In FIG. 4 , another example of a scene 40 is shown. Here, the scene is a forest with a path, some leaves, and shadow areas. The method for detecting objects in the image of this scene 40 is the same as that already described above. Therefore, the forest scene 40 is segmented into different scene areas, in this case, a path area A41, a leaf area A42, and a shadow area A43. Figure 1 Unlike scene 2, these scene areas A41, A42, and A43 are not like Figure 1 In contrast, the path area A41 can be considered as a scene area with good object detection conditions, while both the foliage area A42 and the shadow area A43 can be considered as scene areas with more difficult object detection conditions, because people and other objects may be partially obscured by foliage and difficult to detect in relatively dark or light-changing conditions.

[0047] Likewise, it should be remembered that the same scene area may not necessarily have good or poor object detection conditions for objects of all object classes. Therefore, different object detection thresholds need to be set for detecting objects of different object classes. As described above, the object detection threshold setting process also needs to be started in different scene areas for different classes of objects. It is often more efficient to start by setting object detection thresholds for a particular object class for scene areas that have good object detection conditions, and then track the detected object as it moves into scene areas that have more difficult object detection conditions for that object class. Figure 4 In the example scenario shown, an object detection threshold may first be set for the path region A41. Then, when an object detected there moves into the foliage region A42, the detected object will be tracked, and this tracking will be used as the ground truth to adopt the same approach as for Figure 2a and Figure 2b The object detection threshold is set for the foliage region in the same manner as described. Then, when another object detected in the path region A41 moves into the shadow region A43, the detected other object will be tracked, and the tracking will be used as the basic truth to set the object detection threshold for the shadow region A43 in the same manner.

[0048] Reference Figure 5 , the method of the present invention will now be described in slightly more general terms. Capturing (step S1) e.g. Figure 1 and Figure 4A video of a scene depicting a scene shown in FIG. A first image in the video is segmented (step S2) into at least two different scene regions. The different scene regions have properties such that object detection conditions are expected to be different between the scene regions. A first object detection threshold is set (step S3) for detecting objects in the first scene region. Using the first object detection threshold, a first object is detected (step S4) in the first scene region in the first image. A first confidence value for detecting the first object in the first scene region is higher than the first object detection threshold, otherwise it would not be output as an object detection. In a series of images following the first image, the first object is tracked as it moves into the second scene region. Object detection is performed in the second image, where a second object has moved into the second scene region, and a second confidence value for such attempted object detection in a known position in the second image is determined (step S6). A second object detection threshold is set (step S7) for object detection in the second scene region. The second object detection threshold is set so that the second confidence value is higher than the second object detection threshold. Therefore, using the second object detection threshold, the second object is detected in the second scene region in the second image. In subsequent images, object detection is performed (step S8) using the first object detection threshold for detecting objects in the first scene area and using the second object detection threshold for detecting objects in the second scene area.

[0049] Object detection methods can be used on live video to detect objects in substantially real time. Alternatively or additionally, object detection methods can be performed on recorded video. Such video recording can be performed in the camera that captures the video or remotely from the camera, such as in a computer running a video management system to which the video is transmitted from the camera.

[0050] Go to Figure 6 , an object detection system for detecting objects in a scene captured by a camera will be described. Object detection system 60 is illustrated in simplified form, with only those portions of the object detection system relevant to the present invention described. Object detection system 60 includes circuitry 61 configured to perform the functions required by the present invention. Thus, the circuitry is configured to perform a capture function 62, a segmentation function 63, a threshold setting function 64, an object detection function 65, a confidence value determination function 66, and an object tracking function 67.

[0051] The capture function 62 is arranged to capture a video of the scene, such as Figures 1-4A video of a scene shown in FIG. A segmentation function 63 is configured to segment a first image in the captured video into two or more scene regions, namely, into at least a first scene region and a second scene region. The scene regions have properties such that object detection conditions are expected to differ between different scene regions. A threshold setting function 64 is configured to set a threshold for object detection in images of the video. An object detection function 65 is configured to detect objects in images of the video. A confidence value determination function 66 is configured to determine a confidence value for object detection. An object tracking function 67 is configured to track objects in images of the video. To implement the present invention, the threshold setting function is configured to set a first object detection threshold for the first scene region. The object detection function is configured to detect a first object in a first scene region in a first image, where a first confidence value for detecting the first object is greater than the first object detection threshold. The object tracking function is configured to track the first object in multiple images subsequent to the first image as the first object moves into the second scene region. The confidence value determination function is configured to determine a second confidence value for detecting the first object in the second scene region. The threshold setting function is configured to set a second object detection threshold for the second scene area such that the second confidence value for detecting the first object is greater than the second object detection threshold. For subsequent images, the object detection function is configured to use the first object detection threshold for detecting objects in the first scene area and the second object detection threshold for detecting objects in the second scene area. Thus, object detection system 60 can operate according to the object detection method described above.

[0052] The object detection system 60 may be incorporated into a camera used to capture images in which object detection is to be performed. Alternatively, the object detection system 60 may be provided as a separate device or software in a general-purpose computer. For example, the object detection system 60 may be implemented in a computer that also runs a video management system to which images from the camera are transmitted.

[0053] Functions 62-67 of object detection system 60 can be implemented using circuitry or processing circuitry, including general-purpose processors, special-purpose processors, integrated circuits, ASICs ("application-specific integrated circuits"), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. In this disclosure, circuitry is hardware that performs or is programmed to perform the described functions. Hardware can be any hardware disclosed herein or otherwise known that is programmed or configured to perform the described functions.

[0054] In a pure hardware implementation, each function may have corresponding circuits that are specifically designed to implement the function. The circuits may be in the form of one or more integrated circuits, such as one or more application specific integrated circuits or one or more field programmable gate arrays.

[0055] In implementations that also include software, the circuitry can include a processor. Processors are considered processing circuitry or circuits because they include transistors and other circuitry. In this case, the circuitry can be considered a combination of hardware and software, with the software being used to configure the hardware and / or the processor.

[0056] It is understood that it is also possible to have a combination of hardware and software implementation, which means that some functions are implemented by dedicated circuits and other functions are implemented in the form of software, ie in the form of computer code executed by a processor.

[0057] exist Figure 7 In the figure, it is shown that Figure 6 1 and 2. The object detection system 60 of FIG. 1 is a camera 70. The camera 70 has a lens 71 and an image sensor 72 for capturing video of a scene. In addition, the camera 70 includes image processing circuitry 73 for processing the captured images, an object detection system 60 for detecting objects in the captured images, and a network interface 74 for transmitting video from the camera 70. As will be appreciated by those skilled in the art, the camera 70 includes additional components and functionality, but these will not be discussed here as they are not necessary for understanding the present invention. The object detection system 60 can operate as described above. The object detection system 60 can be used to detect objects in the video captured by the camera. If the camera is operably connected to one or more other cameras, the object detection system 60 can be used to detect objects in images captured by these other cameras.

[0058] It will be appreciated that those skilled in the art can modify the above embodiments in a variety of ways and still utilize the advantages of the invention as shown in the above embodiments. As an example, segmentation has been described as being performed using a semantic segmentation neural network or algorithm. This automatic segmentation is generally the most efficient, but there may also be scenarios where manual segmentation is preferred.

[0059] In the above description, it is assumed that a tracking algorithm or tracking device is used that is separate from the object detector. However, tracking can also be performed by an object detector that acts as a joint detector and tracker. In this case, the object detector will use different detection thresholds for different tasks. The object detection threshold can be used to determine whether the object detector is sufficiently certain that an object (or an object of a particular object class) exists to output an object detection for use by the user or another algorithm. The object tracking threshold can be used to determine whether the object detector is sufficiently certain that a new detection may be the same as a previously detected object and should therefore be added to the track. In these cases, the object detection threshold can be higher than the object tracking threshold so that only detections with a sufficiently high confidence value are considered object detections, but once an object is detected, it will be tracked even if the confidence value is below the object detection threshold. Tracking will only terminate when the confidence value is also below the object tracking threshold.

[0060] Tracking can also be performed with the help of re-identification using the feature vectors of the detected objects.

[0061] So far, the present invention has been described in the context of an object moving from a scene area with relatively good object detection conditions to a scene area with defined poor object detection conditions, such as when a person moves from land into water, or when a boat moves from water onto land. In such a situation, there is a risk of losing objects that would be easily detected in a "good" scene area, but would be more difficult to detect if they moved to a "bad" scene area. However, the inventors foresee that the same principles could be used to set or adjust the object detection threshold for the "good" scene area. In such a case, after tracking the object as it moves into the "good" scene area, the object detection threshold for the "good" scene area could be raised, thereby reducing the risk of false alarms. Figure 2a and Figure 2b In contrast to the event in [ 1 ], if person 3 is first detected in water area A2, then walks on the beach and enters land area A1, a confidence value is determined for detecting person 3 on land. As an example, the confidence value can be determined to be 95%, and the object detection threshold for land area A1 can be previously set (e.g., as a default) to 80%. As can be seen in this example, the object detection threshold for land areas may not be necessarily low and can be increased to, for example, 90%, thereby reducing the risk of false positives.

[0062] As described in the Background section, various methods can be used for object detection. The present invention has been described above in the context of machine learning or deep learning object detection (e.g., machine learning or deep learning object detection using neural networks). Nevertheless, the present invention can also be applied to feature-based object detection algorithms, as long as these algorithms use an adjustable detection threshold.

[0063] Therefore, the present invention should not be limited to the shown embodiments, but should be limited only by the appended claims.

Claims

1. A method for detecting an object in a scene captured by a camera, the method comprising: capturing a video of said scene, segmenting a first image in the video into at least a first scene region and a second scene region, the first scene region and the second scene region having properties such that object detection conditions are expected to be different between the first scene region and the second scene region, Setting a first object detection threshold for the first scene area, detecting a first object in the first scene area in the first image, wherein a first confidence value for detecting the first object is greater than a first object detection threshold, tracking the first object in a plurality of images subsequent to the first image as the first object moves into the second scene area, determining a second confidence value for detecting the first object in the second scene area, setting a second object detection threshold for the second scene area such that the second confidence value for detecting the first object is above the second object detection threshold, and In subsequent images, the first object detection threshold is used to detect objects in the first scene area and the second object detection threshold is used to detect objects in the second scene area.

2. The method according to claim 1, wherein: The first object is associated with a first object class, Setting the first object detection threshold for the first scene area includes setting a first object detection threshold for objects of the first object class in the first scene area, and Setting the second object detection threshold includes setting a second object detection threshold for objects of the first object class in the second scene area.

3. The method according to claim 2, further comprising: Setting a third object detection threshold for detecting objects of the second object class in the first scene area, detecting a second object in the first scene area in a second image, wherein the second object is associated with a second object class, and wherein a third confidence value for detecting the second object is above the third object detection threshold, tracking the second object in a plurality of images subsequent to the first image when the second object moves into the second scene area, determining a fourth confidence value for detecting the second object in the second scene area, setting a fourth object detection threshold for the second scene area such that the fourth confidence value for detecting the second object is higher than the fourth object detection threshold, and In subsequent images, the first object detection threshold is used to detect objects of the first object class in the first scene area and the second object detection threshold is used to detect objects of the first object class in the second scene area, and the third object detection threshold is used to detect objects of the second object class in the first scene area and the fourth object detection threshold is used to detect objects of the second object class in the second scene area. The method of claim 1 , further comprising performing new segmentation on subsequent images after the first image based on a triggering event.

5. The method according to claim 1, further comprising: performing new tracking of subsequent objects in images subsequent to the first image based on a triggering event, determining an updated second confidence value for detecting said subsequent object in said second scene area, setting an updated second object detection threshold for the second scene region such that the updated second confidence value for detecting the subsequent object is above the updated second object detection threshold, and In an image subsequent to the subsequent image, the first object detection threshold is used for detecting objects in the first scene area and the updated second object detection threshold is used for detecting objects in the second scene area.

6. The method according to claim 4, wherein: The triggering event is one of a user input, an elapsed time since setting the second object detection threshold, a frequency of object detection in the first scene area or the second scene area deviating from a historical range, a time of day, a time of year, and a change in lighting conditions in the scene.

7. The method according to claim 1, wherein The segmentation step is performed using a semantic segmentation algorithm.

8. An object detection system for detecting an object in a scene captured by a camera, the system comprising circuitry configured to: a capture function, arranged to capture a video of the scene, a segmentation function configured to segment a first image in the video into at least a first scene region and a second scene region, the first scene region and the second scene region having properties such that object detection conditions are expected to be different between the first scene region and the second scene region, a threshold setting function configured to set a threshold for object detection in images in the video, an object detection function configured to detect objects in images in the video, a confidence value determination function configured to determine a confidence value for object detection, and an object tracking function configured to track an object in said image in said video, in: The threshold setting function is configured to set a first object detection threshold for the first scene area, the object detection function being arranged to detect a first object in the first scene area in the first image, a first confidence value for detecting the first object being above a first object detection threshold, The object tracking function is configured to track the first object in a plurality of images subsequent to the first image as the first object moves into the second scene area, the confidence value determination functionality being arranged to determine a second confidence value for detecting the first object in the second scene area, The threshold setting function is configured to set a second object detection threshold for the second scene area such that the second confidence value for detecting the first object is above the second object detection threshold, and The object detection function is arranged to use the first object detection threshold for detecting objects in the first scene area and the second object detection threshold for detecting objects in the second scene area in subsequent images.

9. The system of claim 8, wherein: the object detection functionality being arranged to determine an object class of a detected object, the first object being associated with a first object class, The threshold setting function is arranged to set the first object detection threshold for objects of the first object class in the first scene area, and to set the second object detection threshold for objects of the first object class in the second scene area.

10. A camera comprising the object detection system according to claim 8.

11. A computer-readable storage medium comprising instructions which, when executed by a device having processing capabilities, cause the device to perform the method of claim 1.

Images