Detecting object reflections in a sequence of image frames

Abstract

To provide a method, a controller, a computer program, and a computer program product for detecting a reflection of an object in a sequence of image frames.SOLUTION: A method includes steps of: detecting objects of a prescribed type in a sequence of image frames; determining a detection score of each of the detected object; determining a distance ratio between unfiltered object position indicators identified for each of the detected objects; determining, for each of the detected objects, a displacement factor between a current location of the detected object and a previous location of the detected object; and determining that one of the two detected objects which has a lower detection score is a reflection of the other of the two detected objects in response to finding a match between normalized sizes and normalized movements of the detected objects.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION The embodiments presented herein relate to a method, a controller, a computer program, and a computer program product for detecting the reflection of an object in a sequence of image frames. [Background technology]

[0002] In general, object detection is a computer technique related to computer vision and image processing that deals with detecting instances of a class of semantic objects (e.g., humans, animals, vehicles, etc.). Some object detection algorithms generate keypoints or other types of object location indicators. Such points or indicators typically represent points of interest of a target object that are tracked from one image frame to the next.

[0003] There are scenarios or environments where object detection like the one described above can be challenging. Take urban environments, for example, which are environments with a large number of reflective surfaces. Reflective surfaces can be mirrors, but also sheets of glass or other types of reflective material. These surfaces generally pose a challenge to object detection algorithms. In some cases, both the detected object and its reflection are counted. This can be a challenge in applications that count the number of detected objects, as some detected objects may be counted twice. In other cases, reflections may be missed entirely. This can be a challenge in applications where detected objects are masked and reflections are left unmasked.

[0004] U.S. Patent Application Publication No. 2020 / 0175694(A1) discloses an information processing device including an identification circuit and a counting circuit. The identification circuit identifies two or more movable objects corresponding to the same real movable object among the movable objects based on similarities in the velocities of the movable objects in an image according to changes in their positions. The counting circuit counts the number of real movable objects corresponding to the movable objects based on the identification result of the identification circuit.

[0005] However, in reality, the reflection does not perfectly match the tracked object it reflects. Also, the reflection may appear smaller or larger than the tracked object. Furthermore, when the reflection occurs on a semi-transparent surface, the reflection in the image frame will have less detail than the tracked object. Furthermore, the reflection may only contain a portion of the tracked object. This may cause the specific circuitry in the information processing device of U.S. Patent Application Publication No. 2020 / 0175694(A1) to make an erroneous determination as to whether two or more movable objects correspond to the same real movable object among the movable objects. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent Application Publication No. 2020 / 0175694(A1) Summary of the Invention

[0007] It is an object of the embodiments herein to address the above challenges and provide improved detection of objects on reflective surfaces.

[0008] Thus, according to a first aspect, a method for detecting a reflection of an object in a series of image frames is presented. The method is executed by a controller. The method includes detecting objects of a predetermined type in the series of image frames and determining a detection score for each detected object. The method includes, for each detected object, determining a distance ratio between unfiltered object position indicators identified for the detected object. The unfiltered object position indicators are identified in a current image frame of the series of image frames. The distance ratio for the detected object defines a normalized size of the detected object. The method includes, for each detected object, determining a displacement coefficient between a current location of the detected object in the current image frame and a previous location of the detected object in a previous image frame of the series of image frames. The displacement coefficient for the detected object defines a normalized motion for the detected object. The method includes, in response to finding a match between the normalized sizes and normalized motions of two of the detected objects, determining that one of the two detected objects having a lower detection score is a reflection of the other of the two detected objects.

[0009] According to a second aspect, a controller for detecting object reflections in a series of image frames is presented. The controller includes processing circuitry. The processing circuitry is configured to cause the controller to detect objects of a predetermined type in the series of image frames and determine a detection score for each detected object. The processing circuitry is configured to cause the controller to determine, for each detected object, a distance ratio between unfiltered object position indicators identified for the detected object. The unfiltered object position indicators are identified in a current image frame in the series of image frames. The distance ratio for the detected object defines a normalized size of the detected object. The processing circuitry is configured to cause the controller to determine, for each detected object, a displacement factor between a current location of the detected object in the current image frame and a previous location of the detected object in a previous image frame in the series of image frames. The displacement factor for the detected object defines a normalized motion for the detected object. The processing circuitry is configured to cause the controller, in response to finding a match between the normalized sizes and normalized movements of two of the detected objects, to determine that one of the two detected objects having the lower detection score is a reflection of the other of the two detected objects.

[0010] According to a third aspect, there is provided a video surveillance system comprising a controller according to the second aspect and a camera for capturing a sequence of image frames.

[0011] According to a fourth aspect, there is presented a computer program for detecting reflections of an object in a sequence of image frames, the computer program comprising computer program code which, when executed on a controller, causes the controller to perform the method according to the first aspect.

[0012] According to a fifth aspect, there is provided a computer program product comprising the computer program according to the fourth aspect and a computer-readable storage medium having the computer program stored thereon. The computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0013] Advantageously, these aspects provide computationally efficient and accurate detection of objects in scenarios or environments having reflective surfaces.

[0014] Other objects, features, and advantages of the embodiments described herein will become apparent from the following detailed disclosure, the attached dependent claims, and the drawings.

[0015] In general, all terms used in the claims shall be interpreted according to their ordinary meaning in the art unless expressly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." shall be openly interpreted as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless expressly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated otherwise.

[0016] The inventive concept will now be described, by way of example, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic diagram illustrating a video surveillance system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating a schematic of an image frame including a detected object and a reflection of the detected object, according to one embodiment; [Figure 3] FIG. 2 is a diagram illustrating a schematic of a detected object and a bounding box surrounding the detected object, according to one embodiment; [Figure 4]FIG. 2 is a diagram illustrating a schematic of detected objects and key points of the detected objects according to one embodiment; [Figure 5] FIG. 2 is a diagram illustrating a schematic of detected object keypoints and detected object reflection keypoints according to one embodiment; [Figure 6] 1 is a flowchart of a method according to an embodiment. [Figure 7] FIG. 2 is a diagram illustrating a schematic representation of an image frame according to an embodiment after a first action has been performed; [Figure 8] FIG. 10 is a diagram schematically illustrating an image frame according to an embodiment after a second action has been performed; [Figure 9] FIG. 2 is a schematic diagram illustrating functional units of a controller according to one embodiment. [Figure 10] FIG. 1 illustrates an example of a computer program product including a computer-readable storage medium according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] The inventive concepts will now be described more fully with reference to the accompanying drawings, in which specific embodiments of the inventive concepts are shown. However, the inventive concepts may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concepts to those skilled in the art. Like numbers refer to like elements throughout the specification. Steps or features indicated with dashed lines should be considered optional.

[0019] FIG. 1 is a schematic diagram illustrating a video surveillance system 100 to which embodiments presented herein can be applied. A camera 120 is configured to capture image frames within a field of view (FoV) 130 of a scene 140. In the illustration of FIG. 1, the scene 140 includes a human-shaped object 150-1. The camera 120 includes a controller 110, and is co-located with, integrated with, or at least operably connected to the controller 110. The object 150-1 is assumed to represent a target object that is tracked from image frame to image frame as captured by the camera 120 and analyzed by the controller 110. To this end, the controller 110 may implement an object detection algorithm.

[0020] The embodiments disclosed herein relate to techniques for detecting the reflection of an object 150-1 in a sequence of image frames. To achieve such techniques, a controller 110, a method executed by the controller 110, and a computer program product including code, e.g., in the form of a computer program, that, when executed on the controller 110, causes the controller 110 to perform the method are provided.

[0021] 2 is a schematic illustration of an image frame 200. The image frame 200 depicts a human-shaped object 150-1, as in the scene 140. The object 150-1 has a location marked in the image frame 200 with reference numeral 220-1. The image frame 200 further depicts a reflective surface 210. The reflective surface 210 has a location marked in the image frame 200 with reference numeral 230. The reflective surface 210 also reveals an object 150-2, which is a reflection of the object 150-1 caused by the object 150-1 being reflected by the reflective surface 210. The object 150-2 has a location marked in the image frame 200 with reference numeral 220-2.

[0022] The inventors have realized that what objects 150-1 and 150-2 have in common is their object location indicators. Reference is now made to FIGS. 3 and 4, which schematically illustrate a human-shaped object 150-1, as in scene 140. Different types of object location indicators are illustrated in FIGS. 3 and 4. In FIG. 3, object 150-1 is surrounded by a bounding box 310. The bounding box 310 is defined by two corner points 320a and 320b and a center point 330. The center point 330 may represent the location 220-1 of object 150-1. The corner points 320a and 320b and the center point 330 may be, for example, CenterNet keypoints. The bounding box may be, for example, a CenterNet bounding box. In this regard, the object location indicator may be defined by more than one bounding box. That is, more than one bounding box may be used to surround the detected object 150-1. That is, a first bounding box may enclose a first portion (e.g., head) of the detected object 150-1, a second bounding box may enclose a second portion (e.g., torso) of the detected object 150-1, etc. In Figure 4, the object 150-1 is overlaid by object position indices 410-1 through 417b-1, as listed in Table 1. These object position indices can be considered simplified versions of COCO keypoints or MediaPipe Pose keypoints. TIFF0007766061000001.tif63170

[0023] The object 150-2, which is the reflection, has the same setup of object position indices, but is reflected. By analyzing the object position indices, it is possible to determine the object-reflection pair at the object level, i.e., one tracked object 150-1 and its reflected corresponding object 150-2, or even at the object position indices level. Reference is now made to FIG. 5, which schematically illustrates the object position indices 410-1 to 417b-1 as listed in Table 1 for object 150-1 (not shown). FIG. 5 also illustrates the position 220-1 of object 150-1. FIG. 5 further illustrates the object position indices 410-2 to 417b-2 for object 150-2 (not shown), which are the reflection of object 150-1 on the reflective surface 210 at location 230. The object position indices 410-2 to 417b-2 correspond to the object position indices listed in Table 1. That is, the object position indicator 410-2 represents the head of the object 150-2, and the object position indicator 417b-2 represents the left foot of the object 150-2.

[0024] 6 is a flow chart illustrating an embodiment of a method for detecting the reflection of an object 150-1 in a sequence of image frames 200, 800, 900. The method is executed by the controller 110. The method is advantageously provided as a computer program 1120.

[0025] S102: The controller 110 detects objects 150-1, 150-2 of a predetermined type in the sequence of image frames 200, 800, 900. The controller 110 determines a detection score for each detected object 150-1, 150-2.

[0026] In some examples, in addition to a detection score determined for each detected object 150-1, 150-2 as a whole, a detection score is also determined for each object location indicator identified for the detected objects 150-1, 150-2. The detection score for an individual object location indicator may only become relevant if the detection score for the detected object 150-1, 150-2 exceeds a certain threshold. Conversely, in some embodiments, the detection score for a given detected object 150-1, 150-2 is a function of the detection scores of all unfiltered object location indicators identified for the given detected object 150-1, 150-2.

[0027] S104: The controller 110 determines, for each detected object 150-1, 150-2, a distance ratio between the unfiltered object position indices identified for the detected objects 150-1, 150-2. The unfiltered object position indices are identified in a current image frame 200, 800, 900 in the sequence of image frames 200, 800, 900. The distance ratio of the detected objects 150-1, 150-2 defines a normalized size of the detected objects 150-1, 150-2.

[0028] Since the size of the detected object is determined for the entire detected object, the size is considered to be the normalized size of the detected object.

[0029] S106: The controller 110 determines, for each detected object 150-1, 150-2, a displacement coefficient between the current location 220-1, 220-2 of the detected object 150-1, 150-2 in the current image frame 200, 800, 900 and the previous location of the detected object 150-1, 150-2 in the previous image frame in the series of image frames 200, 800, 900. The displacement coefficient of the detected object 150-1, 150-2 defines the normalized motion of the detected object 150-1, 150-2.

[0030] Thus, for previous image frames in the sequence of image frames 200, 800, 900, the historical data is used by the controller 110 to obtain information about the location of the detected object in the previous image frame. Based on how much the detected object has moved from the previous image frame to the current image frame, a measure of the motion of the detected object can be obtained. Because the motion of the detected object is determined for each entire detected object, the motion can be considered to be the normalized motion of the detected object.

[0031] The normalized sizes and normalized movements of the two detected objects 150-1, 150-2 can then be compared to one another. That is, the normalized size of one of the two detected objects 150-1, 150-2 is compared to the normalized size of the other of the two detected objects 150-1, 150-2, and the normalized movement of one of the two detected objects 150-1, 150-2 is compared to the normalized movement of the other of the two detected objects 150-1, 150-2. Further in this regard, in general terms, normalized size and normalized movement mean that certain predetermined measures (normalized size size and normalized movement movement) are scaled or otherwise recalculated to a common scale. As a non-limiting example, assuming that object 150-1 and its reflection are viewed at different distances from one another relative to camera 120, this has the effect of the object 150-1 and its reflection appearing to be different sizes from one another, with the object closest to the camera appearing larger and vice versa. By using normalized size, object 150-1 and its reflection are scaled or otherwise recalculated to a common size scale that accounts for this effect so that object 150-1 and its reflection appear to be one and the same size. Similarly, under the same assumptions, there is also the effect of object 150-1 and its reflection appearing to move at different speeds relative to one another, with the object closest to the camera appearing to move faster and vice versa. By using normalized motion, object 150-1 and its reflection are scaled or otherwise recalculated to a common motion scale that accounts for this effect so that object 150-1 and its reflection appear to move at one and the same speed.

[0032] If the normalized size and normalized motion are similar for two detected objects 150-1, 150-2, it can be concluded that the two detected objects 150-1, 150-2 have similar sizes and similar motions, which indicates that one of the two detected objects 150-1, 150-2 is a reflection of the other of the two detected objects 150-1, 150-2.

[0033] S110: In response to finding a match between the normalized sizes and normalized movements of the two detected objects 150-1, 150-2, the controller 110 determines that one of the two detected objects 150-1, 150-2 having a lower detection score is a reflection of the other of the two detected objects 150-1, 150-2.

[0034] Thus, by considering the detection scores of the two detected objects 150-1, 150-2, the detected object 150-2 with the relatively low detection score is classified as a reflection of the detected object 150-1 with the relatively high detection score.

[0035] Next, embodiments relating to further details of the detection of the reflection of the object 150-1 in the sequence of image frames 200, 800, 900 performed by the controller 110 will be disclosed.

[0036] As disclosed above, the controller 110 detects objects 150-1, 150-2 of a predetermined type in the sequence of image frames 200, 800, 900. In this regard, the objects 150-1, 150-2 that the controller 110 is configured to detect may be of different types. In some non-limiting examples, the predetermined types are humans, animals, or vehicles such as cars or trucks. Thus, the controller 110 may be trained and configured to detect objects 150-1, 150-2 of a predetermined type.

[0037] As disclosed above, for each detected object 150-1, 150-2, the controller 110 determines a distance ratio between the unfiltered object position indices identified for the detected objects 150-1, 150-2. Generally speaking, the unfiltered object position indices are object position indices determined without initially filtering out objects detected with low detection scores. That is, the unfiltered object position indices refer to object position indices before thresholding is performed to reduce false detections in the scene. Because reflections generally have lower detection scores than non-reflections, the unfiltered object position indices are used to ensure that detected objects 150-2 that are reflections are not missed. That is, some of the unfiltered object position indices may belong to objects that would otherwise be filtered out or discarded to produce a low detection score. Thus, in some embodiments, the unfiltered object position indices are determined for all detected objects 150-1, 150-2 without filtering out any of the detected objects 150-1, 150-2 (because their detection scores are lower than the detection threshold).

[0038] Further in this regard, there can be different types of object position indicators.

[0039] In some non-limiting examples, each of the object location indicators is a Common Object in Context (COCO) keypoint 410-1:417b-1, 410-2:417b-2, or a MediaPipe Pose keypoint. In this regard, the object location indicator need not be defined by all COCO keypoints or MediaPipe Pose keypoints; rather, a subset of these keypoints can be used as the object location indicator. This is illustrated in FIG. 4, referenced above.

[0040] In another non-limiting example, the object location indicators are the corner points 320s, 320b and center point 330 of a bounding box 310, such as a CenterNet bounding box. This is shown in Figure 3, referred to above.

[0041] As disclosed below, the determination that one of the two detected objects 150-1, 150-2 is a reflection of the other of the two detected objects 150-1, 150-2 is made in response to finding a match between the normalized sizes and / or normalized movements of the two detected objects 150-1, 150-2. Thus, in some embodiments, the controller 110 is configured to perform (optional) step S108.

[0042] S108: The controller 110 finds a match between the normalized size and / or normalized movement of two of the detected objects 150-1, 150-2.

[0043] There may be different ways for controller 110 to find a match between the normalized size and / or normalized movement of two of detected objects 150-1, 150-2. In some aspects, finding this match is performed by comparing the normalized size and / or normalized movement of the first detected object 150-1 with the normalized size and / or normalized movement of the second detected object 150-2. In particular, in some embodiments, controller 110 is configured to perform (optional) step S108a and / or step S108b as part of step S108.

[0044] S108a: The controller 110 verifies that the size difference between the normalized size of one of the two detected objects 150-1, 150-2 and the normalized size of the other of the two detected objects 150-1, 150-2 is less than a size threshold.

[0045] Therefore, the smaller the size difference between the normalized size of one of the two detected objects 150-1, 150-2 and the normalized size of the other of the two detected objects 150-1, 150-2, the more likely a match will occur.

[0046] S108b: The controller 110 verifies that the difference in motion between the normalized motion of one of the two detected objects 150-1, 150-2 and the normalized motion of the other of the two detected objects 150-1, 150-2 is less than a motion threshold.

[0047] Therefore, the smaller the difference in motion between the normalized motion of one of the two detected objects 150-1, 150-2 and the normalized motion of the other of the two detected objects 150-1, 150-2, the more likely a match is.

[0048] In some embodiments, matching is performed at the object position indicator level, i.e., instead of simply comparing the normalized size and / or normalized movement of the first detected object 150-1 with the normalized size and / or normalized movement of the second detected object 150-2, a comparison (with respect to size and / or movement) is made between a pair of object position indicators of one of the two detected objects 150-1, 150-2 and a pair of object position indicators of the other of the two detected objects 150-1, 150-2.

[0049] That is, for purposes of explanation, assume that the two detected objects 150-1, 150-2 represent humans, with both knees detectable and represented by object location indicators on both of the two detected objects 150-1, 150-2. A first comparison can then be made between the normalized distance between the knees according to the object location indicator of one of the two detected objects 150-1, 150-2 and the normalized distance between the knees according to the object location indicator of the other of the two detected objects 150-1, 150-2. Furthermore, a second comparison can be made between the normalized movement of the knees given by the object location indicator of one of the two detected objects 150-1, 150-2 and the normalized movement of the knees given by the object location indicator of the other of the two detected objects 150-1, 150-2. Further such comparisons of size and / or movement can be made for additional portions of the detected objects to accumulate matching scores for the two detected objects 150-1, 150-2.

[0050] In some aspects, the controller 110 performs some action when it determines that one of the two detected objects 150-1, 150-2 is a reflection of the other of the two detected objects 150-1, 150-2. Thus, in some embodiments, the controller 110 is configured to perform (optional) step S112.

[0051] S112: The controller 110 performs an action regarding the detected object 150-2, which is a reflection of the other detected object 150-1.

[0052] The actions performed by the controller 110 in step S112 can be of different types. In some non-limiting examples, the actions include applying a mask 810-2 to the detected object 150-2, which is a reflection of the other detected object 150-1. Reference is now made intermediately to FIG. 7, which schematically illustrates an image frame 800. The image frame 800 depicts how a first mask 810-1 is placed where the object 150-1 appears in FIG. 2, and how a second mask 810-2 is placed where the object 150-2 appears in FIG. 2. In some non-limiting examples, the actions include filtering out the detected object 150-2, which is a reflection of the other detected object 150-1, lowering the detection score threshold for the detected object 150-2, which is a reflection of the other detected object 150-1, etc. Reference is now made intermediately to FIG. 8, which schematically illustrates an image frame 900. Image frame 900 depicts object 150-1 similar to Figure 2, but object 150-2 has been filtered out so that it does not appear in image frame 900. In this regard, it should be noted that although object 150-2, which represents a reflection, is shown as having been filtered out of the image frame, filtering out generally refers to filtering detected object 150-2 out of any counting or other type of calculation, such that object 150-1 is not counted twice.

[0053] 2, the reflection is caused by a reflective surface 210 being positioned between two detected objects 150-1, 150-2. Generally speaking, the reflective surface 210 is positioned at a midpoint between the locations 220-1, 220-2 of the two detected objects 150-1, 150-2. Thus, knowledge of the locations 220-1, 220-2 of the two detected objects 150-1, 150-2 is used to determine the location 230 of the reflective surface 210. Thus, in some embodiments, the controller 110 is configured to perform (optional) step S114.

[0054] S114: The controller 110 determines a location 230 of the reflective surface 210 that makes one of the detected objects 150-1, 150-2 a reflection of the other detected object 150-1, 150-2. The location 230 of the reflective surface 210 is determined as the midpoint between the locations 220-1, 220-2 of the two detected objects 150-1, 150-2.

[0055] In some embodiments, knowledge of the location 230 of the reflective surface 210 is utilized in detecting further objects in the series of image frames 200, 800, 900. The location 230 of the reflective surface 210 may be used as a reference point, or a reference line, or even a reference plane, so that knowledge of the location 230 may improve the chances of detecting a reflective object under difficult lighting conditions in future image frames of the same scene, or at least a scene in which the reflective surface 210 is still present in the scene.

[0056] Note that matching based on the normalized size of objects 150-1 and 150-2 does not require consideration of the mirroring of object position indicators 410-2 through 417b-2 of object 150-2 compared to object position indicators 410-1 through 417b-1 of object 150-1. For the same reason, matching based on the normalized motion of objects 150-1 and 150-2 results in the normalized motion of object 150-1 being the same as the normalized motion of object 150-2, except for a possible change in sign depending on the position of reflecting surface 210 relative to detected objects 150-1 and 150-2. For example, with the position of reflecting surface 210 as shown in FIG. 2, if object 150-1 appears to be moving leftward in FIG. 2, object 150-2 appears to be moving rightward in FIG. 2. To address this potential issue, normalized motion may be represented solely by its magnitude, without considering its direction. Furthermore, in any matching performed at the object position index level, reflections should be taken into account if the match includes any object position index where both a right object position index and a left object position index exist for each object, such as object position indexes 412a-1, 412b-1, etc.

[0057] The method and controller 110 disclosed above may be used as part of a technique for improving the detection of humans and other objects 150-1 in a scene 140 having glass surfaces. Glass surfaces can be both reflective and transparent. The reflected image may be darker than the image of the actual person or object. The same is true for humans or other objects 150-1 viewed through glass, especially under certain lighting conditions. As previously disclosed, if the goal of detecting an object 150-1 is to mask the object 150-1, setting a detection threshold that is too strict risks leaving the mirror image of the object 150-1 unmasked and potentially identifiable. Conversely, if the goal of detecting an object 150-1 is to count it, setting a detection threshold that is too broad risks double-counting the object 150-1.

[0058] Thus, in some embodiments, regions are identified that are likely to produce both reflected and transmitted images. Under certain lighting conditions, it may be easier to identify such regions, and knowledge gained during beneficial conditions may be used during more difficult conditions. If it is possible to identify a surface where reflection 150-2 of object 150-1 is often found, but where reflection is similarly often not found (e.g., keypoints do not match), then this surface is likely both reflective and transmissive.

[0059] Lighting conditions may affect how reflective and / or transmissive the objects 150-1, 150-2 are. For example, it is generally easier to examine highly reflective or highly transmissive surfaces, making detection clearer and more reliable. On the other hand, it is generally more difficult to detect objects under dim lighting conditions. Therefore, lighting condition information may be used when determining whether an object is a reflection. For example, if dim lighting conditions are indicated, the threshold may be lowered. This may also help tune the keypoint reflection algorithm. For example, if it is determined that a given region of the image is more likely to find a reflection rather than a transmission, a more aggressive search for keypoint pairs can be performed in this given region of the image. Furthermore, bright lighting conditions generally produce more reflective objects than transmissive objects. This knowledge can also be used to aid the algorithm. Thus, if lighting conditions indicate a higher likelihood of a reflection, a more aggressive search for keypoint pairs can be performed.

[0060] Here, we briefly describe a possible approach. Reflections can be found using methods similar to those described above. The number of recorded reflections in a given portion of the scene 140 is saved over time. This can be thought of as generating a heat map. If there are many recorded reflections, but also many true targets for a given surface, this is marked in the heat map. In privacy masking applications, a static mask can be applied to this surface to ensure that reflections are masked even if they fall below the current detection threshold. Alternatively, the mask threshold required to mask object 150-2 can be lowered in this region. For each image frame 200, it may be possible to determine whether detected objects 150-1, 150-2 are reflections by finding reflection pairs within the scene 140. In this way, by building data over time with the probability that object 150-1 is a reflection or not, it may be possible to predict the probability that a new object located in the same region of an image frame in the future is a reflection or not.

[0061] 9 illustrates schematically the components of controller 110 according to an embodiment, with respect to the number of functional units. Processing circuitry 1010 is provided using any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers, digital signal processors (DSPs), etc., capable of executing software instructions stored in a computer program product 1110 (such as in FIG. 10), for example in the form of a storage medium 1030. Processing circuitry 1010 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0062] In particular, the processing circuitry 1010 is configured to cause the controller 110 to perform a series of operations, or steps, as disclosed above. For example, the storage medium 1030 may store a series of operations, and the processing circuitry 1010 may be configured to retrieve the series of operations from the storage medium 1030 to cause the controller 110 to perform the series of operations. The series of operations may be provided as a series of executable instructions.

[0063] In this manner, the processing circuitry 1010 is configured to perform methods as disclosed herein. The storage medium 1030 may also include persistent storage, which may be, for example, any single or combination of magnetic, optical, solid-state, or remotely mounted memory. The controller 110 may further include a communication interface 1020 configured at least for communication with other functions, nodes, entities, and / or devices, such as the camera 120, potentially functions, nodes, entities, and / or devices of a video surveillance system. As such, the communication interface 1020 may include one or more transmitters and receivers, including analog and digital components. The processing circuitry 1010 controls the general operation of the controller 110, for example, by sending data and control signals to the communication interface 1020 and the storage medium 1030, by receiving data and reports from the communication interface 1020, and by retrieving data and instructions from the storage medium 1030. Other components and associated functionality of the controller 110 are omitted so as not to obscure the concepts presented herein.

[0064] The controller 110 may be provided as a standalone device or as part of at least one additional device. For example, the controller 110 and the camera 120 may be part of the video surveillance system 100. Optionally, the controller 110 may be integrated into the camera 120. A first portion of the instructions executed by the controller 110 may be performed on a first device, and a second portion of the instructions executed by the controller 110 may be performed on a second device; the embodiments disclosed herein are not limited to a particular number of devices on which the instructions executed by the controller 110 may be performed. Thus, methods according to embodiments disclosed herein are suitable for execution by the controller 110 residing in a cloud computing environment. Thus, although a single processing circuitry 1010 is shown in FIG. 9, the processing circuitry 1010 may be distributed among multiple devices or nodes. The same is true for the computer program 1120 in FIG. 10.

[0065] 10 illustrates an example of a computer program product 1110 including a computer-readable storage medium 1130. The computer-readable storage medium 1130 can have stored thereon a computer program 1120 that can cause the processing circuitry 1010 and entities and devices operatively coupled thereto, such as the communications interface 1020 and the storage medium 1030, to perform methods according to embodiments described herein. Thus, the computer program 1120 and / or the computer program product 1110 can provide means for performing any steps as disclosed herein.

[0066] 10, computer program product 1110 is shown as an optical disc, such as a CD (compact disc) or DVD (digital versatile disc) or Blu-Ray disc. Computer program product 1110 may also be embodied as a memory, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), or more particularly as a non-volatile storage medium of the device in an external memory, such as a flash memory, such as a USB (universal serial bus) memory or a compact flash memory. Thus, although computer program 1120 is shown here schematically as a track on the depicted optical disc, computer program 1120 may be stored in any manner suitable for computer program product 1110.

[0067] Although the inventive concept has been described above primarily with reference to certain embodiments, those skilled in the art will readily recognize that embodiments other than those disclosed above are equally possible within the scope of the inventive concept as defined by the appended claims. [Explanation of symbols]

[0068] 100 Video Surveillance Systems 110 Controller 120 Camera 130 Field of View (FoV) 140 scenes 150-1, 150-2 Object 200, 800, 900 image frame series 210 Reflective surface 220-1, 220-2 position 310 Bounding Box 320a, 320b corner point 330 center point 410-1:417b-1, 410-2:417b-2 Key Points 810-2 Mask 1010 Processing circuit mechanism 1020 Communication Interface 1030 Storage medium 1110 Computer program products 1120 Computer Programs 1130 Computer-readable storage medium

Claims

1. 1. A method for detecting a reflection of an object (150-1) in a series of image frames (200, 800, 900), the method being performed by a controller (110), the method comprising: Detecting (S102) objects of a predetermined type (150-1, 150-2), which may be humans, animals, or vehicles such as cars or trucks, in the series of image frames (200, 800, 900) and determining a detection score for each detected object (150-1, 150-2); determining (S104) for each detected object (150-1, 150-2) a distance ratio between unfiltered object position indices identified for the detected objects (150-1, 150-2), wherein the unfiltered object position indices are identified in a current image frame (200, 800, 900) in the series of image frames (200, 800, 900), the object position indices representing points of interest of a target object to be tracked from one image frame to the next, the distance ratio for the detected objects (150-1, 150-2) defining a normalized size of the detected objects (150-1, 150-2), and the unfiltered object position indices are determined for all detected objects (150-1, 150-2) without any of the detected objects (150-1, 150-2) being filtered out due to a detection score being below a detection threshold; determining (S106), for each detected object (150-1, 150-2), a displacement coefficient between a current location (220-1, 220-2) of the detected object (150-1, 150-2) in the current image frame (200, 800, 900) and a previous location of the detected object (150-1, 150-2) in a previous image frame in the series of image frames (200, 800, 900), wherein the displacement coefficient for the detected object (150-1, 150-2) defines a normalized motion for the detected object (150-1, 150-2), the normalized motion referring to motion on a common scale with respect to camera distance; determining (S110) that one of the two detected objects (150-1, 150-2) having a lower detection score is a reflection of the other of the two detected objects (150-1, 150-2) in response to finding a match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2); A method comprising:

2. Finding the match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2) (S108). The method of claim 1 further comprising:

3. Finding the match Verifying (S108a) that a size difference between the normalized size of one of the two detected objects (150-1, 150-2) and the normalized size of the other of the two detected objects (150-1, 150-2) is less than a size threshold; Verifying (S108b) that a difference in motion between the normalized motion of one of the two detected objects (150-1, 150-2) and the normalized motion of the other of the two detected objects (150-1, 150-2) is less than a motion threshold; The method of claim 2 , comprising:

4. performing an action (S112) regarding the detected object (150-2) being the reflection of the other detected object (150-1); and wherein the action comprises one of applying a mask to the detected object (150-2) that is the reflection of the other detected object (150-1) and filtering out the detected object (150-2) that is the reflection of the other detected object (150-1).

5. determining (S114) a location (230) of a reflecting surface (210) that makes one of the detected objects (150-1, 150-2) the reflection of the other detected object (150-1, 150-2) as a midpoint between the locations (220-1, 220-2) of the two detected objects (150-1, 150-2); The method of claim 1 further comprising:

6. The method of claim 5, wherein knowledge of the location (230) of the reflective surface (210) is utilized to detect further objects in the sequence of image frames (200, 800, 900).

7. The method of claim 1 , wherein each of the object location indicators is a Common Object in Context (COCO), a keypoint (410-1:417b-1, 410-2:417b-2), or a MediaPipe Pose keypoint.

8. The method of claim 1 , wherein the object location indices are corner points (320s, 320b) and a center point (330) of a bounding box (310), such as a CenterNet bounding box.

9. A controller (110) for detecting a reflection of an object (150-1) in a series of image frames (200, 800, 900), said controller (110) comprising processing circuitry (1010), said processing circuitry providing said controller (110) with: Detecting objects (150-1, 150-2) of a predetermined type, such as a human, an animal, or a vehicle such as a car or truck, in the sequence of image frames (200, 800, 900) and determining a detection score for each detected object (150-1, 150-2); determining, for each detected object (150-1, 150-2), a distance ratio between unfiltered object position indicators identified for said detected object (150-1, 150-2), said unfiltered object position indicators being identified in a current image frame (200, 800, 900) in said sequence of image frames (200, 800, 900), said object position indicators representing points of interest of a target object being tracked from one image frame to the next, said distance ratio for said detected object (150-1, 150-2) defining a normalized size of said detected object (150-1, 150-2), said unfiltered object position indicators being determined for all detected objects (150-1, 150-2) without any of said detected objects (150-1, 150-2) being filtered out due to a detection score being below a detection threshold; determining, for each detected object (150-1, 150-2), a displacement coefficient between a current location (220-1, 220-2) of the detected object (150-1, 150-2) in the current image frame (200, 800, 900) and a previous location of the detected object (150-1, 150-2) in a previous image frame in the series of image frames (200, 800, 900), wherein the displacement coefficient for the detected object (150-1, 150-2) defines a normalized motion for the detected object (150-1, 150-2), the normalized motion referring to motion on a common scale with respect to camera distance; In response to finding a match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2), determining that one of the two detected objects (150-1, 150-2) having a lower detection score is a reflection of the other of the two detected objects (150-1, 150-2); A controller (110) configured to cause the

10. A video surveillance system (100) comprising a controller (110) according to claim 9 and a camera (120) for capturing said sequence of image frames (200, 800, 900).

11. A computer program (1120) for detecting a reflection of an object (150-1) in a series of image frames (200, 800, 900), comprising: The computer program includes computer code that, when executed by processing circuitry (1010) of a controller (110), causes the controller (110) to: Detecting (S102) objects of a predetermined type (150-1, 150-2), which may be humans, animals, or vehicles such as cars or trucks, in the series of image frames (200, 800, 900) and determining a detection score for each detected object (150-1, 150-2); determining (S104) for each detected object (150-1, 150-2) a distance ratio between unfiltered object position indices identified for the detected objects (150-1, 150-2), wherein the unfiltered object position indices are identified in a current image frame (200, 800, 900) in the series of image frames (200, 800, 900), the object position indices representing points of interest of a target object to be tracked from one image frame to the next, the distance ratio for the detected objects (150-1, 150-2) defining a normalized size of the detected objects (150-1, 150-2), and the unfiltered object position indices are determined for all detected objects (150-1, 150-2) without any of the detected objects (150-1, 150-2) being filtered out due to a detection score being below a detection threshold; determining (S106), for each detected object (150-1, 150-2), a displacement coefficient between a current location (220-1, 220-2) of the detected object (150-1, 150-2) in the current image frame (200, 800, 900) and a previous location of the detected object (150-1, 150-2) in a previous image frame in the series of image frames (200, 800, 900), wherein the displacement coefficient for the detected object (150-1, 150-2) defines a normalized motion for the detected object (150-1, 150-2), the normalized motion referring to motion on a common scale with respect to camera distance; In response to finding a match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2), determining (S110) that one of the two detected objects (150-1, 150-2) having a lower detection score is a reflection of the other of the two detected objects (150-1, 150-2); A computer program (1120) for causing the computer to perform the above.

12. A computer program product (1110) storing a computer program (1120) according to claim 11.

Images