Detection of reflection of object in sequence of image frames

Abstract

To provide a method, a controller, a computer program, and a computer program product for detecting a reflection of an object in a sequence of image frames.SOLUTION: A method includes steps of: detecting objects of a prescribed type in a sequence of image frames; determining a detection score of each of the detected object; determining a distance ratio between unfiltered object position indicators identified for each of the detected objects; determining, for each of the detected objects, a displacement factor between a current location of the detected object and a previous location of the detected object; and determining that one of the two detected objects which has a lower detection score is a reflection of the other of the two detected objects in response to finding a match between normalized sizes and normalized movements of the detected objects.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The embodiments presented in this specification relate to a method, a controller, a computer program, and a computer program product for detecting reflections of an object in a series of image frames.

Background Art

[0002] Generally, object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, animals, vehicles, etc.). Some object detection algorithms generate keypoints or other types of object position indicators. Such points, or indicators, generally represent the points of interest of a target object that are tracked from one image frame to the next.

[0003] There are also scenarios or environments where object detection as described above becomes difficult. Taking the urban environment as an example, this is a type of environment with a large number of reflective surfaces. Reflective surfaces include not only mirrors but also sheets made of glass or other types of reflective materials. These surfaces generally pose problems for object detection algorithms. In some cases, both the detected object and its reflection are counted. This becomes an issue in applications that count the number of detected objects, and there is a possibility that some detected objects are double-counted. In other cases, the reflection may be completely missed. This may become an issue in applications where the detected object is masked and the reflection is left unmasked.

[0004] U.S. Patent Application Publication No. 2020 / 0175694 (A1) discloses an information processing apparatus including a specific circuit and a counting circuit. The specific circuit identifies two or more moving objects corresponding to the same actual moving object among the moving objects based on the similarity of the speeds according to the changes in the positions of a plurality of moving objects in an image. The counting circuit counts the number of actual moving objects corresponding to the moving objects based on the identification result of the specific circuit.

[0005] However, in reality, reflections do not perfectly match the tracking object that is reflecting them. Furthermore, reflections may appear smaller or larger than the tracking object. Additionally, when reflections occur on translucent surfaces, the detail of the reflection in the image frame will be less than that of the tracking object. Moreover, reflections may only include a portion of the tracking object. This could lead to a specific circuit within the information processing device described in U.S. Patent Application Publication 2020 / 0175694(A1) making an incorrect determination as to whether two or more moving objects correspond to the same actual moving object within the moving object. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Patent Application Publication No. 2020 / 0175694(A1) [Overview of the Initiative]

[0007] The object of the embodiments described herein is to address the above-mentioned problems and to provide improved detection of objects on reflective surfaces.

[0008] Accordingly, according to the first aspect, a method is presented for detecting reflections of objects in a series of image frames. The method is performed by a controller. The method includes detecting objects of a predetermined type in a series of image frames and determining a detection score for each detected object. For each detected object, the method includes determining the distance ratio between unfiltered object position indicators identified for the detected object. The unfiltered object position indicators are identified in the current image frame of the series of image frames. The distance ratio for the detected object defines the normalized size of the detected object. For each detected object, the method includes determining a displacement coefficient between the current location of the detected object in the current image frame and the previous location of the detected object in a previous image frame of the series of image frames. The displacement coefficient for the detected object defines the normalized motion for the detected object. In response to finding a match between the normalized size and normalized motion of two of the detected objects, the method includes determining that the one of these two detected objects with the lower detection score is a reflection of the other of these two detected objects.

[0009] According to a second aspect, a controller is presented for detecting reflections of objects in a series of image frames. The controller includes a processing circuit mechanism. The processing circuit mechanism is configured to cause the controller to detect objects of a predetermined type in a series of image frames and to determine a detection score for each detected object. The processing circuit mechanism is configured to cause the controller to determine, for each detected object, the distance ratio between unfiltered object position indicators identified for the detected object. The unfiltered object position indicators are identified in the current image frame in the series of image frames. The distance ratio for the detected object defines the normalized size of the detected object. The processing circuit mechanism is configured to cause the controller to determine, for each detected object, the displacement coefficient between the current location of the detected object in the current image frame and the previous location of the detected object in a previous image frame in the series of image frames. The displacement coefficient for the detected object defines the normalized motion of the detected object. The processing circuit mechanism is configured to cause the controller to determine, in response to finding a match between the normalized size and normalized motion of two of the detected objects, that the one of the two detected objects having the lower detection score is a reflection of the other of the two detected objects.

[0010] According to a third embodiment, a video surveillance system is presented. The video surveillance system comprises a controller according to the second embodiment and a camera for capturing a series of image frames.

[0011] According to a fourth aspect, a computer program is presented to the controller, which, when executed by the controller, includes computer program code that causes the controller to perform the method according to the first aspect.

[0012] According to the fifth aspect, a computer program product is presented comprising a computer program according to the fourth aspect and a computer-readable storage medium in which the computer program is stored. The computer-readable storage medium may be a non-temporary computer-readable storage medium.

[0013] Advantageously, these embodiments provide computationally efficient and accurate detection of objects in scenarios or environments with reflective surfaces.

[0014] Other objects, features, and advantages of the embodiments described herein will become apparent from the following detailed disclosure, appended dependent claims, and drawings.

[0015] In general, all terms used in the claims shall be construed in accordance with their ordinary meaning in the art unless expressly otherwise provided herein. All references to “one (a) / one (an) / the” elements, apparatus, components, means, modules, steps, etc.” shall be openly construed as referring to at least one instance of an element, apparatus, component, means, module, step, etc. unless expressly otherwise stated. No steps of any method disclosed herein are required to be performed in the exact order disclosed unless expressly stated otherwise.

[0016] Next, the concept of the present invention will be explained as an example with reference to the attached drawings. [Brief explanation of the drawing]

[0017] [Figure 1] This is a schematic diagram showing a video surveillance system according to an embodiment. [Figure 2] This figure schematically shows an image frame including a detected object and its reflection, according to one embodiment. [Figure 3] This figure schematically shows a detected object and a bounding box surrounding the detected object according to one embodiment. [Figure 4]A diagram schematically showing a detected object and key points of the detected object according to one embodiment. [Figure 5] A diagram schematically showing key points of a detected object and key points of a reflection of the detected object according to one embodiment. [Figure 6] A flowchart of a method according to an embodiment. [Figure 7] A diagram schematically showing an image frame according to one embodiment after a first action is performed. [Figure 8] A diagram schematically showing an image frame according to one embodiment after a second action is performed. [Figure 9] A schematic diagram showing functional units of a controller according to one embodiment. [Figure 10] A diagram showing an example of a computer program product including a computer-readable storage medium according to one embodiment.

Best Mode for Carrying Out the Invention

[0018] Hereinafter, the concept of the present invention will be more fully described with reference to the accompanying drawings showing specific embodiments of the concept of the present invention. However, the concept of this invention can be embodied in many different forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete and will fully convey the scope of the concept of the present invention to those skilled in the art. Like numbers refer to like elements throughout this specification. Steps or features shown in dashed lines should be considered optional.

[0019] FIG. 1 is a schematic diagram showing a video surveillance system 100 to which the embodiments presented in this specification can be applied. Camera 120 is configured to capture image frames within the field of view (FoV) 130 of scene 140. In the illustration of FIG. 1, scene 140 includes an object 150-1 in the form of a human. Camera 120 includes a controller 110, is collocated with, integrated with, or at least operably connected to controller 110. Object 150-1 is assumed to represent an object to be tracked from image frame to image frame as captured by camera 120 and analyzed by controller 110. For this purpose, controller 110 may implement an object detection algorithm.

[0020] Embodiments disclosed herein relate to techniques for detecting reflections of object 150-1 in a series of image frames. To obtain such techniques, there is provided a controller 110, a method executed by controller 110, and a computer program product including code, such as in the form of a computer program, that when executed on controller 110 causes controller 110 to execute the method.

[0021] FIG. 2 schematically shows an image frame 200. Image frame 200 depicts an object 150-1 in the form of a human, like scene 140. Object 150-1 has a location marked with reference numeral 220-1 within image frame 200. Image frame 200 further depicts a reflective surface 210. Reflective surface 210 has a location marked with reference numeral 230 within image frame 200. Due to reflective surface 210, an additional object 150-2 is also visible. Object 150-2 is a reflection of object 150-1 caused by object 150-1 being reflected by reflective surface 210. Object 150-2 has a location marked with reference numeral 220-2 within image frame 200.

[0022] The inventors noticed that what objects 150-1 and 150-2 have in common is their object position indicator. Here, we refer to Figures 3 and 4, which schematically show a human-shaped object 150-1 as in Scene 140. Figures 3 and 4 show different types of object position indicators. In Figure 3, object 150-1 is enclosed by a bounding box 310. The bounding box 310 is defined by two corner points 320a, 320b and one center point 330. The center point 330 may represent the location 220-1 of object 150-1. The corner points 320a, 320b and the center point 330 may be, for example, CenterNet keypoints. The bounding box may be, for example, a CenterNet bounding box. In this regard, an object position indicator may be defined by two or more bounding boxes. That is, two or more bounding boxes may be used to enclose the detected object 150-1. In other words, the first bounding box may surround the first part (such as the head) of the detected object 150-1, and the second bounding box may surround the second part (such as the torso) of the detected object 150-1, and so on. In Figure 4, object 150-1 is overlaid with object position indicators 410-1 to 417b-1, as listed in Table 1. These object position indicators can be considered as simplified versions of COCO keypoints or MediaPipe Pose keypoints. TIFF2023179374000002.tif63170

[0023] The reflected object 150-2 has the same setup of object position indicators but is reflected. By analyzing the object position indicators, it is possible to determine object reflection pairs at the object level, i.e., one tracked object 150-1 and its reflected corresponding object 150-2, or even at the object position indicator level. Here, refer to Figure 5 schematically showing the object position indicators 410-1 to 417b-1 of object 150-1 (not shown) as listed in Table 1. Figure 5 also shows the position 220-1 of object 150-1. Figure 5 further shows the object position indicators 410-2 to 417b-2 of object 150-2 (not shown), which is the reflection of object 150-1 on the reflective surface 210 at location 230. The object position indicators 410-2 to 417b-2 correspond to the object position indicators listed in Table 1. In other words, object position indicator 410-2 represents the head of object 150-2, and object position indicator 417b-2 represents the left foot of object 150-2.

[0024] Figure 6 is a flowchart illustrating an embodiment of a method for detecting reflections of object 150-1 in a series of image frames 200, 800, and 900. The method is performed by a controller 110. The method is advantageously provided as a computer program 1120.

[0025] S102: The controller 110 detects objects 150-1 and 150-2 of a predetermined type in a series of image frames 200, 800, and 900. The controller 110 determines a detection score for each detected object 150-1 and 150-2.

[0026] In some examples, in addition to the overall detection score determined for each detected object 150-1, 150-2, a detection score is also determined for each object location index identified for the detected objects 150-1, 150-2. The detection score for individual object location indexes may only be relevant if the detection score for the detected objects 150-1, 150-2 exceeds a certain threshold. Conversely, in some embodiments, the detection score for a given detected object 150-1, 150-2 is a function of the detection scores for all unfiltered object location indexes identified for that given detected object 150-1, 150-2.

[0027] S104: For each detected object 150-1, 150-2, the controller 110 determines the distance ratio between the unfiltered object position indicators identified for the detected objects 150-1, 150-2. The unfiltered object position indicators are identified in the current image frame 200, 800, 900 in a series of image frames 200, 800, 900. The distance ratio of the detected objects 150-1, 150-2 defines the normalized size of the detected objects 150-1, 150-2.

[0028] The size of the detected object is determined for each detected object as a whole, so that size can be considered the normalized size of the detected object.

[0029] S106: For each detected object 150-1, 150-2, the controller 110 determines a displacement coefficient between the current locations 220-1, 220-2 of the detected object 150-1, 150-2 in the current image frames 200, 800, 900 and the previous locations of the detected object 150-1, 150-2 in previous image frames within the series of image frames 200, 800, 900. The displacement coefficient of the detected object 150-1, 150-2 defines the normalized motion of the detected object 150-1, 150-2.

[0030] Therefore, with respect to previous image frames in the series of image frames 200, 800, and 900, the controller 110 uses the historical data to obtain information about the location of the detected object in the previous image frame. A measure of the detected object's movement can be obtained based on how far the detected object has moved from the previous image frame to the current image frame. Since the movement of the detected object is determined for each detected object as a whole, the movement can be considered the normalized movement of the detected object.

[0031] Next, the normalized sizes and normalized motions of the two detected objects 150-1 and 150-2 can be compared with each other. That is, the normalized size of one of the two detected objects 150-1 and 150-2 is compared with the normalized size of the other detected object 150-1 and 150-2, and the normalized motion of one of the two detected objects 150-1 and 150-2 is compared with the normalized motion of the other detected object 150-1 and 150-2. Furthermore, in this respect, in general terms, normalized size and normalized motion mean that a particular given scale (size of normalized size and motion of normalized motion) is scaled to a common scale or otherwise recalculated. As a non-restrictive example, if we assume that object 150-1 and its reflection are viewed at different distances from camera 120, then the effect arises that object 150-1 and its reflection appear to be different sizes from each other, with the one closest to the camera appearing larger and vice versa. By using normalized size, object 150-1 and its reflection are scaled to a common size scale that explains this effect, or are otherwise recalculated, so that object 150-1 and its reflection appear to be one object of the same size. Similarly, under the same assumption, there is also the effect that object 150-1 and its reflection appear to be moving at different speeds from each other, with the one closest to the camera appearing to move faster and vice versa. By using normalized motion, object 150-1 and its reflection are scaled to a common motion scale that explains this effect, or are otherwise recalculated, so that object 150-1 and its reflection appear to be moving at the same speed.

[0032] If the normalized size and normalized motion are similar for two detected objects 150-1 and 150-2, then it can be concluded that these two detected objects 150-1 and 150-2 have similar size and similar motion. This indicates that one of the two detected objects 150-1 and 150-2 is a reflection of the other of the two detected objects 150-1 and 150-2.

[0033] S110: In response to finding a match between the normalized size and normalized motion of the two detected objects 150-1 and 150-2, the controller 110 determines that the one of the two detected objects 150-1 and 150-2 having the lower detection score is a reflection of the other of the two detected objects 150-1 and 150-2.

[0034] Therefore, by considering the detection scores of the two detected objects 150-1 and 150-2, detected object 150-2, which has a relatively low detection score, is classified as a reflection of detected object 150-1, which has a relatively high detection score.

[0035] Next, embodiments relating to further details of the detection of reflections of object 150-1 in a series of image frames 200, 800, and 900, performed by the controller 110 are disclosed.

[0036] As disclosed above, the controller 110 detects objects 150-1 and 150-2 of a predetermined type in a series of image frames 200, 800, and 900. In this regard, the objects 150-1 and 150-2 configured to be detected by the controller 110 can be of different types. In some non-limiting examples, the predetermined type could be a person, an animal, or a vehicle such as a car or truck. Thus, the controller 110 can be trained and configured to detect objects 150-1 and 150-2 of a predetermined type.

[0037] As disclosed above, for each detected object 150-1, 150-2, the controller 110 determines the distance ratio between the unfiltered object position indices identified for the detected objects 150-1, 150-2. Generally speaking, the unfiltered object position indices are the object position indices determined for objects that were initially detected with a low detection score without being excluded by filtering. That is, the unfiltered object position indices refer to the object position indices before thresholding is performed to mitigate false detections in the scene. Generally, reflections have lower detection scores than non-reflectives, so the unfiltered object position indices are used to ensure that detected object 150-2, which is a reflection, is not missed. That is, some of the unfiltered object position indices may belong to objects that would otherwise be excluded or discarded by filtering because they generate a low detection score. Therefore, in some embodiments, the unfiltered object position indices are determined for all detected objects 150-1, 150-2 without either of them being excluded by filtering (because their detection score is below the detection threshold).

[0038] Furthermore, in this regard, there can be different types of object position indicators.

[0039] In some non-restrictive examples, each object position indicator is a Common Object (COCO) keypoint 410-1:417b-1, 410-2:417b-2, or MediaPipe Pose keypoint within the context. In this regard, an object position indicator does not need to be defined by all COCO keypoints or MediaPipe Pose keypoints; rather, a subset of these keypoints can be used as an object position indicator. This is illustrated in Figure 4, mentioned above.

[0040] In other non-restrictive examples, the object position indicators are the corner points 320s, 320b, and center point 330 of a bounding box 310, such as a CenterNet bounding box. This is shown in Figure 3 mentioned above.

[0041] As disclosed below, it is determined that one of the two detected objects 150-1, 150-2 is a reflection of the other of the two detected objects 150-1, 150-2 in response to finding a match between the normalized sizes and / or normalized motions of these two detected objects 150-1, 150-2. Thus, in some embodiments, the controller 110 is configured to perform the (optional) step S108.

[0042] S108: The controller 110 finds a match between two normalized sizes and / or normalized motions of the detected objects 150-1 and 150-2.

[0043] There may be different methods by which the controller 110 finds a match between two normalized sizes and / or normalized motions of two of the detected objects 150-1 and 150-2. In some embodiments, finding this match is performed by comparing the normalized size and / or normalized motion of the first detected object 150-1 with the normalized size and / or normalized motion of the second detected object 150-2. In particular, in some embodiments, the controller 110 is configured to perform (optional) steps S108a and / or S108b as part of step S108.

[0044] S108a: The controller 110 confirms that the size difference between the normalized size of one of the two detected objects 150-1 and 150-2 and the normalized size of the other of the two detected objects 150-1 and 150-2 is less than the size threshold.

[0045] Therefore, the smaller the size difference between the normalized size of one of the two detected objects 150-1 and 150-2 and the normalized size of the other of the two detected objects 150-1 and 150-2, the more likely they are to match.

[0046] S108b: The controller 110 confirms that the difference in motion between the normalized motion of one of the two detected objects 150-1 and 150-2 and the normalized motion of the other of the two detected objects 150-1 and 150-2 is less than the motion threshold.

[0047] Therefore, the smaller the difference in motion between the normalized motion of one of the two detected objects 150-1 and 150-2 and the normalized motion of the other of the two detected objects 150-1 and 150-2, the more likely they are to match.

[0048] In some embodiments, matching is performed at the object position index level. That is, instead of simply comparing the normalized size and / or normalized motion of the first detected object 150-1 with the normalized size and / or normalized motion of the second detected object 150-2, the comparison (in terms of size and / or motion) is performed between a pair of object position indexes of one of the two detected objects 150-1, 150-2 and a pair of object position indexes of the other of the two detected objects 150-1, 150-2.

[0049] That is, for the sake of explanation, let us assume that the two detected objects 150-1 and 150-2 represent humans, with detectable knees on both legs, and that these are represented by object position indicators in both detected objects 150-1 and 150-2. Then, a first comparison can be made between the normalized distance between the knees according to the object position indicator of one of the two detected objects 150-1 and 150-2 and the normalized distance between the knees according to the object position indicator of the other of the two detected objects 150-1 and 150-2. Furthermore, a second comparison can be made between the normalized movement of the knee given by the object position indicator of one of the two detected objects 150-1 and 150-2 and the normalized movement of the knee given by the object position indicator of the other of the two detected objects 150-1 and 150-2. Further such comparisons regarding size and / or movement can be made for further portions of the detected objects in order to accumulate matching scores for the two detected objects 150-1 and 150-2.

[0050] In some embodiments, the controller 110 takes some action when it determines that one of the two detected objects 150-1, 150-2 is a reflection of the other of the two detected objects 150-1, 150-2. Thus, in some embodiments, the controller 110 is configured to perform the (optional) step S112.

[0051] S112: The controller 110 performs an action on the detected object 150-2, which is a reflection of the other detected object 150-1.

[0052] There may be different types of actions performed by the controller 110 in step S112. In some non-limiting examples, the action may include applying a mask 810-2 to detected object 150-2, which is a reflection of the other detected object 150-1. Hereinafter, Figure 7 schematically shows image frame 800. Image frame 800 illustrates how the first mask 810-1 is positioned where object 150-1 appears in Figure 2, and how the second mask 810-2 is positioned where object 150-2 appears in Figure 2. In some non-limiting examples, the action may include filtering out detected object 150-2, which is a reflection of the other detected object 150-1, or lowering the detection score threshold for detected object 150-2, which is a reflection of the other detected object 150-1. Hereinafter, Figure 8 schematically shows image frame 900. In image frame 900, object 150-1 is depicted, similar to Figure 2, but object 150-2 has been filtered out so that it does not appear in image frame 900. In this respect, object 150-2, which represents a reflection, is shown as having been filtered out of the image frame. Note that filtering out generally refers to filtering out the detected object 150-2 from any counting or other type of calculation so that object 150-1 is not counted twice.

[0053] In some embodiments, as shown in Figure 2, reflection is caused by the reflective surface 210 being positioned between two detected objects 150-1, 150-2. Generally speaking, the reflective surface 210 is positioned midway between the locations 220-1, 220-2 of the two detected objects 150-1, 150-2. Thus, knowledge of the locations 220-1, 220-2 of the two detected objects 150-1, 150-2 is used to determine the location 230 of the reflective surface 210. Accordingly, in some embodiments, the controller 110 is configured to perform the (optional) step S114.

[0054] S114: The controller 110 determines the location 230 of the reflective surface 210 such that one of the detected objects 150-1 and 150-2 becomes a reflection of the other detected object 150-1 or 150-2. The location 230 of the reflective surface 210 is determined as the midpoint between the locations 220-1 and 220-2 of the two detected objects 150-1 and 150-2.

[0055] In some embodiments, knowledge of the location 230 of the reflective surface 210 is used to detect further objects in a series of image frames 200, 800, and 900. Since the location 230 of the reflective surface 210 can be used as a reference point, reference line, or even a reference plane, knowledge of the location 230 can improve the chances of detecting reflective objects under challenging lighting conditions in the same scene or in future image frames of a scene in which the reflective surface 210 is still present in the scene.

[0056] It should be noted that, by matching that references the normalized sizes of objects 150-1 and 150-2, there is no need to consider that the object position indices 410-2~417b-2 of object 150-2 are mirrored compared to the object position indices 410-1~417b-1 of object 150-1. For the same reason, by matching that references the normalized motions of objects 150-1 and 150-2, the normalized motion of object 150-1 will be the same as the normalized motion of object 150-2, except for the possibility of a sign change depending on the arrangement of the reflective surface 210 relative to the detected objects 150-1 and 150-2. For example, with the arrangement of the reflective surface 210 as shown in Figure 2, if object 150-1 appears to be moving to the left in Figure 2, then object 150-2 will appear to be moving to the right in Figure 2. To resolve this potential issue, the normalized motion may be represented only by its magnitude, without considering its direction. Furthermore, in any matching performed at the object position index level, reflection should be taken into account if the matching includes any object position index where both a right and a left object position index exist for each object, such as object position index 412a-1, 412b-1, etc.

[0057] The methods and controller 110 disclosed to date may be used as part of a technique to improve the detection of people and other objects 150-1 in a scene 140 having a glass surface. The glass surface may be both reflective and transmissive. Reflected images may be darker than images of actual people or objects. The same is true for people or other objects 150-1 visible through the glass, especially under certain lighting conditions. As already disclosed, if the purpose of detecting object 150-1 is to mask object 150-1, there is a risk that a detection threshold that is too strict may leave a mirror image of object 150-1 unmasked and potentially identifiable. Conversely, if the purpose of detecting object 150-1 is counting, there is a risk of double counting of object 150-1 if the detection threshold is too coarse.

[0058] Therefore, in some embodiments, regions where reflective and transmitted images are likely to occur are identified. Under certain lighting conditions, it may be easier to identify such regions, and knowledge gained under favorable conditions may be used under more difficult conditions. If a surface can be identified where reflections 150-2 of object 150-1 are often found, but reflections are similarly rarely found (e.g., keypoints do not coincide), then this surface is likely to be both reflective and transmissive.

[0059] Illumination conditions may affect how objects 150-1 and 150-2 are reflected and / or transmitted. For example, it is generally relatively easy to investigate highly reflective or highly transmissive surfaces, which makes detection clearer and more reliable. On the other hand, it is generally difficult to detect objects under dim lighting conditions. Therefore, information about illumination conditions may be used when determining whether an object is reflective or not. For example, if dim lighting conditions are indicated, the threshold may be lowered. This can also be useful in tuning keypoint reflection algorithms. For example, if it is found that a given region of an image is more likely to contain reflections than transmissions, a more aggressive search for keypoint pairs can be performed in this given region of the image. Furthermore, bright lighting conditions generally produce more reflective objects than transmissive objects. This knowledge can also be used to assist the algorithm. Therefore, if the lighting conditions indicate a higher likelihood of reflections, a more aggressive search for keypoint pairs may be performed.

[0060] Here, we briefly describe a possible approach. Reflections can be found using the methods described above. The number of recorded reflections in a given portion of scene 140 is saved over time. This can be considered as the generation of a heatmap. If there are many recorded reflections but also many true targets for a given surface, this is marked on the heatmap. In a privacy masking application, a static mask can be applied to this surface to ensure that reflections are masked even if they fall below the current detection threshold. Alternatively, the mask threshold required to mask object 150-2 can be lowered in this area. For each image frame 200, it may be possible to determine whether detected objects 150-1 and 150-2 are reflections by finding reflection pairs in scene 140. In this way, by building data over time using the probability of whether object 150-1 is a reflection or not, it may be possible to predict the probability of whether a new object located in the same area of ​​an image frame in the future is a reflection or not.

[0061] Figure 9 schematically shows the components of the controller 110 according to the embodiment in terms of the number of functional units. The processing circuit mechanism 1010 is provided using one or any combination of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., which can execute software instructions stored in a computer program product 1110 (as shown in Figure 10), for example in the form of a storage medium 1030. The processing circuit mechanism 1010 may further be provided as at least one application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0062] In particular, the processing circuit mechanism 1010 is configured to cause the controller 110 to execute a series of operations or steps as disclosed above. For example, the storage medium 1030 may store a series of operations, and the processing circuit mechanism 1010 may be configured to retrieve a series of operations from the storage medium 1030 in order to cause the controller 110 to execute the series of operations. The series of operations may be provided as a series of executable instructions.

[0063] In this manner, the processing circuit mechanism 1010 is arranged to perform the methods disclosed herein. The storage medium 1030 may also include persistent storage, which may be any single or combination of magnetic memory, optical memory, solid-state memory, or remote-mount memory. The controller 110 may further include at least a communication interface 1020 configured for communication with the camera 120, and other functions, nodes, entities, and / or devices, potentially functions of a video surveillance system. Thus, the communication interface 1020 may include one or more transmitters and receivers, including analog and digital components. The processing circuit mechanism 1010 controls the general operation of the controller 110, for example, by transmitting data and control signals to the communication interface 1020 and the storage medium 1030, by receiving data and reports from the communication interface 1020, and by retrieving data and instructions from the storage medium 1030. Other components and associated functionalities of the controller 110 are omitted in order not to obscure the concepts presented herein.

[0064] The controller 110 may be provided as a standalone device or as part of at least one further device. For example, the controller 110 and camera 120 may be part of a video surveillance system 100. Optionally, the controller 110 may be integrated with the camera 120. A first portion of instructions executed by the controller 110 may be performed on a first device, and a second portion of instructions executed by the controller 110 may be performed on a second device, and the embodiments disclosed herein are not limited to a specific number of devices on which instructions executed by the controller 110 can be performed. Accordingly, the methods according to the embodiments disclosed herein are suitable for execution by a controller 110 residing in a cloud computing environment. Accordingly, although a single processing circuit 1010 is shown in Figure 9, the processing circuit 1010 may be distributed among multiple devices or nodes. The same applies to the computer program 1120 in Figure 10.

[0065] Figure 10 shows an example of a computer program product 1110 including a computer-readable storage medium 1130. This computer-readable storage medium 1130 can store a computer program 1120, which can cause a processing circuit mechanism 1010 and entities and devices operably coupled thereto, such as a communication interface 1020 and the storage medium 1030, to perform the methods according to the embodiments described herein. Thus, the computer program 1120 and / or the computer program product 1110 can provide means for performing any step as disclosed herein.

[0066] In the example in Figure 10, the computer program product 1110 is shown as an optical disc such as a CD (Compact Disc), DVD (Digital Multipurpose Disc), or Blu-ray disc. The computer program product 1110 can also be embodied as a memory such as Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), or Electrically Erasable Programmable Read-Only Memory (EEPROM), and more specifically as a non-volatile storage medium for a device in external memory such as USB (Universal Serial Bus) memory or flash memory such as CompactFlash memory. Thus, although the computer program 1120 is schematically shown here as a track on the optical disc depicted, the computer program 1120 can be stored in any way suitable for the computer program product 1110.

[0067] The concept of the present invention has been described above, primarily with reference to several embodiments. However, as will be readily apparent to those skilled in the art, other embodiments are also possible within the scope of the concept of the present invention as defined by the appended claims. [Explanation of Symbols]

[0068] 100 Video Surveillance Systems 110 Controller 120 Cameras 130 Field of View (FoV) 140 scenes 150-1, 150-2 Object 200, 800, 900 series of image frames 210 Reflective surface 220-1, 220-2 position 310 Bounding Boxes 320a, 320b corner point 330 center point 410-1:417b-1, 410-2:417b-2 Key Points 810-2 Mask 1010 Processing circuit mechanism 1020 Communication Interface 1030 Storage medium 1110 Computer Program Products 1120 Computer Programs 1130 Computer-readable storage medium

Claims

1. A method for detecting a reflection of an object (150-1) in a sequence of image frames (200, 800, 900), the method being executed by a controller (110), the method comprising: Detecting (S102) objects of a predetermined type (150-1, 150-2), which may be humans, animals, or vehicles such as cars or trucks, in the sequence of image frames (200, 800, 900) and determining a detection score for each detected object (150-1, 150-2); determining (S104) for each detected object (150-1, 150-2) a distance ratio between unfiltered object position indices identified for said detected objects (150-1, 150-2), said unfiltered object position indices being identified in a current image frame (200, 800, 900) in said sequence of image frames (200, 800, 900), said object position indices representing points of interest of a target object to be tracked from one image frame to the next, said distance ratio for said detected objects (150-1, 150-2) defining a normalized size of said detected objects (150-1, 150-2), said unfiltered object position indices being determined for all detected objects (150-1, 150-2) without any of said detected objects (150-1, 150-2) being filtered out due to a detection score being below a detection threshold; determining (S106), for each detected object (150-1, 150-2), a displacement coefficient between a current location (220-1, 220-2) of the detected object (150-1, 150-2) in the current image frame (200, 800, 900) and a previous location of the detected object (150-1, 150-2) in a previous image frame in the series of image frames (200, 800, 900), wherein the displacement coefficient for the detected object (150-1, 150-2) defines a normalized motion for the detected object (150-1, 150-2), the normalized motion referring to motion in a common scale with respect to camera distance; determining (S110) that one of the two detected objects (150-1, 150-2) having a lower detection score is a reflection of the other of the two detected objects (150-1, 150-2) in response to finding a match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2); A method comprising:

2. Finding the match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2) (S108). The method of claim 1 further comprising:

3. Finding the match Verifying (S108a) that a size difference between the normalized size of one of the two detected objects (150-1, 150-2) and the normalized size of the other of the two detected objects (150-1, 150-2) is less than a size threshold; Verifying (S108b) that a difference in motion between the normalized motion of one of the two detected objects (150-1, 150-2) and the normalized motion of the other of the two detected objects (150-1, 150-2) is less than a motion threshold; The method of claim 2 , comprising:

4. Performing an action (S112) regarding the detected object (150-2) being the reflection of the other detected object (150-1). and wherein the action comprises one of applying a mask (810-2) to the detected object (150-2) that is the reflection of the other detected object (150-1) and filtering out the detected object (150-2) that is the reflection of the other detected object (150-1).

5. determining (S114) a location (230) of a reflecting surface (210) that makes one of the detected objects (150-1, 150-2) a reflection of the other detected object (150-1, 150-2) as a midpoint between the locations (220-1, 220-2) of the two detected objects (150-1, 150-2); The method of claim 1 further comprising:

6. The method of claim 5 , wherein knowledge of the location (230) of the reflective surface (210) is utilized to detect further objects in the sequence of image frames (200, 800, 900).

7. The method of claim 1 , wherein each of the object location indicators is a Common Object in Context (COCO), a keypoint (410-1: 417b-1, 410-2: 417b-2), or a MediaPipe Pose keypoint.

8. The method of claim 1 , wherein the object position indices are corner points (320s, 320b) and a center point (330) of a bounding box (310), such as a CenterNet bounding box.

9. A controller (110) for detecting a reflection of an object (150-1) in a series of image frames (200, 800, 900), said controller (110) comprising processing circuitry (1010), said processing circuitry providing to said controller (110): detecting objects (150-1, 150-2) of a predetermined type, such as a human, an animal, or a vehicle, such as a car or truck, in the sequence of image frames (200, 800, 900) and determining a detection score for each detected object (150-1, 150-2); determining, for each detected object (150-1, 150-2), a distance ratio between unfiltered object position indices identified for said detected objects (150-1, 150-2), said unfiltered object position indices being identified in a current image frame (200, 800, 900) in said sequence of image frames (200, 800, 900), said object position indices representing points of interest of a target object to be tracked from one image frame to the next, said distance ratio for said detected objects (150-1, 150-2) defining a normalized size of said detected objects (150-1, 150-2), said unfiltered object position indices being determined for all detected objects (150-1, 150-2) without any of said detected objects (150-1, 150-2) being filtered out due to a detection score being below a detection threshold; determining, for each detected object (150-1, 150-2), a displacement coefficient between a current location (220-1, 220-2) of the detected object (150-1, 150-2) in the current image frame (200, 800, 900) and a previous location of the detected object (150-1, 150-2) in a previous image frame in the series of image frames (200, 800, 900), wherein the displacement coefficient for the detected object (150-1, 150-2) defines a normalized motion for the detected object (150-1, 150-2), the normalized motion referring to motion in a common scale with respect to camera distance; in response to finding a match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2), determining that one of the two detected objects (150-1, 150-2) having a lower detection score is a reflection of the other of the two detected objects (150-1, 150-2); A controller (110) configured to:

10. A video surveillance system (100) comprising a controller (110) according to claim 9 and a camera (120) for capturing said sequence of image frames (200, 800, 900).

11. A computer program (1120) for detecting a reflection of an object (150-1) in a sequence of image frames (200, 800, 900), comprising: The computer program includes computer code that, when executed by processing circuitry (1010) of a controller (110), causes the controller (110) to: Detecting (S102) objects of a predetermined type (150-1, 150-2), which may be humans, animals, or vehicles such as cars or trucks, in the sequence of image frames (200, 800, 900) and determining a detection score for each detected object (150-1, 150-2); determining (S104) for each detected object (150-1, 150-2) a distance ratio between unfiltered object position indices identified for said detected objects (150-1, 150-2), said unfiltered object position indices being identified in a current image frame (200, 800, 900) in said sequence of image frames (200, 800, 900), said object position indices representing points of interest of a target object to be tracked from one image frame to the next, said distance ratio for said detected objects (150-1, 150-2) defining a normalized size of said detected objects (150-1, 150-2), said unfiltered object position indices being determined for all detected objects (150-1, 150-2) without any of said detected objects (150-1, 150-2) being filtered out due to a detection score being below a detection threshold; determining (S106), for each detected object (150-1, 150-2), a displacement coefficient between a current location (220-1, 220-2) of the detected object (150-1, 150-2) in the current image frame (200, 800, 900) and a previous location of the detected object (150-1, 150-2) in a previous image frame in the series of image frames (200, 800, 900), wherein the displacement coefficient for the detected object (150-1, 150-2) defines a normalized motion for the detected object (150-1, 150-2), the normalized motion referring to motion in a common scale with respect to camera distance; determining (S110) that one of the two detected objects (150-1, 150-2) having a lower detection score is a reflection of the other of the two detected objects (150-1, 150-2) in response to finding a match between the normalized sizes and the normalized motions of two of the detected objects (150-1, 150-2); A computer program (1120) for causing the computer to perform the above steps.

12. A computer program product (1110) comprising the computer program (1120) of claim 11 and a computer-readable storage medium (1130) storing the computer program.

Images