Method and image processing device for detecting foreign objects on transparent protective cover of camera

Abstract

A method for determining whether a transparent protective cover of a camera including a lens-based optical imaging system is partially covered by a foreign object is disclosed. The method includes: acquiring (402) a first captured image frame captured by the camera at a first depth of field; acquiring (404) a second captured image frame captured by the camera at a second depth of field different from the first depth of field; and determining whether the protective cover is partially covered by the foreign object by analyzing whether the first and second captured image frames are affected by the presence of the foreign object on the protective cover, such that a difference between the first depth of field and the second depth of field results in a difference in brightness patterns of corresponding pixels of the first image frame and the second image frame. The first image frame is based on the first captured image frame and the second image frame is based on the second captured image frame.

Description

Technical Field

[0001] Embodiments of the present invention relate to a method and an image processing device for determining whether a transparent protective cover of a camera is partially covered by a foreign object on the transparent protective cover. In other words, embodiments of the present invention relate to a method and an image processing device for detecting a foreign object adhered to a protective cover of a camera. A corresponding computer program and a computer program carrier are also disclosed. Background Art

[0002] Surveillance of the public using imaging, especially video imaging, is common in many areas around the world. Areas that may need to be monitored are, for example, banks, stores, and other areas that require security, such as schools and government facilities. Other areas that may need to be monitored are processing, manufacturing, and logistics applications where video surveillance is primarily used to monitor the process.

[0003] In order to protect the camera from, for example, dust (dirt) and vandalism (vandalism), many surveillance cameras are equipped with a protective housing. However, monitoring using the camera may still be disturbed by dust or small objects on the protective cover. Dust or objects may reduce the video quality by distorting the image, or even make it more difficult to detect and / or track people or objects that should be detected and / or tracked by the camera.

[0004] To overcome this problem, the protective cover can be cleaned when necessary. However, there is the problem of detecting whether there is dust or an object on the protective cover that needs cleaning, whether it is from deliberate damage or naturally occurring. Summary of the invention

[0005] Therefore, an object of embodiments herein may be to eliminate some of the above problems, or at least reduce their impact. Specifically, an object may be to detect foreign objects adhered to a transparent protective cover of a camera.

[0006] According to one aspect, the object is achieved by a method, which is performed by an image processing device, for determining whether a transparent protective cover of a camera including a lens-based optical imaging system is partially covered by a foreign object on the protective cover.

[0007] The method includes acquiring a first captured image frame captured by the camera at a first depth of field.

[0008] The method further includes acquiring a second captured image frame captured by the camera at a second depth of field different from the first depth of field.

[0009] The method further comprises determining whether the protective cover is partially covered by the foreign object. For example, determining whether the protective cover is partially covered by the foreign object may comprise detecting that the protective cover is partially covered by the foreign object.

[0010] Determining whether the protective cover is partially covered by the foreign object is performed by analyzing whether the first and second captured image frames are affected by the presence of the foreign object on the protective cover such that a difference between the first depth of field and the second depth of field results in a difference in luminance patterns of corresponding pixels of the first image frame and the second image frame. The first image frame is based on the first captured image frame and the second image frame is based on the second captured image frame.

[0011] In order to perform the analysis, at least one of the captured image frames may be geometrically transformed and compensated for differences in exposure values.

[0012] According to another aspect, the object is achieved by an image processing device configured to perform the above method.

[0013] According to a further aspect, the object is achieved by a computer program and a computer program carrier corresponding to the above aspects.

[0014] By analyzing the difference in the brightness pattern of corresponding pixels of the first image frame and the second image frame captured with a difference in depth of field, the determination of whether the protective cover is partially covered by the foreign object is improved, because the image used for the determination will be captured anyway, so that no additional image or image stream is required. Therefore, embodiments herein provide the possibility to detect foreign objects on the protective cover without interfering with the normal operation of the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In the drawings, features that appear in some embodiments are indicated by dashed lines.

[0016] Various aspects of the embodiments disclosed herein, including their specific features and advantages, will be readily understood from the following detailed description and accompanying drawings, in which:

[0017] Figure 1 An exemplary embodiment of an image capture device and a video network system is shown.

[0018] Figure 2a is an overview showing an exemplary embodiment of an image capture device,

[0019] Figure 2b is an overview showing an exemplary embodiment of an image capture device,

[0020] Figure 3is an overview showing an exemplary embodiment of an image capture device,

[0021] Figure 4a is a flow chart showing an embodiment of a method in an image processing device,

[0022] Figure 4b An exemplary embodiment of the present invention is shown.

[0023] Figure 4c A further exemplary embodiment of the present invention is shown.

[0024] Figure 5 A further exemplary embodiment of the present invention is shown.

[0025] Figure 6 is a block diagram illustrating an embodiment of an image processing apparatus. DETAILED DESCRIPTION

[0026] As mentioned above, there may be a problem of detecting whether there is dust or an object on the protective cover of a camera that needs cleaning. Embodiments herein are directed to detecting an object on the protective cover.

[0027] The embodiments of the present invention may be implemented in one or more image processing devices. The one or more image processing devices may be, for example, image capture devices, such as cameras, or video servers, or a combination of these. The cameras may be digital cameras. Figure 1 An image processing device in the form of a camera 120 is depicted, such as a surveillance camera or monitoring camera, such as a pan tilt and zoom (PTZ) camera. Thus, in some embodiments herein, the image processing device is the camera 120.

[0028] Figure 1 Further depicted is an exemplary video network system 150 in which embodiments herein may be implemented. The video network system 150 may include a camera 120 that may capture digital video images 101 and perform image processing thereon. Figure 1 The video server 160 in the embodiment can obtain the digital video image 101 from the camera 120, for example, through a network or the like. Figure 1 Indicated by double-pointing arrows.

[0029] A video server is a computer-based device that is dedicated to transmitting video. Video servers are used in many applications and often have additional features and capabilities that meet the needs of specific applications. For example, video servers used for security, monitoring, and inspection applications are often designed to capture video from one or more cameras and transmit the video over a computer network. In video production and broadcast applications, a video server may have the ability to record and play recorded video and transmit many video streams simultaneously. Today, many video server functions can be built into the camera 120.

[0030] Therefore, in Figure 1 In the embodiment, the video server 160 is connected to the camera 120 via the video network system 150. The video server 160 may also be connected to a video storage 170 for storing video images, and / or to a monitor 180 for displaying video images. In some embodiments, the camera 120 is directly connected to the video storage 170 and / or the monitor 180, such as Figure 1 In some other embodiments, the camera 120 is connected to the video storage 170 and / or the monitor 180 through the video server 160, as shown by the arrows between the video server 160 and other devices.

[0031] Figure 2a is an overview showing an exemplary embodiment of the camera 120 .

[0032] The camera 120 includes an image sensor 201, and a lens-based optical imaging system 202 of the camera 120 images the scene onto the image sensor 201. The lens-based optical imaging system 202 may also be referred to as a lens system. In some embodiments, the lens-based optical imaging system 202 is a zoom system. That is, the focal length of the optical imaging system 202 may be adjustable. Zooming may be used to more easily use captured images for different purposes. For example, a zoomed-out image may be used to find and / or track an object or person, while a zoomed-in image may be used to display details and identify an object or person.

[0033] The optical imaging system 202 of the camera 120 may include an adjustable aperture. The adjustable aperture may control the light intensity at the image sensor 201 of the camera 120.

[0034] The camera 120 may be arranged on a motorized pan and tilt arrangement 205. The motorized pan and tilt arrangement 205 may adjust the orientation of the camera 120 by adjusting the pan and tilt angles so that different scenes may be captured. Figure 2a In , the camera 120 is arranged at an angle alpha relative to the axis of the protective housing. Figure 2b In FIG. 1 , the camera 120 is arranged at an angle beta relative to the axis of the protective housing. Adjustment of the orientation of the camera 120 can be used to scan different sub-areas of a larger area and track objects or people.

[0035] The camera 120 is arranged in a protective housing. The protective housing includes a transparent protective cover 210. The protective housing covers the camera 120 so that no part of the camera 120 is arranged outside the protective housing. Specifically, the entire optical imaging system 202 is covered by the transparent protective cover 210.

[0036] Figure 2a and 2b A field of view 220 of optical imaging system 202 is further shown.

[0037] In order to better understand the embodiments of this document, a digital video image processing system will first be described.

[0038] Figure 3 is an overview of a digital video image processing system 300, in this case, a digital camera, such as camera 120. The digital video image processing system 300 images a scene onto an image sensor 301 with a lens-based optical imaging system 202. In other words, the digital video image processing system 300 captures a scene with the image sensor 301. The image sensor 301 may be equipped with a Bayer filter so that different pixels will receive radiation of a specific wavelength range in a known pattern. Typically, each pixel of a captured image is represented by one or more values ​​representing the intensity of the captured light within a specific wavelength band. These values ​​are typically referred to as color components or color channels. The term "image" may refer to an image frame or video frame that includes information originating from an image sensor that has captured an image.

[0039] Camera 120 may apply different gains to different exposures. For example, a first exposure having a first light intensity per region at image sensor 301 may have a different gain than a second exposure having a second light intensity per region at image sensor 301. By changing the gain, the apparent brightness of the second exposure may be adjusted to match the brightness of the first exposure even if image sensor 301 receives different light intensities per region.

[0040] In general, gain can be described as a means of increasing the ISO and apparent sensitivity to light of the camera 120. In more technical terms, it can be said that gain in a digital imaging device represents the relationship between the number of electrons acquired on the image sensor 301 and the analog-to-digital units (ADU) generated that represent the image signal. Increasing gain amplifies the signal by increasing the ratio of ADU to electrons acquired on the image sensor 301. The result is that increasing gain increases the apparent brightness of the image at a given exposure.

[0041] After reading the signals of the respective sensor pixels of the image sensor 301, the image processing pipeline 302 may perform different image processing actions. The image processing pipeline 302 may include an image processing part 302a and a video post-processing part 302b.

[0042] Typically for video processing, images are included in an image stream. Figure 3 A first video stream 310 is shown from the image sensor 301. The first image stream 310 may include a plurality of captured image frames, such as a first captured image frame 311 and a second captured image frame 312.

[0043] Image processing may include demosaicing, color correction, noise filtering (for removing spatial and / or temporal noise), distortion correction (for removing the effects of, for example, barrel distortion), global and / or local tone mapping (e.g., enabling imaging of scenes containing a wide range of intensities), transformations (e.g., rectification and rotation), flat-field correction (e.g., for removing the effects of vignetting), application of overlays (e.g., privacy masks, explanatory text), etc. The image processing pipeline 302 may also be associated with an analytics engine that performs object detection, recognition, alarms, etc.

[0044] The image processing part 302a may for example perform image stabilization, apply noise filtering, distortion correction, global and / or local tone mapping, transforms and flat field correction. The video post-processing part 302b may for example crop parts of the image, apply overlays, and include an analysis engine.

[0045] After the image processing pipeline 302, the image can be forwarded to the encoder 303, where the information in the image frame is encoded according to a coding protocol such as H.264, and forwarded to a receiving client such as the monitor 180 illustrated in this document, to the video server 160, the storage 170, etc.

[0046] Reference will now be made to the flowchart of FIG. 4 and further to Figure 1 , 2a -2b and 3 to describe an exemplary method according to an embodiment of the present invention.

[0047] 4 is a flowchart illustrating a method performed by an image processing device such as the camera 120 and / or the video server 160 for determining whether the transparent protective cover 210 of the camera 120 including the lens-based optical imaging system 202 is partially covered by a foreign object 215 on the protective cover 210 .

[0048] In other words, these methods are used to detect the foreign object 215 attached to the transparent protective cover 210 of the camera 120. The foreign object 215 may be smaller than the protective cover 210.

[0049] These methods can be used in image processing devices, especially Figure 1 160. However, these methods may also be implemented in the video server 160. The embodiments herein may also be implemented in several image processing devices. For example, in some embodiments herein, the method for detecting an object on the protective cover of the camera 120 is performed in a distributed manner in the camera 120 and one or more additional image processing devices (e.g., the video server 160). One or more additional image processing devices may also be included in a video management system (VMS) or a computer cloud or in a VMS implemented in a computer cloud. More specifically, certain parts of the method, such as controlling different image capture parameters and capturing image frames, may be performed by the camera 120, while other parts, such as analyzing the image, may be performed by one or more additional image processing devices.

[0050] One or more of the following actions presented in Figure 4 may be performed in the following exemplary order. In other examples, the order may be different from that described below.

[0051] Embodiments herein include analyzing a pair of image frames captured at different depths of field from camera 120 and determining whether transparent protective cover 210 of camera 120 is partially covered by foreign object 215 on protective cover 210. In other words, the analysis may result in detection of a foreign object.

[0052] Embodiments herein utilize the fact that changing the depth of field affects how the object 215 is imaged on the image sensor 201, 301. A larger depth of field will sharpen the image of the object 215 compared to when the object 215 is imaged with a shallower depth of field, which will change the distribution of light intensity striking the image sensor 201, 301 at the pixels corresponding to the image of the object 215. The light striking the image sensor 201, 301 triggers a pixel intensity signal that can be stored as a captured image frame 311, 312. For example, the pixel intensity signal can correspond to a linear light quantity, such as a linear RGB component. Such a linear light quantity is directly proportional to physical radiance. Another linear light quantity is relative brightness, sometimes labeled Y. It is proportional to physical radiance weighted by the spectral sensitivity of human vision. Brightness refers to light with a wavelength in the range of about 400nm to 700nm. According to the principles and standards of the International Commission on Illumination (CIE), brightness can also be calculated as a properly weighted sum of the linear light red, green and blue color components.

[0053] Using a linear component may be an advantage because the pixel intensity signal may need to be adjusted, for example by multiplication, in order to correct for differences in exposure values.

[0054] For an RGB color space using ITU-R BT.709 primaries (or sRGB which defines the same primaries), relative brightness can be calculated from the linear RGB components by first converting the gamma-compressed RGB values ​​to linear RGB and then by the following formula:

[0055] Y=0.2126R+0.7152G+0.0722B.

[0056] In embodiments herein, the expression brightness is used to mark the pixel intensity signal of the corresponding image frame. Specifically, the brightness can correspond to the relative brightness of the pixels of the image frame. In addition, in embodiments herein, the brightness pattern marks the pattern of the pixel intensity signal, such as the pattern of the relative brightness of the corresponding pixels of the corresponding image frame. The brightness pattern can also correspond to the pattern of the gradient or derivative of the relative brightness of the pixels.

[0057] Thus, the reduction in brightness due to the object 215 can be concentrated to fewer pixels with an increased depth of field. For example, the first captured image frame 311 can be captured at a first depth of field that is lower than the second depth of field of the second captured image frame 312. The brightness of the first captured image frame 311 may be affected by the object 215 in, for example, 50x50 pixels. For example, the brightness may be reduced by a few percentage points compared to the surrounding pixels. The brightness of the second captured image frame 312 may be affected by the object 215 in 30x30 pixels, and the brightness change compared to the surrounding pixels may be lower than for the first captured image frame 311. Therefore, by comparing the brightness patterns of the two captured image frames 311, 312, the object 215 can be detected. However, since the change in depth of field may change the overall brightness on the image sensor 201, 301, it may be important to compensate for this overall brightness change caused by the change in depth of field. For example, this may be the case if the depth of field is changed by an adjustable aperture.

[0058] In some embodiments herein, the depth of field is changed by adjusting the focal length of the lens-based optical imaging system 202. Then, before the images can be compared, a geometric transformation may need to be performed on at least one of the captured images.

[0059] When detection of a foreign object 215 occurs, the image processing device 120 , 160 , which may instruct the camera 120 and / or the video server 160 , may trigger a warning indicating that the foreign object 215 is attached to the protective cover 210 , which in turn may trigger appropriate cleaning of the protective cover 210 .

[0060] The method allows detection of the foreign object 215 on the protective cover 210 without interfering with the normal image frame stream 310, because the method obtains information from the image frames in the normal image stream. Specifically, if the focal length of the lens-based optical imaging system 202 is adjustable, such as a zoom lens, a change in depth of field will naturally occur when changing the focal length. Therefore, there is no need to capture an image specifically for detecting a foreign object on the protective cover 210.

[0061] Action 401

[0062] In some embodiments herein, the camera 120 may control the optical imaging system 202 to obtain the first depth of field. Therefore, the image processing device 120, 160 may control the optical imaging system 202 to obtain the first depth of field.

[0063] The first depth of field may be obtained, for example, by controlling the focal length of the lens-based optical imaging system 202. Therefore, the first depth of field may correspond to a first focal length or a first zoom position of the lens-based optical imaging system 202. In the case of the same focus distance, a large focal length corresponds to a shallow depth of field compared to a small focal length.

[0064] In some other embodiments herein, the first depth of field is obtained by controlling the adjustable aperture by the camera 120. Therefore, the first depth of field can correspond to the first aperture opening of the lens-based optical imaging system 202. Therefore, in some embodiments in which the optical imaging system 202 of the camera 120 includes an adjustable aperture, the first depth of field is obtained by the first aperture opening of the optical imaging system 202. For example, the first aperture opening can be a full aperture opening, i.e., corresponding to a maximum aperture opening, such as an f-number of 2.8. The full aperture opening can correspond to a shallow depth of field. In another example, the first aperture opening corresponds to a larger depth of field. For example, an f-number of 11 may correspond to a larger depth of field. A large aperture opening corresponds to a shallow depth of field. However, due to diffraction resulting in reduced sharpness at high f-numbers, a larger depth of field can be obtained at an f-number that is also selected based on the requirement for sharpness.

[0065] In some embodiments, the camera 120 controls both the focal length and the aperture opening to obtain a first depth of field. Thus, the first depth of field may correspond to a first setting of the focal length and the aperture opening.

[0066] Action 402

[0067] The image processing device 120 , 160 acquires a first captured image frame 311 captured by the camera 120 at a first depth of field.

[0068] For example, the camera 120 may acquire it by capturing a first captured image frame. The video server 160 may acquire it by receiving the first captured image frame from the camera 120.

[0069] Therefore, in some embodiments herein, the focal length of the optical imaging system 202 is adjustable, and the first captured image frame 311 is captured using a first focal length of the optical imaging system 202 to obtain a first depth of field.

[0070] Since the second captured image frame 312 may be captured using a different exposure value than the first captured image frame 311, these exposure values ​​or a ratio of exposure values ​​may be saved to compensate for the difference. Thus, the first captured image frame 311 may be captured using the first exposure value. The exposure value may be defined by the exposure time, the aperture opening, and the gain of the image sensor 201, 301.

[0071] Action 403

[0072] The camera 120 may control the optical imaging system 202 to obtain a second depth of field that is different from the first depth of field.

[0073] The camera 120 may control the optical imaging system 202 to obtain the second depth of field by controlling the focal length and / or the aperture opening of the optical imaging system 202 .

[0074] For example, when the first aperture opening is a full aperture opening, the second aperture opening may be a smaller aperture opening, such as a minimum aperture opening. In another example, when the first aperture opening is a maximum aperture opening, the second aperture opening may be a minimum aperture opening.

[0075] Action 404

[0076] The image processing device 120 , 160 acquires a second captured image frame 312 captured by the camera 120 at a second depth of field different from the first depth of field.

[0077] The same portion of the protective cover 210 may be at least partially captured by the first and second captured image frames 311, 312 so as to have overlapping images of the foreign object 215. In other words, the second captured image frame 312 may include an image of the same portion of the protective cover as the first captured image frame at least in part. Preferably, a static scene is captured so that the brightness pattern is not affected by dynamic objects in the scene (e.g., a car or a bumblebee).

[0078] When the depth of field is adjusted by adjusting the focal length of the optical imaging system 202 , the second captured image frame 312 is captured at a second focal length of the optical imaging system 202 to obtain a second depth of field.

[0079] In some embodiments, the second depth of field is obtained by a second aperture opening of the optical imaging system 202 .

[0080] The second captured image frame 312 may be captured using a second exposure value that may be different from the first exposure value. In other words, the first exposure value may be different from the second exposure value.

[0081] Action 405

[0082] In the embodiments of the present invention, the image processing device 120, 160 acquires additional image frames 311', 312' based on the first and second captured image frames 311, 312. The additional image frames 311', 312' may also be referred to as adjusted or transformed image frames 311', 312'. The additional image frames 311', 312' are used to compare the brightness patterns between two images captured at different depths of field, because the original captured image frames may be difficult to directly compare due to differences in optical parameters such as focal length and / or aperture opening (which in turn may produce differences in exposure values ​​and fields of view). In addition, the focal length and aperture opening may also affect the image frames in other ways that may require compensation, such as vignetting of the image frames. Before comparing the image frames to each other, the additional image frames 311', 312' may be vignetted.

[0083] To acquire further image frames 311', 312', copies of the first and second captured image frames 311, 312 may be created, for example, in the normal image processing pipeline 302. The further image frames 311', 312' may then be created as (local) transformations of the captured image frames 311, 312 and may be discarded once the comparison of the brightness patterns is completed.

[0084] Therefore, in some embodiments herein, some adjustments need to be made to the captured image frames 311, 312 before the image frames can be compared. Figure 4b In the example shown, the first image frame 311' is generated as a copy of the first captured image frame 311, and the second image frame 312' is generated as a copy of the second captured image frame 312. As described above, the difference in exposure values ​​can be compensated by adjusting the brightness value of the first image frame 311' relative to the brightness value of the first captured image frame 311. Each brightness value is associated with a corresponding pixel of the image frame. In an alternative embodiment, the difference in exposure values ​​is compensated by adjusting the brightness value of the second image frame 312' relative to the brightness value of the second captured image frame 312. The brightness values ​​of both the first image frame 311' and the second image frame 312' can also be adjusted.

[0085] For example, the ratio of the second exposure value to the first exposure value may be used to adjust the brightness value of one of the first or second image frames 311', 312'. If a linear pixel intensity signal, such as relative brightness, is used, the same value, such as the ratio of the second exposure value to the first exposure value, may be used to adjust all pixel intensity values.

[0086] Compensation for the difference in exposure values ​​may be performed after compensating for vignetting.

[0087] In some embodiments where the focal length of the optical imaging system 202 is adjustable, before the brightness patterns can be compared, it may be necessary to geometrically transform at least one of the captured image frames 311, 312 in order to compensate for differences in the field of view due to the different focal lengths.

[0088] Therefore, in some embodiments of the present invention, either the first image frame 311' is obtained by geometrically transforming the first captured image frame 311, or the second image frame 312' is obtained by geometrically transforming the second captured image frame 312. In other words, at least one of the first image frame 311' and the second image frame 312' can be obtained by geometrically transforming the corresponding first and second captured image frames 311, 312.

[0089] For example, if the captured image frames 311 , 312 have different degrees of optical aberrations, such as barrel distortion, it may be necessary to compensate for these aberrations by geometrically transforming one or both of the captured image frames 311 , 312 .

[0090] To compensate for the different focal lengths, scaling of the captured image frames 311, 312 may be performed. Scaling may be performed in conjunction with cropping the captured image frame having the shortest focal length (i.e., the zoomed out image frame). For example, a region of the zoomed out image frame may be cropped. The remaining portion of the zoomed out image frame may then be upscaled to the same size as the zoomed in image frame. Alternatively, the zoomed in image frame may be down-scaled to the size of the cropped zoomed out image frame.

[0091] Action 406

[0092] The image processing device 120, 160 determines whether the protective cover 210 is partially covered by the foreign object 215 by analyzing whether the first and second captured image frames 311, 312 are affected by the presence of the foreign object 215 on the protective cover 210 so that the difference between the first depth of field and the second depth of field results in a difference in the brightness patterns of the corresponding pixels 411', 421' of the first image frame 311' and the second image frame 312'. The first image frame 311' is based on the first captured image frame 311 and the second image frame 312' is based on the second captured image frame 312.

[0093] Figure 4b A first corresponding pixel 411' of the first image frame 311' and a first corresponding pixel 421' of the second image frame 312' are shown. There may be a plurality of corresponding pixels in the respective first and second image frames 311', 312'.

[0094] How to obtain the first image frame 311' from the first captured image frame 311 and the second image frame 312' from the second captured image frame 312 is described above with respect to action 405. For example, before action 406, at least one of the captured image frames 311, 312 may be geometrically transformed and the difference in exposure values ​​may be compensated.

[0095] Thus, the action may include comparing the brightness of the pixels of the two derived image frames 311', 312'. Determining whether the protective cover 210 is partially covered by the foreign object 215 may include detecting the foreign object at the protective cover 210. Thus, the image processing device 120, 160 may detect that the foreign object is adhered to the protective cover 210 by detecting a difference in brightness pattern between one or more pixels of the first image frame 311' based on the first captured image frame 311 and corresponding one or more pixels of the second image frame 312' based on the second captured image frame 312.

[0096] As described above, when describing action 405, the first image frame 311' can be calculated from the first captured image frame 311. Similarly, the second image frame 312' can be calculated from the second captured image frame 312. Such calculations can, for example, include adjusting brightness values. The brightness values ​​can be adjusted based on the ratio of the second exposure value to the first exposure value. The calculations can also include performing geometric transformations on the captured images 311, 312.

[0097] After adjusting at least one of the captured image frames due to differences in exposure and / or focal length, a clean protective cover 210 should result in a difference in brightness of the corresponding pixel of zero. Therefore, if the difference in brightness pattern is above a threshold, it can be determined that the protective cover 210 is partially covered by a foreign object 215. In other words, if the difference in brightness pattern is above a threshold, a foreign object can be detected. The threshold can be predeterminable. Specifically, if a larger depth of field results in a lower brightness of the brightness pattern compared to a shallower depth of field, the image processing device 120, 160 can determine that the protective cover 210 is partially covered by a foreign object 215. The brightness of the brightness pattern can be, for example, the total brightness or average brightness of the brightness pattern of the corresponding pixel. A combination of total brightness and average brightness can also be used for different corresponding pixels. In some other embodiments, the brightness of the brightness pattern can correspond to the corresponding brightness value of the corresponding corresponding pixel. For example, in some embodiments, each corresponding pixel brightness value may need to be distinguished by a threshold value in order to determine that the protective cover 210 is covered by a foreign object 215.

[0098] In other words, if a larger depth of field results in a lower brightness of the corresponding pixel 411', 421' compared to a shallower depth of field, the foreign object 215 can be detected. For example, if the brightness of an image frame based on a captured image frame with a larger depth of field is lower than the brightness of an image frame based on a captured image frame with a shallower depth of field by a threshold difference, the image processing device 120, 160 can determine that the protective cover 210 is partially covered by the foreign object 215. This Figure 4c , where the brightness of some corresponding pixels of the first image frame 311' is compared with the brightness of some corresponding pixels of the second image frame 312'. Figure 4cIn the embodiment of the present invention, the first image frame 311' is based on a shallower depth of field than the second image frame 312'. That is, the first captured image frame 311 is captured with a shallower depth of field than the second captured image frame 312.

[0099] In some embodiments herein, a difference in brightness patterns of corresponding pixels refers to a threshold number of pixels whose brightness difference is greater than a predeterminable threshold. For example, if a low depth of field image frame includes ten corresponding pixels whose brightness is greater than the brightness of ten corresponding pixels of a large depth of field image frame, this may correspond to a difference in brightness patterns. However, if there are only five corresponding pixels whose brightness is significantly different, the method may determine that the difference in brightness patterns is below the threshold and is therefore not significant for detecting foreign object 215.

[0100] In some embodiments, the difference in brightness patterns is calculated based on a derivative of the pixel intensity signal, such as relative brightness. For example, the image processing device 120, 160 can calculate the derivative of the relative brightness of the corresponding image frames. Regions or blocks of pixels can be compared by summing the absolute values ​​of the gradients of the brightness of the pixels and then comparing the sums between different image frames.

[0101] In some embodiments where the first exposure value is different from the second exposure value, the brightness of the pixels of the first image frame 311' is obtained based on the brightness of the corresponding pixels of the first captured image frame 311 and further based on the ratio of the second exposure value to the first exposure value, and / or the brightness of the pixels of the second image frame 312' is obtained based on the brightness of the corresponding pixels of the second captured image frame 312 and further based on the ratio of the second exposure value to the first exposure value.

[0102] Therefore, in order to detect the difference in brightness patterns, the image processing device 120, 160 may consider the ratio of the second exposure value to the first exposure value.

[0103] By analyzing differences in brightness patterns of corresponding pixels of a first image frame and a second image frame captured at different depths of field, the determination of whether the protective cover is partially covered by a foreign object is improved because the images used for the determination would have been captured anyway, so no additional images or image streams are required.

[0104] Different portions of the protective cover 210 will be imaged on different portions of the image sensors 201, 301. Therefore, by analyzing the differences in the brightness patterns of different portions of the image frames corresponding to different portions of the image sensors 201, 301, different portions of the protective cover imaged on the image sensors 201, 301 can be scanned to detect foreign objects. The first pixel 411 of the first captured image frame 311 can be close to the optical axis. The first pixel 411 can then correspond to a portion of the protective cover 210 that is close to the middle of the lens-based optical imaging system 202. Pixels further away on the image sensors 201, 301, such as the second pixel 412 or the third pixel 413, correspond to portions of the protective cover that are far from the middle of the lens-based optical imaging system 202.

[0105] Action 407

[0106] The camera 120 can adjust the direction in which the camera 120 captures the image frames 311, 312. The direction can be adjusted, for example, by a motorized pan and tilt arrangement 205. Therefore, in some embodiments herein, the direction in which the camera 120 captures the image frames 311, 312 can be adjusted by a motorized pan and tilt arrangement 205. By adjusting the direction of the camera 120, the above actions 401-406 can be repeated at multiple orientations of the camera 120. Multiple areas of the transparent protective cover 210 can be analyzed and detected by adjusting the direction. For example, some parts of the transparent protective cover 210 may extend outside the field of view 220 of the camera 120. Then corresponding image frame pairs at multiple camera orientations can also be captured, each camera orientation being defined by a pan angle and a tilt angle, and the results of the detection analysis from different camera orientations can be compared. Other parameters describing the camera direction can also be used. Such parameters can be given, for example, by an accelerometer and a compass arranged in or at the camera 120.

[0107] To be able to compare results from different camera orientations, the captured image frames at least partially capture the same portion of the protective cover 210. For example, to compare detection results of the object 215 on the protective cover 210, the image frames used for comparison capture images of the object 215.

[0108] Such a comparison may confirm the first result or trigger further analysis. For example, a first analysis at a first camera orientation may indicate that there is an object 215 on the protective cover 210. Then a second analysis at a second camera orientation also indicates that there is an object 215 on the protective cover 210. The two analyses then confirm each other, and appropriate actions may be taken accordingly. An example of such an action is given in action 408 below. Figure 5It is shown how another pair of captured images 321 , 321 is converted into another pair of images 321 ′, 322 ′, which can be used to detect objects and compared with the first pair of images 311 ′, 312 ′.

[0109] Thus, the first and second captured image frames 311, 312 may be captured at a first orientation of the camera 120. An additional first captured image frame 321 and an additional second captured image frame 322 may be captured at a second orientation of the camera 120.

[0110] An additional first captured image frame 321 is captured with an additional first depth of field, and an additional second captured image frame 322 is captured with an additional second depth of field that is different from the additional first depth of field.

[0111] The first orientation corresponds to a first value of the translation and tilt of the pan and tilt arrangement 205, and the second orientation corresponds to a second value of the translation and tilt of the pan and tilt arrangement 205 that is at least partially different from the first value. The image frames captured at the first and second orientations at least partially capture the same portion of the protective cover 210.

[0112] In some embodiments herein, determining whether protective cover 210 is partially covered by foreign object 215 is based on evaluating determinations at both the first and second orientations. In other words, detecting foreign object 215 at protective cover 210 can be based on evaluating detections at both the first and second orientations.

[0113] For example, the image processing devices 120, 160 can determine whether the protective cover 210 is partially covered by the foreign object 215 based on analyzing whether the first and second captured image frames 311, 312 are affected by the presence of the foreign object 215 so that the difference between the first depth of field and the second depth of field causes a difference in the brightness patterns of corresponding pixels of the first image frame 311' and the second image frame 312', and further based on whether the additional first and additional second captured image frames 321, 322 are affected by the presence of the foreign object 215 so that the difference between the additional first depth of field and the additional second depth of field causes a difference in the brightness patterns of corresponding pixels of the additional first image frame 321' and the additional second image frame 322'.

[0114] Action 408

[0115] In some embodiments, the image processing device 120, 160 triggers a warning indication in response to determining that the protective cover 210 is partially covered by the foreign object 215. In other words, the image processing device 120, 160 may trigger a warning indication in response to detecting the foreign object 215.

[0116] In some embodiments herein, the warning indication is sent to the VMS or is sent to the operator as an email. In some other embodiments herein, the warning indication is stored in the memory of the camera 120. Then, the VMS or the operator can check the warning, for example, once a day.

[0117] refer to Figure 6 , shows a schematic block diagram of an embodiment of an image processing device 120 , 160 . The image processing device 120 , 160 is configured to determine whether a transparent protective cover 210 of the camera 120 is partially covered by a foreign object 215 .

[0118] As described above, in some embodiments herein, the image processing device 120, 160 is a camera 120. The camera 120 may be a PTZ camera. Therefore, the image processing device 120, 160 may be a PTZ camera.

[0119] The image processing device 120, 160 may include a processing module 601, such as a device for executing the method described herein. The device may be embodied in the form of one or more hardware modules and / or one or more software modules.

[0120] The image processing device 120, 160 may further include a memory 602. The memory may include, for example, contain or store instructions, for example in the form of a computer program 603, which may include computer-readable code units that, when executed on the image processing device 120, 160, cause the image processing device 120, 160 to perform a method for determining whether the transparent protective cover 210 of the camera 120 is partially covered by the foreign object 215. For example, the computer program 603 may be executed on one or more processors (e.g., processing circuit 604) of the image processing device 120, 160 to cause the one or more processors to perform a method for determining whether the transparent protective cover 210 of the camera 120 is partially covered by the foreign object 215.

[0121] According to some embodiments of the present invention, the image processing device 120, 160 and / or the processing module 601 includes a processing circuit 604 as an exemplary hardware module, which may include one or more processors. Therefore, the processing module 601 may be embodied in the form of or "implemented by" the processing circuit 604. The instructions may be executed by the processing circuit 604, whereby the image processing device 120, 160 may be operated to perform Figure 4a As another example, the instructions, when executed by the image processing device 120, 160 and / or the processing circuit 604, may cause the image processing device 120, 160 to perform a method according to Figure 4a method.

[0122] In view of the above, in one example, an image processing device 120, 160 is provided for determining whether the transparent protective cover 210 of the camera 120 is partially covered by the foreign object 215. Again, the memory 602 contains instructions executable by the processing circuit 604, so that the image processing device 120, 160 can be operated to perform according to Figure 4a Method:

[0123] The image processing devices 120, 160 may also be operable to perform the above combination Figure 4a The method of the detailed embodiment is described.

[0124] Figure 6 Further shown is a carrier 605 or program carrier, which includes the above-mentioned computer program 603. The carrier 605 may be one of an electronic signal, an optical signal, a radio signal and a computer-readable medium.

[0125] In some embodiments, the image processing device 120, 160 and / or the processing module 601 may include an acquisition module 610, a determination module 620, a pan-tilt adjustment module 630, an optical control module 640, and a warning trigger module 650 as exemplary hardware modules. In other examples, one or more of the above exemplary hardware modules may be implemented as one or more software modules.

[0126] Furthermore, the processing module 601 may include an input / output unit 606. According to an embodiment, the input / output unit 606 may include an image sensor 201, 301 configured to capture the above-mentioned image frames, such as the first captured image frame 311 and the second captured image frame 312.

[0127] Therefore, the image processing device 120 , 160 is configured to determine whether the transparent protective cover 210 of the camera 120 is partially covered by the foreign object 215 .

[0128] Therefore, according to the above-mentioned various embodiments, the image processing device 120, 160 and / or the processing module 601 and / or the acquisition module 610 are configured to acquire a first captured image frame 311 captured by the camera 120 at a first depth of field and to acquire a second captured image frame 312 captured by the camera 120 at a second depth of field different from the first depth of field.

[0129] In some embodiments, the image processing device 120 , 160 and / or the processing module 601 and / or the acquisition module 610 are configured to at least partially capture the same portion of the protective cover 210 via the first and second captured image frames 311 , 312 .

[0130] The image processing device 120, 160 and / or the processing module 601 and / or the determination module 620 are configured to determine whether the protective cover 210 is partially covered by the foreign object 215 by analyzing whether the first and second captured image frames 311, 312 are affected by the presence of the foreign object 215 on the protective cover 210 so that the difference between the first depth of field and the second depth of field results in a difference in the brightness patterns of the corresponding pixels 411', 421' of the first image frame 311' and the second image frame 312'. The first image frame 311' is based on the first captured image frame 311 and the second image frame 312' is based on the second captured image frame 312.

[0131] In some embodiments herein, the focal length of the optical imaging system 202 is adjustable. Then the image processing device 120, 160 and / or the processing module 601 and / or the acquisition module 610 may be configured to capture the first captured image frame 311 at the first focal length of the optical imaging system 202 to obtain a first depth of field and to capture the second captured image frame 312 at the second focal length of the optical imaging system 202 to obtain a second depth of field.

[0132] When the focal length of the optical imaging system 202 is adjustable and the second focal length is different from the first focal length, the image processing device 120, 160 and / or the processing module 601 and / or the acquisition module 610 may be configured to either acquire a first image frame 311' by performing a geometric transformation on the first captured image frame 311, or to acquire a second image frame 312' by performing a geometric transformation on the second captured image frame 312.

[0133] In some embodiments, the optical imaging system 202 of the camera 120 includes an adjustable aperture. Then the image processing device 120, 160 and / or the processing module 601 and / or the acquisition module 610 may be configured to acquire a first depth of field through a first aperture opening of the optical imaging system 202, and acquire a second depth of field through a second aperture opening of the optical imaging system 202.

[0134] The image processing devices 120 , 160 and / or the processing module 601 and / or the determination module 620 may be configured to determine that the protective cover 210 is partially covered by the foreign object 215 , ie, the foreign object is detected, if the difference in the brightness pattern is above a threshold.

[0135] The image processing device 120 , 160 and / or the processing module 601 and / or the determining module 620 may be configured to determine that the protective cover 210 is partially covered by the foreign object 215 if a greater depth of field results in a lower brightness of the brightness pattern than a shallower depth of field.

[0136] In some embodiments, where the first captured image frame 311 is captured using a first exposure value, and the second captured image frame 312 is captured using a second exposure value different from the first exposure value, the image processing device 120, 160 and / or the processing module 601 and / or the acquisition module 610 may then be configured to acquire the brightness of the pixels of the first image frame 311' based on the brightness of the corresponding pixels of the first captured image frame 311 and further based on the ratio of the second exposure value to the first exposure value, and / or acquire the brightness of the pixels of the second image frame 312' based on the brightness of the corresponding pixels of the second captured image frame 312 and further based on the ratio of the second exposure value to the first exposure value.

[0137] In some embodiments, the additional first captured image frame 321 and the additional second captured image frame 322 are captured at the second orientation of the camera 120, and the additional first captured image frame 321 is captured using the additional first depth of field, and the additional second depth of field captured image frame 3222 is captured using the additional second depth of field that is different from the additional first depth of field. The image processing device 120, 160 and / or the processing module 601 and / or the determination module 620 are then configured to determine whether the protective cover 210 is partially covered by the foreign object 215 based further on whether the additional first and additional second captured image frames 321, 322 are affected by the presence of the foreign object 215 so that the difference between the additional first depth of field and the additional second depth of field results in a difference in the brightness patterns of corresponding pixels of the additional first image frame 321' and the additional second image frame 322'.

[0138] The image processing device 120 , 160 and / or the processing module 601 and / or the pan-tilt adjustment module 640 may be configured to adjust the motorized pan and tilt arrangement to adjust the direction in which the camera 120 captures the image frames 311 , 312 .

[0139] The image processing device 120 , 160 and / or the processing module 601 and / or the warning triggering module 650 may be configured to trigger a warning indication in response to determining that the protective cover 210 is partially covered by the foreign object 215 .

[0140] The image processing devices 120 , 160 and / or the processing module 601 and / or the optical control module 540 may be configured to control the optical imaging system 202 to acquire a first depth of field, and control the optical imaging system 202 to acquire a second depth of field different from the first depth of field.

[0141] As used herein, the term "module" may refer to one or more functional modules, each of which may be implemented as one or more hardware modules and / or one or more software modules and / or a combined software / hardware module. In some examples, a module may represent a functional unit implemented as software and / or hardware.

[0142] As used herein, the term "computer program carrier", "program carrier" or "carrier" may refer to one of an electronic signal, an optical signal, a radio signal, and a computer-readable medium. In some examples, the computer program carrier may exclude transient, propagating signals such as electronic, optical and / or radio signals. Thus, in these examples, the computer program carrier may be a non-transitory carrier, such as a non-transitory computer-readable medium.

[0143] As used herein, the term "processing module" may include one or more hardware modules, one or more software modules, or a combination thereof. Any such module, whether hardware, software, or a combined hardware-software module, may be an acquisition device, a determination device, a pan-tilt adjustment device, an optical control device, a warning trigger device, etc. disclosed herein. As an example, the expression "device" may be a module corresponding to the modules listed above in conjunction with the accompanying drawings.

[0144] As used herein, the term "software module" may refer to a software application, a dynamic link library (DLL), a software component, a software object, an object according to the Component Object Model (COM), a software component, a software function, a software engine, an executable binary software file, etc.

[0145] The term "processing module" or "processing circuit" herein may encompass a processing unit including, for example, one or more processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The processing circuit, etc. may include one or more processor cores.

[0146] As used herein, the expression “configured to / for” may mean that the processing circuit is configured to, for example, be suitable for or operable to perform one or more of the actions described herein through software configuration and / or hardware configuration.

[0147] As used herein, the term "action" may refer to an action, step, operation, response, reaction, activity, etc. It should be noted that an action herein may be split into two or more sub-actions when applicable. In addition, it should also be noted that, where applicable, two or more actions described herein may be combined into a single action.

[0148] As used herein, the term "memory" may refer to a hard disk, a magnetic storage medium, a portable computer floppy disk or disk, flash memory, random access memory (RAM), etc. In addition, the term "memory" may refer to an internal register memory of a processor or the like.

[0149] As used herein, the term "computer-readable medium" may be a Universal Serial Bus (USB) memory, a DVD disk, a Blu-ray disk, a software module received as a data stream, a flash memory, a hard drive, a memory card such as a memory stick, a MultiMediaCard (MMC), a Secure Digital (SD) card, etc. One or more of the above examples of computer-readable media may be provided as one or more computer program products.

[0150] As used herein, the term "computer readable code unit" may be the text of a computer program, a binary file representing part or all of a computer program in a compiled format, or anything in between.

[0151] As used herein, the terms "number" and / or "value" may be any kind of number, such as binary, real, imaginary or rational numbers, etc. In addition, a "number" and / or "value" may be one or more characters, such as letters or strings of letters. A "number" and / or "value" may also be represented by a bit string, i.e., a plurality of zeros and / or a plurality of ones.

[0152] As used herein, the expression "in some embodiments" is used to indicate that features of a described embodiment can be combined with any other embodiments disclosed herein.

[0153] Although various embodiments have been described, many different changes, modifications, etc. will become apparent to those skilled in the art. Therefore, the described embodiments are not intended to limit the scope of the present disclosure.

Claims

1. A method for determining whether a transparent protective cover of a camera including a lens-based optical imaging system is partially covered by a foreign object on the protective cover, performed by an image processing device, wherein the focal length of the optical imaging system is adjustable, the method comprising: Acquire a first captured image frame captured by the camera at a first depth of field, wherein the first captured image frame is captured at a first focal length of the optical imaging system to obtain the first depth of field; Acquire a second captured image frame captured by the camera at a second depth of field different from the first depth of field, wherein the second captured image frame is captured at a second focal length of the optical imaging system to obtain the second depth of field; and determining whether the protective cover is partially covered by the foreign object by analyzing whether the first captured image frame and the second captured image frame are affected by the presence of the foreign object on the protective cover such that a difference between the first depth of field and the second depth of field results in a difference above a threshold in brightness patterns of corresponding pixels of the first image frame and the second image frame, wherein the first image frame is calculated from the first captured image frame and the second image frame is calculated from the second captured image frame; The brightness pattern corresponds to a pattern of a gradient of the brightness of the corresponding pixel, wherein the brightness of the corresponding pixel is a relative brightness of the corresponding pixel.

2. The method according to claim 1, wherein: The first image frame is obtained by performing a geometric transformation on the first captured image frame, and / or the second image frame is obtained by performing a geometric transformation on the second captured image frame.

3. The method according to claim 1, wherein: The optical imaging system of the camera comprises an adjustable aperture, and wherein the first depth of field is obtained by the first focal length and a first aperture opening of the optical imaging system, and wherein the second depth of field is obtained by the second focal length and a second aperture opening of the optical imaging system.

4. The method according to claim 1, wherein: If the difference in the brightness patterns is above a threshold, it is determined that the protection cover is partially covered by the foreign object.

5. The method according to claim 1, wherein: If a greater depth of field results in a lower brightness of the brightness pattern compared to a shallower depth of field, it is determined that the protective cover is partially covered by the foreign object.

6. The method according to claim 1, wherein: The first captured image frame is captured at a first exposure value, and the second captured image frame is captured at a second exposure value different from the first exposure value, and wherein the brightness of the pixels of the first image frame is obtained based on the brightness of the corresponding pixels of the first captured image frame and further based on the ratio of the second exposure value to the first exposure value, and / or the brightness of the pixels of the second image frame is obtained based on the brightness of the corresponding pixels of the second captured image frame and further based on the ratio of the second exposure value to the first exposure value.

7. The method according to claim 1, wherein: The direction in which the camera captures the image frames is adjustable by a motorized pan and tilt arrangement, and wherein the first captured image frame and the second captured image frame are captured in a first orientation of the camera, and wherein an additional first captured image frame and an additional second captured image frame are captured in a second orientation of the camera, wherein the additional first captured image frame is captured at an additional first depth of field and the additional second captured image frame is captured at an additional second depth of field different from the additional first depth of field, and wherein the first orientation corresponds to a first value of pan and tilt of the pan and tilt arrangement, and wherein the second orientation corresponds to a second value of pan and tilt of the pan and tilt arrangement that is at least partially different from the first value, and wherein the image frames captured in the first orientation and the second orientation at least partially capture the same portion of the protective cover, and wherein determining whether the protective cover is partially covered by the foreign object is further based on whether the additional first captured image frame and the additional second captured image frame are affected by the presence of the foreign object such that a difference between the additional first depth of field and the additional second depth of field results in a difference in the brightness patterns of corresponding pixels of the additional first image frame and the additional second image frame.

8. The method according to claim 1, further comprising: A warning indication is triggered in response to determining that the protective cover is partially covered by the foreign object.

9. The method according to claim 1, wherein: The same portion of the protective cover is at least partially captured by the first captured image frame and the second captured image frame. 10 . An image processing device configured to perform the method according to claim 1 .

11. The image processing apparatus according to claim 10, wherein: The image processing device is the camera.

12. The image processing apparatus according to claim 10, wherein: The camera is a PTZ camera.

13. A non-transitory storage device having stored therein a computer program, the computer program comprising computer readable code means which, when executed on an image processing device, cause the image processing device to perform the method according to claim 1.

14. The non-transitory storage device according to claim 13, wherein: The first image frame is obtained by performing a geometric transformation on the first captured image frame, and / or the second image frame is obtained by performing a geometric transformation on the second captured image frame.

15. The non-transitory storage device of claim 13, wherein: The optical imaging system of the camera comprises an adjustable aperture, and wherein the first depth of field is obtained by the first focal length and a first aperture opening of the optical imaging system, and wherein the second depth of field is obtained by the second focal length and a second aperture opening of the optical imaging system.

16. The non-transitory storage device of claim 13, wherein: If the difference in the brightness patterns is above a threshold, it is determined that the protection cover is partially covered by the foreign object.

17. The non-transitory storage device of claim 13, wherein: If a greater depth of field results in a lower brightness of the brightness pattern compared to a shallower depth of field, it is determined that the protective cover is partially covered by the foreign object.

18. The non-transitory storage device of claim 13, wherein: The first captured image frame is captured at a first exposure value, and the second captured image frame is captured at a second exposure value different from the first exposure value, and wherein the brightness of the pixels of the first image frame is obtained based on the brightness of the corresponding pixels of the first captured image frame and further based on the ratio of the second exposure value to the first exposure value, and / or the brightness of the pixels of the second image frame is obtained based on the brightness of the corresponding pixels of the second captured image frame and further based on the ratio of the second exposure value to the first exposure value.

19. The non-transitory storage device of claim 13, wherein: The direction in which the camera captures the image frames is adjustable by a motorized pan and tilt arrangement, and wherein the first captured image frame and the second captured image frame are captured in a first orientation of the camera, and wherein an additional first captured image frame and an additional second captured image frame are captured in a second orientation of the camera, wherein the additional first captured image frame is captured at an additional first depth of field and the additional second captured image frame is captured at an additional second depth of field different from the additional first depth of field, and wherein the first orientation corresponds to a first value of pan and tilt of the pan and tilt arrangement, and wherein the second orientation corresponds to a second value of pan and tilt of the pan and tilt arrangement that is at least partially different from the first value, and wherein the image frames captured in the first orientation and the second orientation at least partially capture the same portion of the protective cover, and wherein determining whether the protective cover is partially covered by the foreign object is further based on whether the additional first captured image frame and the additional second captured image frame are affected by the presence of the foreign object such that a difference between the additional first depth of field and the additional second depth of field results in a difference in the brightness patterns of corresponding pixels of the additional first image frame and the additional second image frame.

20. The non-transitory storage device of claim 13, wherein the method further comprises: A warning indication is triggered in response to determining that the protective cover is partially covered by the foreign object.

Images