Lens detection method and electronic device

Abstract

This disclosure provides a lens detection method and electronic device. The method includes: acquiring a plurality of image frames captured by a lens and performing an object tracking operation on the image frames; and determining the lens occlusion state based on the correlation between the at least one second image frame captured at least one shooting time and the determination that a target object is tracked in at least one first image frame and the target object is lost in at least one second image frame.

Description

Technical Field

[0001] This disclosure relates to a detection mechanism, and more particularly to a lens detection method and electronic device. Prior Technology

[0002] In existing technologies, cameras or image capturing devices typically lack a dedicated mechanism to detect foreign objects on the lens / glass surface of their lenses or image-capturing elements. Therefore, when there are stains, dust, or fingerprints on the lens, it will cause abnormalities in the captured images, such as blurriness, flare or shadows, reduced contrast, ghosting or halos, and distortion.

[0003] Furthermore, stains or fingerprints can scatter light, causing images to lose sharpness. Dust particles can create bright spots or dark patches on images. Foreign objects can also reduce color vibrancy and contrast, and even reflect light, producing halos or ghosting effects. The shape or location of foreign objects can also cause deformation or distortion in certain areas of the image, affecting image quality and detail. Summary of the Invention

[0004] In view of this, this disclosure provides a lens detection method and electronic device, which can be used to solve the above-mentioned technical problems.

[0005] This disclosed embodiment provides a lens detection method, including: acquiring a plurality of image frames captured by a lens by an electronic device, and performing an object tracking operation on the image frames; and in response to determining that a target object is tracked in at least one first image frame among the image frames, and that the target object is lost in at least one second image frame among the image frames, the electronic device determines a occlusion state of the lens based on the correlation of the at least one second image frame captured at at least one shooting time.

[0006] This disclosure provides an electronic device including a storage circuit and a processor. The storage circuit stores code. The processor is coupled to the storage circuit and accesses the code to execute: acquiring a plurality of image frames captured by a lens and performing an object tracking operation on the image frames; in response to determining that a target object is tracked in at least one first image frame of the image frames, and that the target object is lost in at least one second image frame of the image frames, determining a lens occlusion state based on a correlation of the at least one second image frame captured at at least one shooting time. Simple Explanation of the Diagram

[0007] Figure 1 is a schematic diagram of an electronic device and a lens according to one embodiment of the present disclosure. Figure 2 is a flowchart illustrating a lens detection method according to one embodiment of the present disclosure. Figure 3 is an application scenario diagram illustrated according to one embodiment of the present disclosure. Implementation

[0008] Please refer to Figure 1, which is a schematic diagram of an electronic device and a lens according to one embodiment of this disclosure. In different embodiments, the electronic device 100 may be implemented as various smart devices and / or computer devices, but is not limited thereto.

[0009] In Figure 1, the electronic device 100 includes a storage circuit 102 and a processor 104.

[0010] The storage circuit 102 may be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or combination of these devices, and may be used to record a plurality of code or modules.

[0011] The processor 104 is coupled to the storage circuit 102 and may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors incorporating a digital signal processor core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of integrated circuit, a state machine, a processor based on an advanced RISC machine (ARM), and the like.

[0012] Lens 110 may be, for example, a charge-coupled device (CCD) lens, a complementary metal-oxide-semiconductor (CMOS) lens, an infrared lens, a thermal imaging lens, a laser lens, a hyperspectral lens, a microlens array lens, a liquid lens, or an aspherical lens, but may not be limited thereto.

[0013] In some embodiments, the lens 110 may be mounted on a specific image capturing device (e.g., a monitor) and may transmit the captured image frames to the electronic device 100 via wired or wireless means, but is not limited thereto.

[0014] In one embodiment, the electronic device 100 can receive image frames captured by the lens 110 and perform subsequent analysis accordingly.

[0015] In the embodiments disclosed herein, the processor 104 can access the modules and program code recorded in the storage circuit 102 to implement the lens detection method proposed herein, the details of which are described below.

[0016] Please refer to Figure 2, which is a flowchart illustrating a lens detection method according to one embodiment of this disclosure. The method of this embodiment can be executed by the electronic device 100 of Figure 1. The details of each step in Figure 2 will be explained below with reference to the components shown in Figure 1. In addition, to make the concept of this invention easier to understand, Figure 3 will be used as an example for explanation below, wherein Figure 3 is an application scenario diagram illustrating one embodiment of this disclosure.

[0017] In step S210, a plurality of image frames 30, 31, ..., 3(N-1), 3N (where N is a positive integer) captured by lens 110 are obtained, and object tracking operation is performed on image frames 30 to 3N.

[0018] In the scenario shown in Figure 3, image frames 30 to 3N are, for example, image frames captured by lens 110 at time points T, T+1, ..., T+N-1, and T+N (T being a time index), but are not limited to these.

[0019] In the embodiments disclosed herein, image frames 30-3N are, for example, images continuously captured by lens 110 of a specific scene. For example, electronic device 100 is, for example, a monitor system, lens 110 is, for example, the lens of a monitor with a fixed position, and image frames 30-3N are, for example, image frames continuously captured by lens 110 of the specific scene (e.g., certain places) it monitors for a period of time, but are not limited to this.

[0020] In the scenario of Figure 3, processor 104 can perform the required object tracking operations on image frames 30-3N. For example, processor 104 can track target objects that may appear in image frames 30-3N based on existing object tracking algorithms and / or object tracking models. In different embodiments, the target object can be a moving object or an object with a fixed position.

[0021] In different embodiments, the object tracking algorithms used by processor 104 may be, for example, the KLT (Kanade-Lucas-Tomasi) optical flow algorithm, mean shift, and Kalman filter. Additionally, the object tracking models used by processor 104 may be, for example, multi-object tracking (MOT), tracker-by-detection frameworks, and deep learning models such as DeepSORT, FairMOT, and ByteTrack, but are not limited to these.

[0022] In the scenario of Figure 3, the processor 104 is configured to track a target object, such as a (moving) human body, and can mark the tracked target object with a corresponding indicator (e.g., a rectangle). However, this is only an example and is not intended to limit the possible implementations disclosed herein.

[0023] However, since object tracking operations based on image frames are highly dependent on the image quality of the target object in the image frame, if the lens cannot clearly capture the target object for some reason (such as being obscured by an object (at least partially)), the target object may lose tracking in the image frame, thereby reducing the performance of the object tracking operation.

[0024] Therefore, the occlusion state of the lens can be determined by using the object tracking results through the subsequent steps in FIG2 in the present embodiment.

[0025] Specifically, in step S220, in response to the determination that a target object is tracked in at least one first image frame in image frames 30-3N, and that the target object is lost in at least one second image frame in image frames 30-3N, the processor 104 determines the lens occlusion state based on the correlation of the at least one second image frame captured at at least one shooting time.

[0026] In one embodiment, the correlation of the at least one second image frame may be characterized, for example, as temporal consistency associated with the at least one second image frame, but is not limited thereto.

[0027] In the scenario of Figure 3, it is assumed that after the processor 104 performs object tracking operation on image frames 30 to 3N, it determines that the target object O (e.g., a human body) has been tracked in both image frames 30 and 32 to 3N.

[0028] In this embodiment, it is assumed that there is an obstruction 399 on the lens 106, so that each image frame 30 to 3N has a portion of the image area that is not clear due to the obstruction 399. In different embodiments, the obstruction 399 is, for example, various foreign objects located on the surface of the lens 106, such as dust, hair, fingerprints, water droplets, etc. that are attached to the surface of the lens 106.

[0029] In Figure 3, suppose that the target object O is blocked by the occluder 399 at time point T+1 corresponding to image frame 31, making it impossible for the lens 106 to clearly capture the target object O, causing the processor 104 to fail to track the target object O in image frame 31. In this case, the processor 104 may not be able to correctly mark the target object O with a rectangle in image frame 31, and therefore determines that the target object O has lost tracking in image frame 31.

[0030] Therefore, image frames 30, 32~3N in Figure 3 can be understood as the first image frame considered in step S220, and image frame 31 can be understood as the second image frame considered in step S220, but it is not limited to this.

[0031] In other embodiments, the processor 104 may also determine whether the target object O has lost tracking based on other methods.

[0032] In one embodiment, after determining that the target object O has been tracked at a certain time point (hereinafter referred to as the first time point), the processor 104 may, for example, acquire a plurality of reference image frames captured by the lens 110 based on an observation window at another subsequent time point (hereinafter referred to as the second time point), and determine individually for these reference image frames whether the target object O has been tracked.

[0033] In one embodiment, the aforementioned reference image frame is, for example, an image frame captured by lens 110 near the second time point. For instance, assuming the second time point is time point T and the length of the observation window is M (M is a positive integer), the aforementioned reference image frame is, for example, M image frames captured by lens 110 from time point Ta to time point T-a+M-1 (a is a positive integer), but this is not limited to this. In different embodiments, the values ​​of M and a can be determined by the designer to be reasonable values ​​according to requirements.

[0034] In this embodiment, the processor 104 can determine that a reference image frame in which the target object O can be tracked is a first reference image frame, and can determine that a reference image frame in which the target object O cannot be tracked is a second reference image frame.

[0035] Then, the processor 104 can determine the proportion of the second reference image frame in the aforementioned reference image frame. In response to determining that this proportion is higher than a preset threshold (e.g., 40%), the processor 104 can determine that the target object O has lost tracking, and can determine at least a portion of the aforementioned second reference image frame as the second image frame considered in step S220, but is not limited to this.

[0036] On the other hand, if the ratio is not higher than a preset threshold, the processor 104 can determine that the target object O has not lost tracking, but it is not limited to this.

[0037] In one embodiment, the processor 104 may obtain a plurality of historical image frames associated with the specific scene and determine the temporal consistency associated with the at least one second image frame as the association of the at least one second image frame.

[0038] For ease of explanation, image frame 31 in FIG3 will still be used below as the second image frame considered in step S220. In this case, processor 104 can obtain a plurality of historical image frames associated with the specific scene and determine the temporal consistency associated with image frame 31. In some embodiments, the aforementioned historical image frames are, for example, other image frames captured by lens 110 for the specific scene before the time point corresponding to the second image frame, but are not limited to this.

[0039] In one embodiment, determining the temporal consistency of image frame 31 may involve analyzing the changes of the aforementioned historical image frames over time to identify whether the specific scene remains consistent at different times. Commonly used methods include: photometric invariance analysis, feature point matching (which can be implemented based on algorithms such as Scale-Invariant Feature Transform (SIFT) or Speeded-Up Robust Features (SURF), background modeling and difference analysis, image entropy and texture analysis, temporal models and deep learning (which can be implemented based on models such as Recurrent Neural Network (RNN) or Transformer), shadow and reflection analysis, but are not limited to these.

[0040] In one embodiment, determining the temporal consistency of image frame 31 can be done by determining whether the target object O in a historical image frame taken at the same time as image frame 31 also lost tracking. For example, since there is usually similar lighting at the same time of day, the target object O is likely to lose tracking at the same time due to similar lighting.

[0041] In one embodiment, in response to determining that the temporal consistency associated with the at least one second image frame is normal (e.g., consistent across different times), the processor 104 can determine that the lens 110 is not obscured. Specifically, if the temporal consistency associated with the at least one second image frame is normal, this means that the target object O was not lost tracking in the at least one second image frame because the lens 110 was obscured, but rather because of changes in lighting in a specific scene.

[0042] In another embodiment, in response to determining that the temporal consistency associated with the at least one second image frame is abnormal (e.g., inconsistent at different times), the processor 104 may determine that the lens 110 is at least partially occluded. Specifically, if the temporal consistency associated with the at least one second image frame is abnormal, this means that the target object O lost tracking in the at least one second image frame not due to changes in lighting in a specific scene, but possibly because the lens 110 was occluded.

[0043] Taking Figure 3 as an example, the processor 104 can determine that the temporal consistency associated with the image frame 31 is abnormal, and then determine that the lens 110 is at least partially obscured (e.g., obscured by the object 399), but it may not be limited to this.

[0044] In one embodiment, in response to the determination that the lens 110 is at least partially obscured, the processor 104 may generate a notification, which may be used, for example, to remind relevant personnel to take appropriate measures, such as cleaning the surface of the lens 110, but is not limited thereto.

[0045] In one embodiment, in response to the determination that the temporal consistency associated with the at least one second image frame is abnormal, the processor 104 performs an image inpainting operation on the at least one second image frame based on an image inpainting algorithm.

[0046] Taking Figure 3 as an example, after the processor 104 determines that the temporal consistency associated with image frame 31 is abnormal, the processor 104 can perform an image inpainting operation on image frame 31 based on an image inpainting algorithm. In this way, missing areas in image frame 31 (such as areas affected by occlusion 399) can be filled or repaired, so that the repaired image frame 31 looks continuous and natural with other image frames.

[0047] In different embodiments, image inpainting algorithms may include, for example, texture synthesis, inpainting diffusion, structure and texture merging, block matching, deep learning methods, and frequency domain inpainting, but are not limited to these.

[0048] In another embodiment, the reaction to determining that the temporal consistency associated with the at least one second image frame is normal (i.e., the lens 110 is not obscured) means that there should be no defective areas in the at least one second image frame caused by the obstruction of the lens 110. In this case, the processor 104 may maintain the at least one second image frame (i.e., without image patching), but is not limited to this.

[0049] In some embodiments, in response to the determination that the temporal consistency associated with the at least one second image frame is abnormal, the processor 104 may also store the relevant abnormal information for subsequent analysis and optimization of the algorithm used for object tracking.

[0050] Furthermore, although the above description assumes that the lens 110 is located outside the electronic device 100, in other embodiments, the lens 110 may also be built into the electronic device 100. In addition, the electronic device 100 can determine the occlusion state of the lens 110 based on the image frames provided by the lens 110, and can also determine the corresponding occlusion states of other built-in lenses and / or external lenses based on the image frames provided by other lenses, but it is not limited to this.

[0051] In summary, the technical solution provided by this disclosure embodiment can determine the lens occlusion state based on the correlation (e.g., temporal consistency) of image frames when it is determined that a target object has lost tracking. If the correlation of the image frames indicates that the lens is at least partially occluded, this disclosure embodiment can notify relevant personnel to take measures such as cleaning the lens. This avoids the impact of obstructions (e.g., dust, fingerprints, etc.) on the lens surface on the object tracking performance. Furthermore, this disclosure embodiment can also perform corresponding image repair operations on the image frames when the correlation of the image frames indicates that the lens is at least partially occluded. This allows the repaired image frames to provide better image quality.

[0052] From another perspective, the technical solution provided in this disclosure embodiment can continuously monitor lens contamination through automated computer vision analysis, thus significantly improving the accuracy and efficiency of detection. Furthermore, by actively performing image inpainting, this disclosure embodiment minimizes the impact of image contamination and defects on object tracking performance. Therefore, the technical solution provided in this disclosure embodiment can be applied to fields such as industrial visual inspection, security monitoring, and autonomous driving, and has broad application prospects.

[0053] Although this disclosure has been presented above with reference to embodiments, it is not intended to limit this disclosure. Anyone skilled in the art may make some modifications and refinements without departing from the spirit and scope of this disclosure. Therefore, the scope of protection of this disclosure shall be determined by the appended claims.

[0054] 100: Electronic devices 102: Storage Circuit 104: Processor 110: Lens S210, S220: Steps 399: Covering T, T+1, T+2, T+N-1, T+N: Time points 30, 31, 3(N-1), 3N: Image frames O: Target object

Claims

1. A lens detection method, comprising: An electronic device acquires a plurality of image frames captured by a lens and performs an object tracking operation on these image frames; In a specific scenario, in response to determining that a target object is tracked in at least one first image frame among the image frames, and that the target object is lost in at least one second image frame among the image frames, the electronic device obtains a plurality of historical image frames associated with the specific scenario based on at least one shooting time of the at least one second image frame, and determines the temporal consistency associated with the at least one second image frame; in response to determining that the temporal consistency associated with the at least one second image frame is normal, the electronic device determines that the lens is not obscured; and in response to determining that the temporal consistency associated with the at least one second image frame is abnormal, the electronic device determines that the lens is at least partially obscured.

2. The method as described in claim 1, comprising: In response to determining that the target object has been tracked at a first time point, at a second time point, at least one reference image frame captured by the lens is obtained based on an observation window, and for each of the at least one reference image frame, it is determined whether the target object has been tracked; the at least one reference image frame in which the target object can be tracked is determined to be at least one first reference image frame, and the at least one reference image frame in which the target object cannot be tracked is determined to be at least one second reference image frame; a proportion of the at least one second reference image frame in the at least one reference image frame is determined; in response to determining that the proportion is higher than a preset threshold, at least a portion of the at least one second reference image frame is determined to be the at least one second image frame.

3. The method as described in claim 1, further comprising: In response to the determination that the temporal consistency associated with the at least one second image frame is abnormal, the electronic device performs an image repair operation on the at least one second image frame based on an image repair algorithm.

4. An electronic device comprising: A storage circuit that stores a piece of code; The system also includes a processor coupled to the storage circuit and accessing the code to execute: acquiring a plurality of image frames captured by a lens and performing an object tracking operation on the image frames; in a specific scene, in response to determining that a target object is tracked in at least one first image frame of the image frames and that the target object is lost in at least one second image frame of the image frames, acquiring a plurality of historical image frames associated with the specific scene based on at least one shooting time of the at least one second image frame, and determining the temporal consistency associated with the at least one second image frame; in response to determining that the temporal consistency associated with the at least one second image frame is normal, determining by the electronic device that the lens is not obstructed; and in response to determining that the temporal consistency associated with the at least one second image frame is abnormal, determining by the electronic device that the lens is at least partially obstructed.

5. The electronic device of claim 4, wherein the processor is configured to perform: in response to determining that the target object has been tracked at a first time point, acquiring at least one reference image frame captured by the lens based on an observation window at a second time point, and individually determining whether the target object has been tracked for each of the at least one reference image frames; determining that the at least one reference image frame in which the target object can be tracked is at least one first reference image frame, and determining that the at least one reference image frame in which the target object cannot be tracked is at least one second reference image frame; determining a proportion of the at least one second reference image frame in the at least one reference image frame; in response to determining that the proportion is higher than a preset threshold, determining at least a portion of the at least one second reference image frame as the at least one second image frame.

6. The electronic device as claimed in claim 4, wherein the processor is further configured to: in response to determining that the temporal consistency associated with the at least one second image frame is abnormal, perform an image inpainting operation on the at least one second image frame based on an image inpainting algorithm.

Images