Method and apparatus for occlusion detection

Abstract

The invention provides a method and device for shielding detection, and the method for shielding detection comprises the steps: setting a sensor parameter of a camera as a first parameter value used for judging whether the camera is shielded or not when a first condition is satisfied; and based on an image captured by the camera using the first parameter value, performing occlusion detection for determining whether the camera is occluded, the occlusion detection including: detecting whether the number of white pixels in the image captured by the camera using the first parameter value is greater than a first threshold, and if yes, determining whether the number of white pixels in the image is greater than the first threshold; and when the number of the white pixels is greater than a first threshold value, judging that the camera is in a shielding undetermined state.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a signal processing device, in particular to a method and device for occlusion detection. BACKGROUND

[0002] Cameras (may also be referred to as cameras) have a wide range of applications in the fields of security monitoring, traffic management, intelligent driving, intelligent medical treatment, industrial automation, entertainment and social interaction, etc. For example, a driver state monitoring system (DMS) integrates an in-vehicle camera and an electronic control unit (ECU) to monitor the behavior state of the driver. The DMS reduces the risk of fatigue and distraction by continuously assessing the driver's attention, thereby reducing the accident rate, enhancing passenger safety and meeting regulatory standards. In order to ensure that the camera is running, a method and device are needed to quickly and reliably detect whether the camera is occluded. SUMMARY

[0003] The present disclosure proposes a method and device for occlusion detection, and a system and vehicle using the device.

[0004] According to an aspect of the present disclosure, a method for occlusion detection is provided, comprising: when a first condition is met, setting a sensor parameter of a camera to a first parameter value for judging whether the camera is occluded; and performing occlusion detection for judging whether the camera is occluded based on an image captured by the camera using the first parameter value, wherein performing occlusion detection comprises: detecting whether the number of white pixels in the image captured by the camera using the first parameter value is greater than a first threshold, and in the case where the number of white pixels is greater than the first threshold, judging that the camera is in an occlusion pending state.

[0005] According to the above-mentioned embodiments, reliable detection of camera occlusion, reduction of computational complexity, improvement of detection speed, and reduction of consumed computing resources can be achieved.

[0006] According to an embodiment of the present disclosure, further comprising: after completing the occlusion detection, prohibiting the occlusion detection and restoring the sensor parameter of the camera to the previous setting; when the interval time reaches the first period since the occlusion detection is prohibited, setting the sensor parameter of the camera to the first parameter for judging whether the camera is occluded, performing occlusion detection for judging whether the camera is occluded based on an image captured by the camera using the first parameter value, and repeating the above-mentioned steps.

[0007] According to the above-mentioned embodiments, reliable detection of camera occlusion and stable output of detection results can be achieved without affecting the implementation of functions related to the camera.

[0008] According to an embodiment of the present disclosure, the method further includes: setting the sensor parameter of the camera to a second parameter value for face tracking detection when a second condition is met, for use in face tracking detection.

[0009] According to an embodiment of the present disclosure, the method further includes: in a case where the interval time from when the last occlusion detection is prohibited to reach a first period of time, setting the sensor parameter of the camera to the first parameter value, performing the occlusion detection based on an image captured by the camera using the first parameter value, prohibiting the occlusion detection when the occlusion detection is completed, and restoring the sensor parameter of the camera to a second parameter value for face tracking detection when the interval time from when the last occlusion detection is prohibited to reach the first period of time.

[0010] According to the above-mentioned embodiments, the camera occlusion can be reliably detected while ensuring the implementation of the camera-related functions.

[0011] According to an embodiment of the present disclosure, in a case where the number of continuous occlusion detections in which the camera is determined to be in the occlusion pending state is greater than a second threshold value, it is determined that the camera is occluded.

[0012] According to the above-mentioned embodiments, the reliability of the occlusion detection result can be improved to prevent false detection.

[0013] According to an embodiment of the present disclosure, the sensor parameter of the camera includes one or more of an exposure time and a gain.

[0014] According to an embodiment of the present disclosure, a plurality of parameter value groups for gradually increasing, gradually decreasing, or randomly changing the brightness of the image are set, each parameter value group including a value of the exposure time and a value of the gain, and wherein setting the sensor parameter of the camera to the second parameter value for face tracking detection includes: setting the sensor parameter of the camera to each of the plurality of parameter value groups in a cycle.

[0015] According to the above-mentioned embodiments, the implementation of the camera-related functions is effectively ensured.

[0016] According to an embodiment of the present disclosure, the first condition is that the duration of the loss of the indication of the head exceeds a third threshold value.

[0017] According to an embodiment of the present disclosure, the second condition is that the duration of the loss of the region of interest exceeds a fourth threshold value.

[0018] According to an embodiment of the present disclosure, detecting whether the number of white pixels in the image captured using the first parameter value is greater than a first threshold value includes: setting a detection region of white pixels in the image, setting a plurality of pixel check points uniformly or non-uniformly in the detection region, and detecting whether the pixels corresponding to the pixel check points are white.

[0019] According to the above embodiments, it is possible to improve the calculation speed and save computing resources in occlusion detection.

[0020] According to an embodiment of this disclosure, the initial position of the detection area is close to a first side of the image, the first side being the side of the illumination device of the camera that captures the image. The method further includes: moving the detection area away from the first side in a direction perpendicular to the first side, detecting whether the pixel corresponding to the pixel checkpoint at the current position of the detection area is white; and repeating the above steps when the cumulative number of white pixels is not greater than the first threshold.

[0021] According to the above embodiments, it is possible to quickly detect white pixels caused by occlusion and save computing resources.

[0022] According to embodiments of this disclosure, when the detection area is moved away from the first side multiple times in a direction perpendicular to the first side, the step size of each movement is uniform or non-uniform, wherein the step size of the movement is less than the length of the detection area in the direction perpendicular to the first side.

[0023] According to an embodiment of this disclosure, when multiple pixel checkpoints are uniformly arranged in the detection area, the step size of the movement is a value other than an integer multiple of the interval between the pixel checkpoints uniformly arranged in a direction perpendicular to the first side.

[0024] According to the above embodiments, it is possible to avoid repeatedly detecting the same pixels, thus saving computing resources.

[0025] According to embodiments of this disclosure, the pixels in the image corresponding to the pixel checkpoint are located at positions that offset the pixel checkpoint horizontally and / or vertically by random values.

[0026] According to the above embodiments, the reliability of the detection results can be improved by avoiding inaccurate detection results caused by bad pixels in the image sensor.

[0027] According to another aspect of this disclosure, an apparatus for occlusion detection is provided, comprising: a brightness control module configured to set sensor parameters of a camera to a first parameter value for determining whether the camera is occluded when a first condition is met; and an occlusion detection module configured to perform occlusion detection for determining whether the camera is occluded based on an image captured by the camera using the first parameter value, wherein the occlusion detection module is further configured to: detect whether the number of white pixels in the image captured by the camera using the first parameter value is greater than a first threshold, and if the number of white pixels is greater than the first threshold, determine that the camera is in an occlusion pending state.

[0028] According to the above embodiments, reliable detection of camera obstruction can be achieved, computational complexity can be reduced, detection speed can be increased, and computational resources can be reduced.

[0029] According to an embodiment of this disclosure, the brightness control module is further configured to disable occlusion detection after completion of occlusion detection, restore the sensor parameters of the camera to the previous settings, and when the interval time from the start of disabling occlusion detection reaches a first time period, set the sensor parameters of the camera to a first parameter for determining whether the camera is occluded. The occlusion detection module is further configured to perform occlusion detection for determining whether the camera is occluded based on the image captured by the camera using the first parameter value. The brightness control module and the occlusion detection module repeatedly perform the above operations.

[0030] According to the above embodiments, reliable detection of camera obstruction and stable output of detection results can be achieved without affecting the implementation of camera-related functions.

[0031] According to an embodiment of this disclosure, the brightness control module is further configured to: when a second condition is met, set the sensor parameters of the camera to a second parameter value for face tracking detection, so that the face tracking module can perform face tracking detection.

[0032] According to embodiments of this disclosure, the brightness control module is further configured to set the camera's sensor parameters to the first parameter value when the interval from the last time occlusion detection was prohibited reaches a first time period. The occlusion detection module is further configured to perform the occlusion detection based on the image captured by the camera using the first parameter value. The brightness control module is further configured to prohibit the occlusion detection module from performing occlusion detection when the occlusion detection is completed, and restore the camera's sensor parameters to the second parameter value used for face tracking detection when the interval from the last time occlusion detection was prohibited reaches the first time period.

[0033] According to the above embodiments, reliable detection of camera obstruction can be achieved while ensuring the realization of camera-related functions.

[0034] According to embodiments of this disclosure, if the number of consecutive occlusion detections in which the occlusion detection module determines that the camera is in an occlusion pending state is greater than a second threshold, the camera is determined to be occluded.

[0035] According to the above embodiments, the reliability of occlusion detection results can be improved and false detections can be prevented.

[0036] According to embodiments of this disclosure, the sensor parameters of the camera include one or more of exposure time and gain.

[0037] According to embodiments of this disclosure, multiple parameter value groups are set for gradually increasing, gradually decreasing, or randomly changing the image brightness. Each parameter value group includes an exposure time value and a gain value. The brightness control module is also configured to: cyclically set the camera's sensor parameters to each of the multiple parameter value groups with respect to a second parameter value used for face tracking detection.

[0038] According to the above embodiments, the implementation of camera-related functions is effectively guaranteed.

[0039] According to an embodiment of this disclosure, the first condition is that the duration of the indication of a lost head output by the face tracking module exceeds a third threshold.

[0040] According to an embodiment of this disclosure, the second condition is that the duration of the lost region of interest output by the face tracking module exceeds a fourth threshold.

[0041] According to an embodiment of this disclosure, the occlusion detection module is further configured to: set a detection area for white pixels in the image, uniformly or non-uniformly set multiple pixel checkpoints in the detection area, and detect whether the pixels in the image corresponding to the pixel checkpoints are white.

[0042] According to the above embodiments, it is possible to improve the calculation speed and save computing resources in occlusion detection.

[0043] According to an embodiment of this disclosure, the initial position of the detection area is close to a first side of the image, which is the side of the illumination device of the camera that captures the image. The occlusion detection module is further configured to: when the number of white pixels detected at the initial position of the detection area is not greater than a first threshold, move the detection area away from the first side in a direction perpendicular to the first side, detect whether the pixel corresponding to the pixel checkpoint at the current position of the detection area is white, and repeat the above steps when the cumulative number of white pixels is not greater than the first threshold.

[0044] According to the above embodiments, it is possible to quickly detect white pixels caused by occlusion and save computing resources.

[0045] According to embodiments of this disclosure, when the detection area is moved away from the first side multiple times in a direction perpendicular to the first side, the step size of the movement is uniform or non-uniform, wherein the step size of the movement is less than the length of the detection area in the direction perpendicular to the first side.

[0046] According to an embodiment of the present disclosure, when multiple pixel checkpoints are uniformly arranged in the detection area, the step size of each movement is a value other than an integer multiple of the interval between the pixel checkpoints uniformly arranged in a direction perpendicular to the first side.

[0047] According to the above embodiments, it is possible to avoid repeatedly detecting the same pixels, thus saving computing resources.

[0048] According to embodiments of this disclosure, the pixels in the image corresponding to the pixel checkpoint are located at the same position as the pixel checkpoint or at a position where the pixel checkpoint is horizontally and / or vertically offset by a random value.

[0049] According to the above embodiments, the reliability of the detection results can be improved by avoiding inaccurate detection results caused by bad pixels in the image sensor.

[0050] According to another aspect of this disclosure, an apparatus for occlusion detection is provided, comprising: a processor; and a memory storing instructions thereon and coupled to the processor, the instructions, when executed by the processor, causing the apparatus to perform a method for occlusion detection according to an embodiment of this disclosure.

[0051] According to another aspect of this disclosure, a non-transitory computer-readable medium is provided that stores instructions which, when executed by a processor, cause the processor to perform a method for occlusion detection according to embodiments of this disclosure.

[0052] According to another aspect of this disclosure, a driver monitoring system is provided, including a camera and an electronic control unit, the electronic control unit including means for occlusion detection according to embodiments of this disclosure.

[0053] According to another aspect of this disclosure, a vehicle is provided that includes a driver monitoring system, the driver monitoring system including means for occlusion detection according to embodiments of this disclosure.

[0054] According to various embodiments of this disclosure, the methods and apparatus proposed in this disclosure at least solve the above-mentioned problems and other problems that can be identified by those skilled in the art, and can achieve rapid and reliable detection of obstructions and ensure the stable operation of the system with the camera installed. Attached Figure Description

[0055] Figure 1 This is a block diagram of a control unit for occlusion detection according to an embodiment of the present disclosure.

[0056] Figure 2 This is a flowchart of a method for occlusion detection according to an embodiment of the present disclosure.

[0057] Figure 3 This is a schematic diagram of a method for detecting occlusion according to an embodiment of the present disclosure.

[0058] Figure 4 This is a schematic diagram of the panel of a camera according to an embodiment of the present disclosure.

[0059] Figure 5 This is a schematic diagram of an image captured when the camera is obstructed at close range, according to an embodiment of the present disclosure.

[0060] Figure 6 A schematic diagram of a pixel detection region according to this disclosure is shown.

[0061] Figure 7 This is a block diagram of an apparatus for occlusion detection according to an embodiment of the present disclosure.

[0062] Figure 8 This is a block diagram of a driver monitoring system according to an embodiment of the present disclosure.

[0063] Figure 9 This is a block diagram of a vehicle according to an embodiment of the present disclosure. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0065] Furthermore, the various functions described below can be implemented or supported by one or more computer programs, each computer program being formed by computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in appropriate computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, optical disc (CD), digital video disc (DVD), or any other type of storage. "Non-transitory" computer-readable media excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable media includes media that can permanently store data and media that can store and later rewrite data, such as rewritable optical discs or erasable memory devices.

[0066] The terminology used herein to describe embodiments of this application is not intended to limit and / or restrict the scope of this application. For example, unless otherwise defined, the technical or scientific terms used in this disclosure should be understood in their ordinary sense by one of ordinary skill in the art to which this application pertains.

[0067] It should be understood that the terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Unless the context clearly indicates otherwise, the singular forms “a,” “one,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one.

[0068] As used herein, any reference to “an example” or “example,” “an embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The phrases “in one embodiment” or “in one example” appearing in different places in the specification do not necessarily refer to the same embodiment.

[0069] The various embodiments discussed below for describing the principles of this disclosure in this patent document are for illustrative purposes only and should not be construed in any way as limiting the scope of this disclosure.

[0070] In some camera applications, such as Driver Monitoring Systems (DMS), it is necessary to detect whether the camera is obstructed. In existing technologies, one example detection method is to divide the image captured by the camera into multiple units, calculate the variance of the grayscale image of each unit, and consider the camera to be obstructed when the number of units with low variance exceeds a predetermined threshold (e.g., 85%).

[0071] Because the interior environment of a vehicle is relatively uniform, this method may also detect camera obstruction when the driver's face is covered by an object such as a mobile phone. Therefore, this obstruction method has the drawback of not being able to accurately detect camera obstruction at close range (e.g., less than 5cm).

[0072] Furthermore, the DMS includes a brightness control module that continuously adjusts the camera's sensor parameters to obtain multiple images ranging from maximum to minimum brightness for the face tracking system to locate the driver's head. In this process, existing occlusion detection methods often yield incorrect and / or unstable detection results.

[0073] Therefore, there is a need for a method and apparatus for camera occlusion detection that can correctly detect when a camera is occluded at close range (e.g., less than 5cm, less than 10cm, etc.) and output stable detection results.

[0074] Figure 1 This is a block diagram of a control unit 100 for occlusion detection according to an embodiment of the present disclosure.

[0075] refer to Figure 1 The control unit 100 includes an image buffer device 110, a face tracking device 120, an occlusion detection device 130, and a management device 140.

[0076] According to an embodiment of this disclosure, the image buffer device 110 buffers a series of images captured by a camera (not shown) and provides the images to the face tracking device 120 and the occlusion detection device 130 respectively.

[0077] According to embodiments of this disclosure, the face tracking device is used to track and detect the driver's face in an image and provide the occlusion detection device 130 with a region of interest (ROI). The ROI (Region of Interest) and head status indication. The ROI may or may not contain the driver's face. When the face tracking device does not detect an ROI, it does not provide this signal to the occlusion detection device 130. The head status indication is a signal indicating whether a head is present in the image; for example, it may indicate that a head is present in the image, or that a head is absent or missing, i.e., the driver's head is not found in the image.

[0078] According to embodiments of this disclosure, the occlusion detection device 130 includes a brightness control module 131 and an occlusion detection module 132, both of which can receive images from the image buffer device 110. The brightness control module obtains a region of interest (ROI) from the face tracking and detection module, calculates the brightness on the ROI, and configures the camera sensor parameters to maintain stable brightness on the ROI. The camera sensor parameters include one or more of exposure time and gain. The brightness control module 131 also receives a head status indication from the face tracking device 120. Based on the ROI and the head status indication, the brightness control module 131 provides an enable / disable signal to the occlusion detection module 132 to control whether the occlusion detection module 132 performs occlusion detection. The occlusion detection module 132 performs occlusion detection to determine whether occlusion may exist and sends the detection result to the management device 140. According to embodiments of this disclosure, the face tracking device 120 and the occlusion detection device 130 operate in parallel.

[0079] According to an embodiment of this disclosure, the management device 140 receives detection results from the face tracking device 120 and the occlusion detection device 130, and determines whether occlusion has occurred based on the number and continuity of the detection results.

[0080] Figure 2 This is a flowchart of a method for occlusion detection according to an embodiment of the present disclosure.

[0081] According to embodiments of this disclosure, the method for occlusion detection can be provided by... Figure 1 The brightness control module 131 and occlusion detection module 132 of the device 130 for occlusion detection shown are executed, but this method can also be executed by other devices, and this embodiment does not limit the device that implements the method.

[0082] In step S201, when the first condition is met, the sensor parameters of the camera are set to a first parameter value used to determine whether the camera is occluded. The first condition may be whether the duration of the missing head, indicated by a signal representing the head state in the current image, is greater than a predetermined threshold (i.e., a third threshold). This threshold can be set according to one or more of the camera's shooting speed, the operating speed of the face tracking device and / or the occlusion detection device, and system requirements; for example, it can be set to 1 second. Camera parameters include exposure time and gain. Exposure time represents the length of time the image sensor is exposed to light; a longer exposure time makes the image brighter, while a shorter exposure time makes the image darker. Gain is also called photosensitivity. Photosensitivity, also known as ISO (International Organization of Standardization) in the field of photography, represents the sensitivity of the camera's sensor to changes in light. The higher the ISO value, the stronger the sensor's sensitivity to light, and the brighter the image can be obtained in low-light conditions, but it also increases image noise. The first parameter value is a parameter value associated with the occlusion detection method and the reflective characteristics of the occluder. Gain is measured in dB (decibels), and the relationship between gain and sensitivity varies depending on the type of camera.

[0083] In step S202, based on the image captured by the camera using the first parameter value, occlusion detection is performed to determine whether the camera is occluded.

[0084] According to embodiments of this disclosure, it is possible to detect whether the number of white pixels in an image captured by the camera using a first parameter value is greater than a first threshold. If the number of white pixels is greater than the first threshold, it is determined that the camera is in an occlusion pending state.

[0085] In step S203, after occlusion detection is completed, occlusion detection is disabled, and the camera's sensor parameters are restored to their previous settings. When the interval from when occlusion detection was disabled reaches the first time period, the camera's sensor parameters are set to the first parameter. Occlusion detection is performed based on the image captured by the camera using the first parameter value. The above steps are repeated. The number of repetitions can be set as needed.

[0086] According to embodiments of this disclosure, the length of the first time period is related to the camera's shooting speed and the operating speed of the face tracking device. For example, if the camera captures images at 30 frames per second, and according to the operating speed of the face tracking detection device, 15 frames per second is sufficient to detect the head position, then the time for setting the camera's sensor parameters to the first parameter value used to determine whether the camera is occluded should not exceed 14 consecutive frames; otherwise, it will affect the normal operation of the face tracking detection device. The length of the first time period is, for example, 1 second, to ensure the operating time of the face tracking detection device.

[0087] According to embodiments of this disclosure, during each first time period, the sensor parameter settings of the camera are restored to the previous settings, at which time the face tracking and detection module can continue to search for the head in the image captured according to the previous settings.

[0088] In step S204, when the second condition is met after the occlusion detection is prohibited, the sensor parameters of the camera are set to the second parameter value for face tracking detection, so that it can be used for face tracking detection.

[0089] According to embodiments of this disclosure, the second condition is that the duration of the lost region of interest (ROI) exceeds a predetermined threshold (i.e., a fourth threshold). For example, depending on the operating speed of the face tracking detection device and the system design requirements, this predetermined threshold can be set to 30 seconds.

[0090] According to embodiments of this disclosure, the second parameter value can be cyclically set to multiple parameter value groups, which are used to gradually increase, gradually decrease, or randomly change the brightness of the captured image. Each parameter value group includes one or more of the exposure time value and gain value of the sensor used by the camera.

[0091] In step S205, if the interval time since the last occlusion detection was prohibited reaches the first time period, the sensor parameters of the camera are reset to the first parameter value for occlusion detection, and occlusion detection is performed based on the image captured by the camera using the first parameter value.

[0092] According to embodiments of this disclosure, the time for which the sensor parameters of the camera are set to the first parameter value should also not exceed 14 consecutive frames, so as not to affect the normal operation of the face tracking and detection device.

[0093] In step S206, when occlusion detection is completed, occlusion detection is disabled again, and the camera parameters are restored to the parameter value group that was being used in the multiple parameter value groups before the occlusion detection was performed, so that face tracking detection can continue.

[0094] According to embodiments of this disclosure, when the number of consecutive occlusion detections that indicate the camera is in an occlusion pending state exceeds a predetermined threshold (i.e., a second threshold), the camera can be determined to be occluded, for example, by a management device. The predetermined threshold can be selected based on system sensitivity and reliability requirements, for example, it can be selected as 4 times or more.

[0095] According to embodiments of this disclosure, when face tracking detection detects a head, or when a head is present in a head status indicator image, occlusion detection is stopped.

[0096] Figure 3 This is a schematic diagram of a method for detecting occlusion according to an embodiment of the present disclosure.

[0097] refer to Figure 3 When the signal indicating a missing head status lasts for more than 1 second, the brightness control module sets the camera's sensor parameters to values ​​specified in the configuration for occlusion detection. This configuration allows setting one or more values ​​for exposure time and gain. The brightness control module then enables the occlusion detection module, which can then begin performing occlusion detection.

[0098] According to embodiments of this disclosure, for some types of cameras, when determining whether a camera is obstructed, the exposure time of the camera sensor can be set to a range of 5-10 milliseconds or 10-20 milliseconds, and the gain can be set to a range of 0.5-1 or 1-1.5. As mentioned above, the correspondence between the values ​​of exposure time and / or gain and the sensitivity varies depending on the type of camera; therefore, the aforementioned values ​​are exemplary values.

[0099] According to embodiments of this disclosure, occlusion detection can, for example, continue for up to 10, 12, 14, or other consecutive images. After occlusion detection is complete, the brightness control module disables the occlusion detection module and enters a brightness control ROI mode (i.e., a brightness control region of interest mode). In the brightness control ROI mode, the camera's sensor uses the sensor configuration specified for the previously effective ROI, under which the face tracking detection module locates and tracks the driver's face.

[0100] According to an embodiment of this disclosure, when the interval time from the last time occlusion detection was disabled reaches a first time period, such as 0.5 seconds, 1 second, or 1.5 seconds, the brightness control module again sets the sensor parameters of the camera to the values ​​specified by the configuration for occlusion detection and enables the occlusion detection module, thereby starting to perform occlusion detection.

[0101] According to embodiments of this disclosure, after occlusion detection is completed, the brightness control module disables the occlusion detection module again and enters the brightness control ROI mode to perform face tracking detection. If no head is detected, the above process is repeated.

[0102] During the above process, if the face tracking and detection module loses its region of interest (ROI) for more than 30 seconds, the brightness control module enters a brightness control recovery mode. In this mode, the camera's sensor parameters can be cyclically set to values ​​specified by multiple head detection configurations. (Reference) Figure 3 The diagram illustrates seven different head detection configurations that are cyclically set, including head detection configuration 1, head detection configuration 2, ..., head detection configuration 7, but this disclosure is not limited thereto. Each head detection configuration corresponds to a different image brightness. Multiple head detection configurations can vary the brightness of the image captured by the camera from high to low, from low to high, or randomly, until the face tracking detection module detects a head or outputs an alarm signal.

[0103] During the brightness control recovery mode, when the interval reaches the first time period since the last occlusion detection was disabled, the brightness control module pauses the brightness control recovery mode, sets the camera's sensor parameters to the values ​​specified in the occlusion detection configuration, enables the occlusion detection module, and the occlusion detection module can begin performing occlusion detection. After occlusion detection is completed, the brightness control module disables the occlusion detection module and resumes the brightness control recovery mode, continuing to pause the previous head detection configuration and cycling through multiple head detection configurations.

[0104] According to embodiments of this disclosure, when face tracking detects a head, or when a head is present in a head status indicator image, occlusion detection is stopped and the brightness control recovery mode is exited.

[0105] Figure 4 This is a schematic diagram of the panel of a camera according to an embodiment of the present disclosure.

[0106] The camera panel 400 includes a lens 410 disposed on the left side of the panel and an illumination device 420 disposed on the right side of the panel. According to embodiments of this disclosure, the lens 410 can be any type of lens, such as a fixed-focus lens, a zoom lens, a wide-angle lens, etc. For example, the illumination device 420 can be a light-emitting diode (LED), which can emit any suitable type of light, such as infrared light, visible light, or ultraviolet light, so that the camera's sensor can receive the reflection of these lights to expose and form an image.

[0107] Based on the layout of the lens 410 and the illumination device 420, the image captured by the lens of the camera includes a side of the illumination device that is close to the camera (hereinafter referred to as the first side) and a side of the illumination device that is far away from the camera (hereinafter referred to as the second side).

[0108] Figure 5 This is a schematic diagram of an image captured when the camera is obstructed at close range, according to an embodiment of the present disclosure.

[0109] According to embodiments of this disclosure, when the camera is obstructed at close range, such as less than 5 cm, and the obstruction is a hand or other object with similar reflective properties, the camera's sensor parameters are set using the method described in embodiments of this disclosure. The obstruction reflects light emitted by the illumination device back to the sensor in the lens, resulting in overexposure and capturing an image containing light-saturated areas, which appear white. Depending on the shape and / or form of the obstruction, the light-saturated areas in the captured image exhibit different distributions.

[0110] refer to Figure 5 (a) shows an image captured when the camera is obscured by a relatively flat obstruction (e.g., the front or back of a phone or the back of a phone case), where the light saturation area is relatively wide and substantially covers the entire image; (b) shows an image captured when the camera is obscured by a curved or uneven obstruction (e.g., the inside of a palm), where the light saturation area is present on the side of the image close to the camera's illumination device.

[0111] According to embodiments of this disclosure, the presence of light saturation areas in an image can be determined by detecting the number of white pixels in the image, and further, it can be determined whether the camera is obstructed.

[0112] According to embodiments of this disclosure, when detecting the number of white pixels, a detection region for white pixels is set in the image, and multiple pixel checkpoints are uniformly or non-uniformly arranged within the detection region. Whether the pixel in the image corresponding to the pixel checkpoint is white is detected. For example, the multiple pixel checkpoints can be arranged in a matrix.

[0113] It should be noted that after image preprocessing, the light-saturated areas can also be uniformly adjusted to pixels of other gray levels or black pixels. In this case, the detection of white pixels in this scheme can be converted to the detection of black pixels or other pixels with the same gray level.

[0114] According to embodiments of this disclosure, the length of the detection region can be equal to or less than the length of the first side of the image, preferably slightly less than the length of the first side, to ensure that the detection region can cover most of the image in the direction perpendicular to the first side. The width of the detection region is less than the third side of the image perpendicular to the first side, and the width can be arbitrarily set, for example, it can be an integer value of 1 / n of the length of the third side of the image (e.g., expressed in pixels).

[0115] According to embodiments of this disclosure, the initial position of the detection region is set close to the first side of the image to ensure a higher probability of detecting white pixels, thereby reducing computational load. When the number of white pixels detected at the initial position of the detection region is not greater than a first threshold, the detection region is moved away from the first side in a direction perpendicular to the first side. The system then detects whether the pixel corresponding to the pixel checkpoint at the current position of the detection region is white. When the accumulated number of white pixels is not greater than the first threshold, the above steps are repeated until a predetermined number of white pixels have been detected, indicating that the entire image has been analyzed or the detection region has moved to the second side of the image opposite the first side.

[0116] According to embodiments of this disclosure, when the detection area is moved away from the first side in a direction perpendicular to the first side, the step size of the movement is uniform or non-uniform. Preferably, the step size is smaller than the width of the detection area.

[0117] According to embodiments of this disclosure, when the moving step size is less than the width of the detection area and multiple pixel checkpoints are uniformly arranged within the detection area, the moving step size is a value other than an integer multiple of the interval between the pixel checkpoints uniformly arranged in a direction perpendicular to the first side, in order to avoid repetition of checkpoint positions. The moving step size value is an integer.

[0118] According to embodiments of this disclosure, the pixels in the image corresponding to the pixel check being detected can be located at the same position as the pixel check point, or they can be located at a position after the pixel check point has been horizontally and / or vertically offset by a random value within a predetermined range, in order to avoid detecting erroneous pixels caused by bad points in the sensor.

[0119] Figure 6 A schematic diagram of a pixel detection region according to this disclosure is shown.

[0120] refer to Figure 6The camera captures images with a resolution of 1280*800, with the right side of the image being closer to the lighting device. The detection area is 200*600 pixels in size, containing 12 checkpoints arranged in a matrix of 4 rows and 3 columns. The interval between any two points in each row is 100 pixels, and the interval between any two points in each column is 200 pixels. The initial position of the detection area is shown in Region 1 of the image. The right edge of the detection area is 130 pixels from the right edge of the image, and the top and bottom edges are 100 pixels from the top and bottom edges of the image, respectively. The step size for moving the detection area is uniform, 150 pixels at a time. The position of the moved detection area is shown in the image. Figure 6 As shown in regions 2, 4, 5, ... Furthermore, according to embodiments of this disclosure, each pixel position can also be offset by random values ​​in the x and y directions based on the pixel checkpoint, with the random values ​​ranging from [-15, +15].

[0121] According to embodiments of this disclosure, the method for occlusion detection can reliably detect oversaturated regions in an image caused by close-range occlusion without the risk of false detection.

[0122] According to embodiments of this disclosure, the algorithm used for occlusion detection is very simple and consumes very little computational resources.

[0123] For example, when dividing an image into 160 units, some schemes perform occlusion detection by calculating the variance of each unit:

[0124] The number of units to be analyzed is 160 units. The number of operations per column in each unit is 40 (columns) * 4 = 160 operations. The number of operations per row in each unit is 40 (rows) * 3 = 120 operations. The number of operations per unit is 160 + 120 + 8 = 288 operations. The total number of operations for each image is 46080 operations.

[0125] If based on the same software operation and hardware resources, the occlusion detection method disclosed herein has the following characteristics: Number of regions to be analyzed: 6 regions; Number of operations per region: 7 + 12 (pixels) * (4 + 2) = 79 operations; Total number of operations per image: 474. Each operation refers to one computation process at the register level. The aforementioned estimation of the number of operations is for illustrative purposes only.

[0126] As can be seen, the number of operations in the calculation process corresponding to the solution in this application is reduced by an order of magnitude. Therefore, the solution in this application greatly improves efficiency and reduces the computational burden, making it suitable for scenarios with real-time operation or limited computing resources.

[0127] Figure 7This is a block diagram of an occlusion detection apparatus 700 according to an embodiment of the present disclosure.

[0128] According to embodiments of this disclosure, the apparatus 700 for occlusion detection includes a processor 710 and a memory 720. The processor 710 can be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 720 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, solid-state drives (SSDs), hard disk drives (HDDs), and combinations thereof. One or more non-transitory computer-readable media (e.g., memory 720) can collectively store data and instructions that, when executed by the processor 802, are used by the apparatus 700 for occlusion detection to perform operations, such as those described in the method for occlusion detection of this disclosure.

[0129] Figure 8 This is a block diagram of a driver monitoring system according to an embodiment of the present disclosure.

[0130] According to another aspect of this disclosure, a driver monitoring system 800 is proposed, which includes an electronic control unit 810 and a camera 820. The electronic control unit 810 may be the control unit 100 of this disclosure. The electronic control unit 810 may include the occlusion detection device 130 of this disclosure.

[0131] Figure 9 This is a block diagram of a vehicle according to an embodiment of the present disclosure.

[0132] According to another aspect of this disclosure, a vehicle 900 is proposed, which includes a driver monitoring system 800 as described above, the driver monitoring system 800 including a device 130 for occlusion detection. The vehicle may be an electrified vehicle, such as a battery electric vehicle (BEV), a hybrid electric vehicle (HEV), a plug-in hybrid electric vehicle (PHEV), a range-extended electric vehicle (REEV), or a fuel cell electric vehicle (FCEV). The vehicle may also be a hydrogen fuel cell vehicle.

[0133] As those skilled in the art will understand, the occlusion detection method in this disclosure is not limited to occlusion detection of cameras used in vehicles, but can be applied to occlusion detection of cameras in any field.

[0134] Those skilled in the art will understand that the illustrative embodiments described above are not intended to be limiting. It should be understood that any two or more of the embodiments disclosed herein can be combined in any combination. Furthermore, other embodiments may be utilized and other changes may be made without departing from the spirit and scope of the subject matter presented herein. It will be readily understood that aspects of this disclosure, as generally described herein and illustrated in the accompanying drawings, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0135] Those skilled in the art will understand that the various illustrative logic blocks, modules, circuits, and steps described herein can be implemented in hardware, software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in the form of sets of functions. Whether such sets of functions are implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described sets of functions in different ways for each specific application, but such design decisions should not be construed as departing from the scope of this application.

[0136] The steps of the methods or algorithms described in this application may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0137] The above description is merely an exemplary embodiment of this application and is not intended to limit the scope of protection of this application. The scope of protection of this application is determined by the appended claims.

Claims

1. A method for occlusion detection, comprising: When the first condition is met, the sensor parameters of the camera are set to the first parameter value used to determine whether the camera is blocked. as well as Based on the image captured by the camera using the first parameter value, occlusion detection is performed to determine whether the camera is obstructed. The occlusion detection includes: The system detects whether the number of white pixels in the image captured by the camera using a first parameter value is greater than a first threshold. If the number of white pixels is greater than the first threshold, the camera is determined to be in an occluded pending state.

2. The method according to claim 1, further comprising: After completing the occlusion detection, disable occlusion detection and restore the camera's sensor parameters to their previous settings. When the occlusion detection is disabled, and the first time interval has elapsed, the camera's sensor parameters are set to the first parameter used to determine whether the camera is occluded. Based on the image captured by the camera using the first parameter value, occlusion detection is performed to determine whether the camera is obstructed. Repeat the above steps.

3. The method according to claim 1 or 2, further comprising: When the second condition is met, the camera's sensor parameters are set to the second parameter value used for face tracking detection.

4. The method according to claim 3, further comprising: If the interval since the last occlusion detection was prohibited reaches the first time period, the camera's sensor parameters are set to the first parameter value. The occlusion detection is performed based on the image captured by the camera using the first parameter value. When the occlusion detection is completed, occlusion detection should be disabled, and The camera's sensor parameters are restored to the second parameter values ​​used for face tracking detection when the interval from the last time occlusion detection was prohibited reaches the first time period.

5. The method according to claim 1 or 4, wherein, If the number of consecutive occlusion detections that indicate the camera is in an occlusion pending state exceeds the second threshold, the camera is determined to be occluded.

6. The method according to claim 1 or 3, wherein, The camera's sensor parameters include one or more of exposure time and gain.

7. The method according to claim 6, wherein, Set multiple parameter value groups to gradually increase, gradually decrease, or randomly change the image brightness. Each parameter value group includes the exposure time value and the gain value. Setting the camera's sensor parameters to a second parameter value for face tracking detection includes: cyclically setting the camera's sensor parameters to each of a plurality of parameter value groups.

8. The method according to claim 3, wherein, The first condition is that the duration of the indication of a missing head exceeds the third threshold.

9. The method according to claim 3, wherein, The second condition is that the duration of the lost region of interest exceeds the fourth threshold.

10. The method according to claim 1, wherein, Detecting whether the number of white pixels in an image captured using the first parameter value is greater than a first threshold includes: Set a detection region of white pixels in the image. Multiple pixel checkpoints are set uniformly or non-uniformly within the detection area. Detect whether the pixel corresponding to the pixel checkpoint is white.

11. The method according to claim 10, wherein, The initial position of the detection area is close to a first side of the image, which is the side near the illumination device of the camera that captures the image. Wherein, when the number of white pixels detected at the initial position of the detection area is not greater than a first threshold, the method further includes: The detection area moves away from the first side in a direction perpendicular to the first side. Detect whether the pixel corresponding to the checkpoint at the current location in the detection area is white. Repeat the above steps when the cumulative number of white pixels is not greater than the first threshold.

12. The method according to claim 11, wherein, When the detection area is moved away from the first side multiple times in a direction perpendicular to the first side, is the step size of each movement uniform or non-uniform? The step size of the movement is smaller than the length of the detection area in the direction perpendicular to the first side.

13. The method according to claim 12, wherein, When multiple pixel checkpoints are uniformly set within the detection area, the step size of the movement is a value other than an integer multiple of the interval between the pixel checkpoints uniformly set in a direction perpendicular to the first side.

14. The method according to claim 11, wherein, The pixels in the image corresponding to the pixel checkpoint are located at positions that are horizontally and / or vertically offset by random values ​​from the pixel checkpoint.

15. An apparatus for occlusion detection, comprising: The brightness control module is configured to set the sensor parameters of the camera to a first parameter value used to determine whether the camera is blocked when a first condition is met. as well as The occlusion detection module is configured to perform occlusion detection based on the image captured by the camera using a first parameter value, to determine whether the camera is obstructed. The occlusion detection module is further configured as follows: The system detects whether the number of white pixels in the image captured by the camera using a first parameter value is greater than a first threshold. If the number of white pixels is greater than the first threshold, the camera is determined to be in an occluded pending state.

16. The apparatus according to claim 15, in, The brightness control module is also configured to disable occlusion detection after it is completed, restore the camera's sensor parameters to their previous settings, and, when the interval from when occlusion detection is disabled reaches a first time period, set the camera's sensor parameters to the first parameter used to determine whether the camera is occluded. The occlusion detection module is further configured to perform occlusion detection based on the image captured by the camera using the first parameter value to determine whether the camera is occluded. The brightness control module and the occlusion detection module repeatedly perform the above operations.

17. The apparatus according to claim 15 or 16, wherein, The brightness control module is also configured to: When the second condition is met, the camera's sensor parameters are set to the second parameter value for face tracking detection, so that the face tracking module can perform face tracking detection.

18. The apparatus according to claim 17, in, The brightness control module is also configured to set the camera's sensor parameters to the first parameter value when the interval since the last timeout was reached (when the interval since the last timeout was met) is reached. The occlusion detection module is further configured to perform occlusion detection based on an image captured by the camera using a first parameter value. The brightness control module is further configured to, when the occlusion detection is completed, prohibit the occlusion detection module from performing occlusion detection, and restore the sensor parameters of the camera to the second parameter values ​​used for face tracking detection when the interval from the last time occlusion detection was prohibited reaches the first time period.

19. The apparatus according to claim 15 or 18, wherein, If the number of consecutive occlusion detections that the occlusion detection module determines the camera is in an occlusion pending state exceeds the second threshold, the camera is determined to be occluded.

20. The apparatus according to claim 15 or 17, wherein, The camera's sensor parameters include one or more of exposure time and gain.

21. The apparatus according to claim 20, in, Set multiple parameter value groups to gradually increase, gradually decrease, or randomly change the image brightness. Each parameter value group includes the exposure time value and the gain value. The brightness control module is further configured to: cyclically set the camera's sensor parameters to each of a plurality of parameter value groups with respect to the second parameter value used for face tracking detection.

22. The apparatus according to claim 17, wherein, The first condition is that the duration of the missing head indication output by the face tracking module exceeds the third threshold.

23. The apparatus according to claim 17, wherein, The second condition is that the duration of the lost region of interest output by the face tracking module exceeds the fourth threshold.

24. The apparatus according to claim 15, wherein, The occlusion detection module is also configured to: Set a detection region of white pixels in the image. Multiple pixel checkpoints are set uniformly or non-uniformly within the detection area. Detect whether the pixels in the image corresponding to the pixel checkpoint are white.

25. The apparatus according to claim 24, wherein, The initial position of the detection area is close to a first side of the image, which is the side near the illumination device of the camera that captures the image. The occlusion detection module is further configured to detect when the number of white pixels detected at the initial position of the detection area is not greater than a first threshold: The detection area moves away from the first side in a direction perpendicular to the first side. Detect whether the pixel corresponding to the checkpoint at the current location in the detection area is white. Repeat the above steps when the cumulative number of white pixels is not greater than the first threshold.

26. The apparatus according to claim 25, wherein, When the detection area is moved away from the first side multiple times in a direction perpendicular to the first side, is the step size of the movement uniform or non-uniform? The step size of the movement is smaller than the length of the detection area in the direction perpendicular to the first side.

27. The apparatus according to claim 26, wherein, When multiple pixel checkpoints are uniformly set within the detection area, the step size for each movement is a value other than an integer multiple of the interval between the pixel checkpoints uniformly set in a direction perpendicular to the first side.

28. The apparatus according to claim 25, wherein, The pixel in the image corresponding to the pixel checkpoint is located at the same position as the pixel checkpoint or at a position where the pixel checkpoint is horizontally and / or vertically offset by a random value.

29. An apparatus for occlusion detection, comprising: processor; as well as A memory storing instructions and coupled to the processor, the instructions, when executed by the processor, causing the device to perform the method for occlusion detection according to any one of claims 1 to 14.

30. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method for occlusion detection according to any one of claims 1 to 14.

31. A driver monitoring system comprising a camera and an electronic control unit, the electronic control unit comprising the means for occlusion detection according to any one of claims 15-29.

32. A vehicle including a driver monitoring system, said driver monitoring system including means for occlusion detection according to any one of claims 15-29.

Images