Obstruction detection method and related device

Abstract

The application provides a kind of shelter detection method and related equipment, the method includes: determining first area set, first area set includes at least one region of interest;Obtain the environmental image of the detection area collected when camera is not blocked;Judge whether there is a face in the environmental image;If there is a face in the environmental image, determine the target area of face in the environmental image;According to the target area updates first area set to form second area set;Obtain detection image;Extract the gray level co-occurrence matrix feature of the second area set of environmental image and detection image;Obtain the first similarity of the gray level co-occurrence matrix of environmental image and detection image;According to the texture threshold range of first similarity, judge whether camera is blocked, update the region of interest by the face detection result in the environmental image, and judge whether camera is blocked based on the texture feature of the region of interest, to improve the speed of feature extraction, enhance the real-time of shelter detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an occlusion detection method and related equipment. Background Technology

[0002] To enhance security, cameras are typically installed near the door locks at the entrances and exits of offices, factories, and homes to capture images of users entering and exiting the premises. However, some malicious users often try to avoid being captured by these cameras by using limbs (fingers, palms, bodies, etc.), sticky objects (chewing gum, tape, etc.), or fixed objects (wooden boards, metal blocks, etc.).

[0003] How to detect when a camera is obstructed in order to promptly alert users and prevent malicious incidents is an urgent problem to be solved. Most existing obstruction detection methods involve large amounts of computation, require high equipment performance, and have excessively long detection times, which cannot meet the needs of real-time early warning. Summary of the Invention

[0004] This application discloses an occlusion detection method and related equipment, which can detect in real time whether a camera is occluded, so as to promptly remind the user.

[0005] The first aspect of this application discloses an occlusion detection method, the method comprising:

[0006] Determine a first set of regions, wherein the first set of regions includes at least one region of interest;

[0007] When the camera is not obstructed, acquire environmental images of the detection area captured by the camera;

[0008] Determine whether a human face exists in the environmental image;

[0009] If a human face is present in the environmental image, determine the target region of the human face in the environmental image;

[0010] The first region set is updated based on the target region to form a second region set;

[0011] Extract the first gray-level co-occurrence matrix features of the second region set of the environmental image;

[0012] Acquire the detection image of the detection area captured by the camera;

[0013] Extract the second gray-level co-occurrence matrix features of the second region set of the detected image;

[0014] Obtain the first similarity between the features of the first gray-level co-occurrence matrix and the second gray-level co-occurrence matrix;

[0015] The camera is determined to be occluded based on the first similarity and texture threshold range.

[0016] Thus, face detection is performed based on environmental images captured by the camera to determine whether a face exists in the environmental image. If a face is found, the first set of detection regions is updated based on the face position to form a second set of regions. Since the second set of regions is formed based on the actual user's face position, it can more accurately define the actual location of the user's face. Texture features are extracted from the second set of regions in the environmental image and the detection image, and the similarity of texture features is used to determine whether the camera is occluded. Because the texture feature extraction area is only the second set of regions, that is, only a part of the environmental image and the detection image, the area for texture feature extraction is significantly reduced, and the content included in the texture features is also reduced accordingly. This reduces the computational load of texture feature extraction and comparison based on texture features, enhances the real-time performance of occlusion detection, and improves the efficiency of occlusion detection.

[0017] In some embodiments, the texture threshold range includes a first threshold and a second threshold, wherein the first threshold is greater than or equal to the second threshold, and determining whether the camera is occluded based on the first similarity and the texture threshold range includes:

[0018] Determine whether the first similarity is greater than the first threshold or less than the second threshold;

[0019] If the first similarity is greater than the first threshold, it is determined that the camera is not obstructed;

[0020] If the first similarity is less than the second threshold, it is determined that the camera is obstructed.

[0021] Thus, by dividing the texture threshold range into a first threshold and a second threshold, comparing the first similarity with the first threshold and the second threshold respectively, and making a judgment based on the relationship between the first similarity and the first threshold and the second threshold, the hierarchical nature of the comparison process is improved by setting the threshold range in layers, thereby improving the accuracy of judging whether the camera is occluded based on the threshold range.

[0022] In some embodiments, the method further includes:

[0023] Extract the first color feature of the second region set of the environmental image;

[0024] Extract the second color features of the second region set of the detected image;

[0025] Obtain the second similarity between the first color feature and the second color feature;

[0026] If the first threshold is greater than the second threshold, and the first similarity is greater than the second threshold but less than the first threshold, the camera is determined to be occluded based on the second similarity and the color threshold range.

[0027] Specifically, when texture features alone cannot accurately determine whether a camera is occluded (e.g., when the first similarity is less than a first threshold but greater than a second threshold), supplementary judgment can be made based on the extracted colors of the environment image and the detection image. By combining texture and color features, the accuracy of occlusion detection can be improved. Furthermore, occlusion detection can also be performed independently based on the color features of the environment image and the detection image, meaning that occlusion detection based on texture features and occlusion detection based on color features are implemented independently.

[0028] In some embodiments, the color threshold range includes a third threshold and a fourth threshold, wherein the third threshold is greater than or equal to the fourth threshold, and determining whether the camera is occluded based on the second similarity and the color threshold range includes:

[0029] Determine whether the second similarity is greater than the third threshold or less than the second threshold;

[0030] If the second similarity is greater than the third threshold, it is determined that the camera is not obstructed.

[0031] If the second similarity is less than the fourth threshold, it is determined that the camera is blocked.

[0032] Thus, by dividing the color threshold range into a third threshold and a fourth threshold, comparing the first similarity with the third and fourth thresholds respectively, and making a judgment based on the relationship between the second similarity and the third and fourth thresholds, the hierarchical nature of the comparison process is improved by setting the color threshold range in layers, thereby improving the accuracy of judging whether the camera is blocked based on the color threshold range.

[0033] In some embodiments, the camera has a pyroelectric infrared sensor, and the method further includes:

[0034] Acquire detection information, which is generated from the sensing and detection area of ​​the pyroelectric infrared sensor;

[0035] If the third threshold is greater than the fourth threshold, and the second similarity is greater than the fourth threshold but less than the third threshold, then the detection area is determined to contain a human body based on the detection information.

[0036] If the detection information indicates the presence of a human body in the detection area, the camera is determined to be unobstructed.

[0037] If the detection information indicates that there is no human body in the detection area, the camera is determined to be obstructed.

[0038] In the case where any one of the following three scenarios is met: the texture features based on the above embodiments cannot determine whether the camera is occluded; the color features based on the above embodiments cannot determine whether the camera is occluded; or neither the texture features nor the color features based on the above embodiments can determine whether the camera is occluded, occlusion detection can be performed using the detection signal of the pyroelectric infrared sensor. The detection signal can be used to determine whether there is a human body in the detection area, so as to prevent people from entering the detection area and causing false detection.

[0039] In some embodiments, updating the first region set according to the target region to form a second region set includes:

[0040] Obtain the union region of the target region and the first region set;

[0041] The second region set is formed based on the union region. The union region is the union of the target region and the first region set. The regions of interest in the first region set are updated by the target region, for example, by increasing the number or size of the regions of interest in the first region set.

[0042] In some embodiments, forming the second region set based on the union region includes:

[0043] Determine whether the number of regions of interest in the union region is greater than a threshold.

[0044] If the number of regions of interest in the union region is greater than the number threshold, determine the excess number of regions of interest in the union region that exceed the number threshold;

[0045] Based on the determination time of the regions of interest in the union region, delete the excess number of regions of interest in the union region with the earliest determination time to form a second region set.

[0046] By limiting the number of regions of interest (ROIs), the performance of the device implementing the occlusion detection method is ensured. By deleting the earliest identified ROIs, invalid ROIs in the second set of regions are avoided, thereby improving the accuracy and timeliness of the ROIs.

[0047] In some embodiments, the method further includes:

[0048] The system receives a first setting signal input by the user and adjusts the second region set according to the first setting signal. The adjustment includes at least one of the following: increasing the number of regions of interest, decreasing the number of regions of interest, changing the shape of the regions of interest, and moving the position of the regions of interest.

[0049] In this way, the regions of interest are manually updated so that the regions of interest in the second set of regions can meet personalized needs and personalized scenarios.

[0050] In some embodiments, determining the first set of regions of the detection region includes:

[0051] Receive a second setting signal input by the user, and form a first region set based on the second setting signal.

[0052] The regions of interest are manually updated so that the regions of interest in the first set can meet personalized needs and personalized scenarios.

[0053] In some embodiments, forming a first region set based on the second setting signal includes:

[0054] A first region subset is formed based on the second setting signal, the first region subset including at least one region of interest;

[0055] Obtain system preset parameters, and form a second region subset based on the system preset parameters. The second region subset includes at least one region of interest.

[0056] The first region set is formed based on the first region subset and the second region subset.

[0057] The first region set is formed by combining system preset parameters and manual settings to ensure that the first region set can more accurately meet the user's needs. The location defined by the first region is closer to the actual location of the user's face in the detection area.

[0058] In some embodiments, after determining that the camera is obstructed, the method further includes triggering an obstruction warning.

[0059] By triggering an occlusion warning, users can be quickly alerted.

[0060] In some embodiments, after determining the first set of regions of the detection region, the method further includes:

[0061] Extract the third gray-level co-occurrence matrix features of the first region set of the environmental image;

[0062] Extract the fourth gray-level co-occurrence matrix features of the first region set of the detected image;

[0063] Obtain the third similarity between the features of the third gray-level co-occurrence matrix and the features of the fourth gray-level co-occurrence matrix;

[0064] The camera is determined to be occluded based on the third similarity and texture threshold range.

[0065] Thus, after setting the first region of interest set, occlusion detection of the camera can be achieved.

[0066] The second aspect of this application discloses a computer-readable storage medium storing computer instructions that, when executed on an occlusion detection device, cause the occlusion detection device to perform the occlusion detection method as described in the first aspect.

[0067] The third aspect of this application discloses an occlusion detection device, which includes a processor and a memory. The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the occlusion detection device to perform the occlusion detection method as described in the first aspect.

[0068] The fourth aspect of this application discloses a chip system applied to an occlusion detection device; the chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected by a line; the interface circuit is used to receive signals from the memory of the occlusion detection device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the chip system executes the occlusion detection method as described in the first aspect.

[0069] It should be understood that the computer-readable storage medium described in the second aspect, the occlusion detection device described in the third aspect, and the chip system described in the fourth aspect all correspond to the method described in the first aspect. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the application scenario of the camera provided in the embodiments of this application.

[0071] Figure 2 This is a schematic diagram of the structure of the smart door lock provided in the embodiment of this application.

[0072] Figure 3 This is a flowchart illustrating the occlusion detection method provided in the embodiments of this application.

[0073] Figure 4 This is a schematic diagram of the region of interest provided in the embodiments of this application.

[0074] Figure 5 This is a schematic diagram of a first region set provided in an embodiment of this application.

[0075] Figure 6This is a schematic diagram of a first region set provided in another embodiment of this application.

[0076] Figure 7 This is a schematic diagram of a first region set provided in yet another embodiment of this application.

[0077] Figure 8 This is a schematic diagram illustrating the formation of a first region set provided in an embodiment of this application.

[0078] Figure 9 This is a schematic diagram illustrating the formation of a second region set provided in an embodiment of this application.

[0079] Figure 10 This is a schematic diagram of the formation of a second region set provided in another embodiment of this application.

[0080] Figure 11 This is a schematic diagram illustrating the formation of a second region set provided in another embodiment of this application.

[0081] Figure 12 This is a schematic diagram of a user-defined second region set provided in an embodiment of this application.

[0082] Figure 13 This is a schematic diagram illustrating the extraction of color features provided in an embodiment of this application.

[0083] Figure 14 This is a schematic diagram illustrating the extraction of gray-level co-occurrence matrix features according to an embodiment of this application.

[0084] Figure 15 This is a flowchart illustrating a similarity and threshold range judgment provided in an embodiment of this application.

[0085] Figure 16 This is a flowchart illustrating an occlusion detection method provided in another embodiment of this application.

[0086] Figure 17 This is a flowchart illustrating an occlusion detection method provided in another embodiment of this application.

[0087] Figure 18 This is a schematic diagram of the occlusion detection device provided in the embodiments of this application. Detailed Implementation

[0088] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0089] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0090] The following section introduces the relevant terms and concepts that may be involved in the embodiments of this application.

[0091] YUV is a color encoding method where "Y" represents luminance (or Luma), which is the grayscale value, and "U" and "V" represent chrominance (or Chrominance), which are used to describe the color and saturation of an image and specify the color of a pixel. The Y component is the characteristic of the luminance part in YUV.

[0092] The gray-level co-occurrence matrix (GLCM) is a matrix function of pixel distance and angle. It reflects the comprehensive information of the image in terms of direction, interval, magnitude of change, and speed by calculating the correlation between the gray levels of two points at a certain distance and in a certain direction. GLCM features are used to describe the texture features of an image. GLCM features can include one or more features such as ASM energy (angular second moment), contrast, inverse different moment, entropy, autocorrelation, sum average, sum variance, sum entropy, difference variance, difference average, difference entropy, relevant information measure, maximum correlation coefficient, uniformity, and variance. One or more of the above features can be combined into a vector to form the GLCM features.

[0093] Region of interest (ROI) is the area in machine vision and image processing that needs to be processed, outlined in the image using shapes such as boxes, circles, ellipses, and irregular polygons.

[0094] Color features, visual features used in image retrieval, are less dependent on the size, orientation, and viewing angle of the image itself compared to other visual features, thus exhibiting higher robustness. Color features can be RGB color features, HIS color features, CMYK color features, etc. Furthermore, methods for describing color features include color histograms, color moments, color sets, color aggregation vectors, and color correlation graphs.

[0095] A pyroelectric infrared sensor is an infrared sensor that passively absorbs the infrared heat energy emitted by an animal's body during activity. The infrared sensor converts the changes in infrared thermal radiation energy between the active human body and background objects into a corresponding electrical signal, which is the detection signal.

[0096] HSV color characteristics, where H (Hue) represents hue, S (Saturation) represents saturation, and V (Value) represents lightness. The HSV (hue, saturation, value) color space model corresponds to a cone-shaped subset in a cylindrical coordinate system. The color gamut represented by the HSV color model is a subset of the CIE chromaticity diagram. At the vertex of the cone (i.e., the origin), V = 0, H and S are undefined, representing black. At the center of the top surface of the cone, S = 0, V = 1, and H is undefined, representing white. In other words, different VHS values ​​identify different color characteristics.

[0097] To obtain information about people in the detection area, cameras are typically installed at pre-set locations. For example... Figure 1 As shown, in order to obtain facial information of people entering and exiting the entrance, a camera is installed on the entrance. The camera is used to capture images of people entering and exiting the entrance. Malicious users often use obstacles to block the camera in order to prevent the camera from capturing their faces, causing the camera to be unable to capture facial information in the detection area.

[0098] When a camera is obstructed, in order to promptly alert the user, the similarity between the images before and after obstruction is often compared. However, this comparison method involves a large amount of computation, places high demands on system performance, and takes a long time, making it impossible to alert the user in real time.

[0099] For ease of explanation, this application uses a smart door lock as an example. Figure 2 According to some embodiments of this application, a schematic diagram of a smart door lock is shown. The smart door lock 100 is installed on the entrance door and includes an obstruction detection device 10 and a camera 20. The obstruction detection device 10 and the camera 20 are electrically connected.

[0100] The camera 20 is used to collect environmental images of the detection area and send the collected environmental images to the occlusion detection device 10. The occlusion detection device 10 determines whether the camera 20 is occluded based on the environmental images.

[0101] Camera 20 may include cameras with a large field of view, such as cat-eye cameras, fisheye cameras, etc.

[0102] The occlusion detection device 10 may include a processor or microcontroller unit (MCU) for occlusion detection based on the image captured by the camera 20. Of course, the occlusion detection device 10 may also include other components, such as memory, display screen, etc.

[0103] Furthermore, camera 20 can also be used to acquire 3D images containing depth information, which relates to the user's biometrics. When performing face recognition using 3D images containing depth information, more accurate facial features can be extracted from the 3D images, helping to improve the accuracy of face recognition. Camera 20 that can be used to acquire 3D images may include TOF cameras, structured light cameras, binocular stereo cameras, depth cameras, infrared cameras, etc.

[0104] In some smart lock technologies, upon initial installation, camera 20 captures an environmental image of the detection area when it is unobstructed and sends this image to occlusion detection device 10. Occlusion detection device 10 extracts the texture features of the environmental image. When a user uses the smart lock, camera 20 is triggered to capture a detection image of the detection area and sends it to occlusion detection device 10. Occlusion detection device 10 extracts the texture features of the detection image and calculates the similarity between the texture features of the environmental image and the detection image. Occlusion detection device 10 determines whether the similarity is less than a threshold. If the similarity is less than the threshold, meaning the environmental image and the detection image are significantly different, then camera 20 is considered occluded. If the similarity is greater than the threshold, meaning the environmental image and the detection image are quite similar, then camera 20 is considered unobstructed.

[0105] The above-mentioned occlusion detection method has the following problems: The occlusion detection device 10 needs to extract the texture features of the detection image each time it performs occlusion detection. Since the area of ​​the image captured by the camera 20 is large, the time for the occlusion detection device 10 to extract the texture features of the entire detection image is long. In addition, since there are many texture features in the entire detection image, the computational workload of the occlusion detection device 10 to calculate the similarity of the texture features of the detection image and the environment image is large. The large occlusion detection delay leads to low real-time performance and high performance requirements for the occlusion detection device 10. Furthermore, due to the large detection area, the probability of false detection increases (for example, if user A only passes through the detection area but does not use the door lock, the occlusion detection device 10 mistakenly believes that the user is using the smart door lock).

[0106] To reduce the computational load of occlusion detection and improve its real-time performance, this application provides an occlusion detection method. This method determines a region of interest (ROI) within the detection area and updates this ROI based on parameters such as the user's body characteristics and usage habits. Feature extraction only requires extracting texture and color features from the ROI in both the environmental and detection images, thus reducing the computational load of feature extraction and similarity calculations. This reduces the latency of occlusion detection and improves its real-time performance. Furthermore, the combined use of texture and color features enhances detection accuracy.

[0107] The following section will use a smart door lock application scenario as an example to detail the technical solution of this application. It is understood that the technical solution of this application can be applied to any electronic device with a camera, including but not limited to smart door locks and security equipment.

[0108] Please see Figure 3 As shown below, an occlusion detection method provided by an embodiment of this application will be described by way of example.

[0109] S301. Determine a first set of regions, which includes at least one region of interest.

[0110] In one embodiment, the first region set is the area within the detection area captured by the camera 20 where a face may appear. The location of the face is not limited to the area where the face is located when the user uses the smart lock, but may also include the area where the face is located when the user performs other operations within the detection area (e.g., looking closely at the camera).

[0111] In one embodiment, after the smart lock is installed, the occlusion detection device 10 sets a first region set according to system preset parameters. The system preset parameters may be at least one of the user's height and usage habits, including door-opening habits (left-handed or right-handed opening), standing habits (standing on the left or right side of the smart lock when opening the door), etc. The system preset parameters are pre-set in the occlusion detection device 10; for example, the smart lock manufacturer sets the first region set based on the height and habits of ordinary users to facilitate user operation. The system preset parameters may also include the location and shape of the region of interest.

[0112] The first region set can be a planar coordinate system, which defines the range of the first region set. For example, an XY coordinate system can be constructed with camera 20 as the origin. The coordinates of the first region set are (X1 = 45CM, X2 = 80CM, Y1 = 10CM, Y2 = 20CM). The area enclosed by the X-axis coordinates of 45CM and 80CM, and the Y-axis coordinates of 10CM and 20CM, is the area defined by the first region set.

[0113] In one embodiment, the first region set is the area in the environmental image captured by the camera 20 where a face may appear. The first region set can be the pixel coordinates of the environmental image. For example, if the size of the environmental image is 400*400, and an image coordinate system is constructed with the center of the environmental image as the origin, then the pixel coordinates are A(20, 30), B(20, 40), C(10, 30), and D(10, 40). The area enclosed by pixel coordinates A, B, C, and D is the area defined by the first region set.

[0114] It is understandable that the first region set can be planar coordinates in a coordinate system established based on the actual position of the camera, or pixel coordinates in the image captured by the camera, and the conversion between planar coordinates and pixel coordinates can be achieved through different coordinate systems.

[0115] In one embodiment, the first region set includes a region of interest, which can be a regular shape, such as a circle, ellipse, rectangle, or square. Of course, the region of interest can also be an irregular shape. That is, this application does not limit the shape of the region of interest.

[0116] For example, please see Figure 4 , Figure 4 The first example set of regions includes a schematic diagram of a region of interest, where Figure 4 (a) shows an irregularly shaped region of interest. Figure 4 (b) shows a region of interest with a regular shape and that shape is rectangular.

[0117] In one embodiment, the first region set includes at least two regions of interest, which are independent of each other, i.e., there is no overlap between the at least two regions of interest.

[0118] For example, please see Figure 5 This is a schematic diagram of the first region set provided in this application. The first region set includes four regions of interest, namely A, B, C, and D. The four regions of interest A, B, C, and D are independent of each other, and the first region set is the union of the four regions of interest A, B, C, and D. In this embodiment, the region of interest is rectangular. It can be understood that in other embodiments, the region of interest may also be circular, elliptical, or other irregular shapes. The number of regions of interest may be two, three, or even more.

[0119] In one embodiment, the first region set includes at least two regions of interest, and there may be overlapping regions between the at least two regions of interest.

[0120] For example, please see Figure 6 This is a schematic diagram of the first region set of the environmental image provided in this application. The environmental image includes four regions of interest, namely A, B, C and D. There is an overlapping area between regions of interest A and B. The first region set is the union of the four regions of interest A, B, C and D.

[0121] In one embodiment, the occlusion detection device 10 is further configured to receive a setting signal input by a user and form a first region set based on the setting signal. The user can input the setting signal via a keypad or touchscreen of the occlusion detection device 10. The setting signal includes region of interest (ROI) information, which includes the shape and location of the ROI. The ROI location is the position of the user-defined region within the detection area, and can be coordinate information or an area directly drawn by the user.

[0122] Please see Figure 7 In some embodiments, when the smart lock is first installed and used, the smart lock displays a boot-up process through the client, such as... Figure 7 As shown in (a), during the power-on process, the prompt "Please set the area of ​​interest" guides the user to input a setting signal via touch on the display interface. The smart lock then sets the area of ​​interest based on the setting signal, such as... Figure 7 As shown in (b), the user manually defines four rectangular regions A, B, C, and D, which are the user-defined regions of interest. A first set of regions is formed based on these user-defined regions of interest. The power-on process can be displayed on the smart lock's monitor or on a display device connected to the smart lock.

[0123] In some embodiments, after the smart door lock is installed, the occlusion detection device 10 presets a first subset of regions, wherein the first subset of regions includes at least one region of interest; the occlusion detection device 10 guides the user to determine a second subset of regions by manually defining or setting parameters, the second subset of regions including at least one region of interest, and a first set of regions is formed based on the first subset of regions and the second subset of regions. The first set of regions is the union of the regions defined by the first subset of regions and the second subset of regions.

[0124] For example, please see Figure 8 , Figure 8 (a) illustrates a first subset of regions preset by the occlusion detection device 10, the first subset of regions including a rectangular region of interest A. Figure 8 (b) illustrates a second subset of regions manually defined by the user, which includes a circular region of interest B, and regions of interest A and B have overlapping areas. Figure (c) illustrates a first set of regions, which is the union of the first subset of regions and the second subset of regions. The union of regions of interest A and regions of interest B forms region of interest C, which is also the first set of regions.

[0125] In some embodiments, after the smart lock is installed, the occlusion detection device 10 first determines a first region set and extracts the gray-level co-occurrence matrix features and color features of the first region set. When the user uses the smart lock or performs other operations, the camera 20 acquires a detection image of the detection area. The occlusion detection device 10 extracts the gray-level co-occurrence matrix features and color features of the first region set of the detection image and calculates the similarity of the gray-level co-occurrence matrix features and color features of the first region set of the environmental image and the detection image, respectively. First, it is determined whether the similarity of the gray-level co-occurrence matrix of the first region set of the environmental image and the detection image is less than a preset texture threshold. If yes, it is determined that the camera is occluded and an occlusion warning is triggered. If no, it is determined whether the similarity of the color features of the first region set of the environmental image and the detection image is less than a preset color threshold. If yes, it is determined that the camera is occluded and an occlusion warning is triggered. If no, it is determined that the camera is not occluded. After the smart lock is installed, the occlusion detection device can extract the gray-level co-occurrence matrix (GLCM) features and color features of a first set of regions defined by the occlusion detection device. By comparing the similarity of the GLCM features and color features of the first set of regions in the environmental image and the detection image, it can detect whether the camera is occluded. The first set of regions only includes areas where the user's face may appear in the detection area. Compared to the entire environmental image, the occlusion detection device 10 only needs to extract the GLCM features and color features of the regions defined by the first set of regions in the environmental image. The area from which the occlusion detection device 10 extracts features is smaller, reducing the computational load and improving the extraction speed of GLCM features and color features. It is understood that in other embodiments, the occlusion detection device 10 may extract only one of the GLCM features and color features from the environmental image and the detection image, and compare the similarity of the extracted GLCM features or color features with the corresponding threshold range to determine whether the camera is occluded.

[0126] S302. Acquire environmental images of the detection area captured by the camera.

[0127] The environmental images are those captured by the camera when it is not obstructed.

[0128] In this embodiment, as Figure 1 As shown, the detection area is the area captured by the camera after the smart door lock is installed.

[0129] Specifically, the size and position of the shooting area can be adjusted by adjusting parameters such as the camera's focal length and optical axis direction, so that the shooting area of ​​the camera can include the area where the user is normally using the smart lock, as well as the area where the user is not normally using the smart lock (such as the area where the user is when using tools to damage the smart lock).

[0130] Environmental images can include visible light images and infrared light images.

[0131] In one embodiment, camera 20 simultaneously captures visible light images and infrared light images. It is understood that camera 20 can directly capture visible light images, but when the ambient light is low, camera 20 cannot capture a clear visible light image. Infrared light can be supplemented by a supplemental light source so that camera 20 can capture an infrared light image.

[0132] In one embodiment, the camera 20 is equipped with an ambient light sensor and an infrared fill light. The ambient light sensor is used to sense the brightness of the ambient light in the detection area. When the camera 20 determines that the brightness of the ambient light is less than the brightness threshold, the camera 20 sends a control command to the infrared fill light to turn on the infrared fill light. The infrared fill light sends infrared light to the detection area, and the camera 20 captures the image of the detection area to obtain an infrared fill light map.

[0133] In one embodiment, the camera 20 periodically acquires images of the detection area, for example, the camera acquires images of the detection area every 30 seconds. Of course, the camera 20 can also acquire images of the detection area under trigger conditions, such as the user using the smart door lock, the camera 20 receiving a trigger signal from an obstruction detection device or other electronic device, but is not limited to these.

[0134] Optionally, the camera 20 may capture one environmental image at a time. Of course, the camera 20 may also capture multiple environmental images at a time, such as ten environmental images each time. The occlusion detection device 10 determines the quality of the environmental images (e.g., sharpness parameters) and selects the one with the best quality as the environmental image.

[0135] It is understood that steps S302 and S301 can be executed simultaneously, or they can be executed sequentially. That is, this application does not limit the execution order of S302 and S301.

[0136] S303. Determine whether a human face exists in the environmental image.

[0137] In one embodiment, facial recognition technology is used to determine whether a human face exists in an environmental image.

[0138] In other embodiments, the presence of a human face in an environmental image can be determined using other techniques, such as machine learning, texture features, etc.

[0139] If not, proceed to step S302, where the occlusion detection device 10 continues to perform face detection based on the environmental image captured by the camera 20 to detect whether a face exists in the environmental image.

[0140] If so, proceed to step S304;

[0141] S304. Determine the target region of the face in the environmental image.

[0142] In one embodiment, the camera 20 periodically acquires environmental images of the detection area or acquires environmental images of the detection area according to trigger conditions. The occlusion detection device 10 performs face detection based on the environmental images acquired by the camera 20, and then determines the target area where the user's face is located within the detection area based on the face detection results. By continuously performing face detection on the environmental images of the detection area acquired by the camera, the actual target area of ​​the user's face within the detection area can be accurately determined.

[0143] S305. Update the first region set based on the target region to form the second region set.

[0144] After the smart lock is installed, a first region set is formed based on the region of interest preset by the occlusion detection device 10 or the region of interest manually set by the user. The first region set is only the region of interest preset by the occlusion detection device 10 or the user. This region of interest is different from the actual area where the user's face is located when using the smart lock or performing other operations in the detection area. By continuously detecting the face in the environmental image of the detection area captured by the camera 20, the actual target area of ​​the user's face in the detection area is accurately determined, that is, the actual position of the face in the detection area. The region of interest in the first region set is updated based on the target area of ​​the face so that the region of interest in the second region set is closer to the actual area where the face appears.

[0145] For example, the region of interest in the first region set is used to define the location of the face of a user with a height of 170 cm. However, the actual height of the user of the smart door lock is 180 cm. Thus, the region of interest in the first region set cannot accurately define the area where the actual user's face is located. The occlusion detection device 10 detects the target area where the face is located in the environmental image and updates the region of interest in the first region set according to the target area, so that the region of interest in the second region set can more accurately define the actual location of the user's face in the detection area.

[0146] For example, if the tenants of the room where the smart lock is installed change, and different tenants have different height characteristics and usage habits (e.g., the previous tenant preferred to open the door from the left side of the lock, while the current tenant prefers to open the door from the right side), the change in tenants will cause the position of the user's face in the detection area to change. As a result, the region of interest stored by the occlusion detection device 10 cannot accurately define the actual position of the current user's face. The occlusion detection device 10 continuously performs face detection on the environmental images captured by the camera 20 to determine the actual target area where the current tenant's face is located in the detection area, and updates the region of interest based on the target area so that the updated region of interest can accurately define the position of the actual user's face in the detection area.

[0147] In one embodiment, when a user is within the detection area, camera 20 captures an environmental image of the detection area. Occlusion detection device 10 performs face detection based on the environmental image captured by camera 20 and determines the actual position of the user's face within the detection area, i.e., the target area. It is understood that when a user is within the detection area, the user's actions are not limited to using a smart lock. Occlusion detection device 10 updates the region of interest (ROI) of the first region set based on the target area to form a second region set. By continuously performing face detection on the environmental image, the area defined by the second region set is made closer to the user's face position when within the detection area.

[0148] Specifically, a second region set is formed by updating the first region set. The updating methods include, but are not limited to, increasing the number of regions of interest (ROIs), adjusting the size of the ROIs, changing the position of the ROIs, and decreasing the number of ROIs. Because different users have different behavioral habits, there are differences in the position, angle, and size of a user's face within the detection area. By adjusting the size, number, and position of the ROIs, the actual position of the user within the detection area can be accurately defined.

[0149] For example, please see Figure 9 This is a schematic diagram illustrating the formation of a second region set of an environmental image provided in an embodiment of this application, wherein... Figure 9 The diagram on the left shows the region of interest in the first region set and the target region where the face is located in the environment image. Figure 9 The diagram on the right shows the second set of regions formed by the union of the first set of regions and the target region. Figure 9The environmental image on the left includes four regions of interest (ROIs): A, B, C, and D. The occlusion detection device 10 performs face detection on the environmental image and determines that a face exists in region E of the environmental image. Furthermore, region E does not overlap with any of the ROIs in the first region set. By determining the union of region E and the ROIs in the first region set, a face detection function is formed. Figure 9 The second region set on the right continuously performs face detection on the environmental image and updates the number of regions of interest based on the target region of the face in the environmental image, so that the regions of interest can more accurately define the actual face position of the user within the detection area.

[0150] For example, please see Figure 10 This is a schematic diagram illustrating the formation of a second set of regions of an environmental image provided in another embodiment of this application, wherein... Figure 10 The diagram on the left shows the region of interest in the first region set and the target region where the face is located in the environment image. Figure 10 The diagram on the right shows the second set of regions formed by the union of the first set of regions and the target region. Figure 10 The second region set in the environmental image on the left includes four regions of interest (ROIs): A, B, C, and D. By performing face detection on the environmental image, it is determined that a face exists in region E of the environmental image, and region E overlaps with the ROIs in the first region set. By merging region E and ROI B to form a new ROI B, it forms the second region set together with ROIs A, C, and D. By continuously performing face detection on the environmental image and updating the size of the ROIs based on the target area of ​​the face in the environmental image, the ROIs can more accurately define the actual face position of the user within the detection area.

[0151] In one embodiment, the number of regions of interest in the environmental image is limited and meets a number threshold. The number of regions of interest in the second region set is fixed. For example, if the number threshold is 4, then the number of regions of interest is a fixed value of 4.

[0152] For example, please see Figure 11 This is a schematic diagram illustrating the formation of a second region set of an environmental image provided in another embodiment of this application. Figure 11 The diagram on the left shows the union of the region of interest in the first region set and the target region containing the face in the environment image. Figure 11 The diagram on the right shows the second set of regions formed by the union of the first set of regions and the target region. Figure 11The second region set in the left-hand environmental image includes four regions of interest (ROIs): A, B, C, and D. Face detection is performed on the environmental image, determining that a face exists in region E, and that region E does not overlap with any ROIs in the first region set. The occlusion detection device 10 determines that the number of ROIs in the union of the first region set and the target region is 5. By comparing this number with a threshold (4), it determines that the number of ROIs in the union region exceeds the threshold. The occlusion detection device 10 obtains the determination time of the ROIs in the union region, and based on this determination time, identifies region A as the first determined ROI in the first region set. Region A is deleted, and region E is added to the first region set to ensure that the number of ROIs in the union region meets the threshold, thus forming the second region set. By continuously detecting faces in environmental images and updating regions of interest (ROIs) based on the target regions of faces in the environmental images and the determination time of ROIs, the updated ROIs are ensured to be obtained based on the target regions determined by the most recent face detection. At the same time, the earliest determined ROI in the first region set is deleted to ensure that the ROI can accurately define the position of the nearest user's face in the detection area. Since the earliest determined ROI is significantly different from the current user's face position in the detection area, the earliest determined ROI is deleted to avoid invalid ROIs in the second region set, thereby improving the accuracy and timeliness of ROIs.

[0153] It is understood that in the above embodiment, the target region contains one region of interest. In other embodiments, the target region may contain two, three, or even more regions of interest. For example, if Zhang San and Li Si are both within the detection area, the occlusion detection device performs face detection on the environmental image to determine the existence of a target region. This target region includes two regions of interest, representing the areas where Zhang San and Li Si's faces are located. Thus, in the above embodiment, if the number of regions of interest in the union region is two times greater than the threshold, the two regions of interest determined earlier need to be deleted to ensure that the number of regions of interest in the union region meets the threshold.

[0154] Optionally, the occlusion detection device determines whether the number of regions of interest (ROIs) in the union region exceeds a threshold. If the number of ROIs in the union region exceeds the threshold, it determines the excess number of ROIs in the union region that exceeds the threshold, where the excess number is the number of ROIs in the union region that exceeds the threshold. Based on a determined time, it determines and deletes the earliest number of excess ROIs in the union region to form a second region set. The determined time is the time of the ROI updated based on face detection, the user-set ROI, or the system-preset ROI.

[0155] The occlusion detection device can determine the order of regions of interest (ROIs) in the union region based on a predetermined time, and delete the excess ROIs in the union region whose predetermined time is the earliest. For example, the union region includes six ROIs A, B, C, D, E, and F. The number threshold is 4, so there are 2 excess ROIs. The predetermined time of the six ROIs is obtained and the order is arranged according to the predetermined time: B, C, D, E, F, A. The two ROIs with the earliest predetermined time, B and C, are deleted. Then the second region set includes four ROIs: D, E, F, and A.

[0156] It is understood that in some embodiments, the number of regions of interest in the second region set is within a preset range, for example, the number of regions of interest is greater than or equal to 1 and less than or equal to 4. It is also understood that the number of regions of interest can be adjusted based on the processing capabilities of the smart lock and the actual application environment.

[0157] In one embodiment, after the smart lock is installed, it stores a first set of regions. During use, the smart lock prompts the user with the message "Please set the region of interest," guiding the user to input a setting signal via touch on the display interface. The smart lock adjusts the region of interest based on this signal; for example, the user can manually define the region. The occlusion detection device 10 updates the first set of regions stored in the device based on the user-defined region of interest, forming a second set of regions. The prompt can be displayed on the smart lock's screen or on a display device connected to the smart lock. Thus, during use, the user updates the region of interest in the detected areas according to actual needs, ensuring that the regions of interest in the second set can more promptly meet the user's personalized requirements.

[0158] Adjusting the region of interest (ROI) includes at least one of the following: increasing the number of ROIs, decreasing the number of ROIs, increasing or decreasing the size of ROIs, changing the shape of ROIs, and moving the position of ROIs. For example, a user can manually delete ROIs to reduce the number of ROIs. A user can also adjust the range of ROIs by sliding them. Alternatively, a user can drag an ROI from position A to position B in the first set of ROIs to change its position. For example, if the environment of the smart lock changes, such as a pillar being installed in the detection area, and the area where the pillar is located overlaps with an ROI in the second set of ROIs, the user can manually delineate and delete the overlapping ROIs to ensure the accuracy of the ROIs in the second set of ROIs.

[0159] Of course, in other embodiments, during the use of the smart lock, the user can trigger the setting of the region of interest by sending control commands (such as the smart lock's key or voice prompts). The setting method is not limited to manual delineation, but can also limit the region of interest by displaying the "coordinates", "shape" and other options of the setting parameters of the setting section.

[0160] In another embodiment, please refer to Figure 12 The occlusion detection device 10 is also used to acquire the feature information of the user to be detected, including height information and usage habits (such as opening the door with the left hand, opening the door with the right hand, etc.). The occlusion detection device 10 matches the feature information and the detection area information to form a matching result, determines the setting area based on the matching result, and updates the first area set pre-stored in the occlusion detection device 10 based on the setting area.

[0161] In one embodiment, the preset information is the correspondence between feature information and region of interest. For example, if the user's feature information is a height of 175 cm and a habit of opening doors with the left hand, the occlusion detection device 10 will match the coordinate information of the region of interest corresponding to the height and usage habit in the detection area information (X1=10, X2=20; Y1=30, Y2=40). That is, the area enclosed by X-axis coordinates 10, X-axis coordinates 20, Y-axis coordinates 30 and 40 is the region of interest corresponding to the coordinate information.

[0162] Thus, by updating the first region set based on the detection results of the environmental image, a second region set is obtained. By continuously updating the region of interest, the region of the second region set can accurately locate the area where the user's face is when using the smart lock.

[0163] S306. Extract the first color feature and the first gray-level co-occurrence matrix feature of the second region set of the environmental image.

[0164] In one embodiment, please refer to Figure 13 , Figure 13 This is a schematic diagram for extracting color features. Figure 13The diagram illustrates a visible light image P1 and a low-light image P2, both of which are visible light environment images of the detection area directly captured by the camera 20. Visible light image P1 is the image captured by the camera 20 under normal lighting conditions, while low-light image P2 is the image captured by the camera 20 in a low-light scene. The occlusion detection device 10 extracts color features (e.g., red and its proportion) from visible light image P1 and low-light image P2, and graphically presents these color features to obtain a color feature map P3. In this embodiment, the color features are displayed graphically; however, in other embodiments, the color features may be data parameters. In this embodiment, by extracting color features from the visible light images of the detection area under different lighting conditions, this extraction method ensures that occlusion detection based on color features is applicable to different lighting conditions. In this embodiment, two visible light images are used to extract color features; however, in other embodiments, multiple visible light images may be used, such as five, six, or more visible light images taken at different times of day.

[0165] The following explanation uses the extraction of HSV color features from an environmental image as an example: The visible light image is converted to the HSV color space, and after scaling the visible light image to a predetermined size, it is divided into multiple cells. Based on the defined HSV color space division, the number of pixels in the corresponding color intervals within each cell is counted, and the pixel ratio of the color intervals, the total number of pixels, and the average position are calculated. It is then determined whether a dominant color exists in each cell. If a dominant color exists, its features are extracted and described; if no dominant color exists, features of a random mosaic pattern are extracted and described. Combining the features of the dominant color and the mosaic pattern, a color feature descriptor is obtained. A dominant color is defined as a color whose pixel count accounts for more than 40% of the total pixel count. Based on the extracted features, a color feature vector set is constructed, which is the HSV color feature. It is understood that the color feature extraction method of this application is not limited to this.

[0166] In one embodiment, please refer to Figure 14 , Figure 14 This is a schematic diagram for extracting the gray-level co-occurrence matrix. Figure 14 The diagram illustrates the Y component image P4 and the infrared light image P5. The Y component image P4 is formed based on the visible light image P1 captured by the camera 20. The texture features of the Y component image P4 and the infrared light image P5 are extracted, and the gray-level co-occurrence matrix feature image P6 is formed based on the texture features.

[0167] In one embodiment, the texture feature extraction method includes: dividing the Y component image P4 and the infrared light image P5 into pixel blocks according to the basic pixel blocks; for each pixel block, calculating the gray-level mean, variance, and gray-level co-occurrence matrix in the 0°, 45°, 90°, and 135° directions respectively; extracting the contrast, correlation, energy, homogeneity, etc. of the gray-level co-occurrence matrix as feature descriptors; forming a feature vector based on the feature descriptors; and the feature vector is the gray-level co-occurrence matrix feature, which can be used to describe the texture features of the image.

[0168] The occlusion detection device 10 performs face detection on the environmental image captured by the camera 20 to determine the actual location of the user's face in the detection area, i.e., the target area. Based on the target area, the region of interest of the first region set stored in the occlusion detection device is updated to obtain the second region set. This allows the second region set to more accurately define the actual location of the user's face within the detection area. Moreover, the area defined by the second region set is only the area where the user's face may exist in the environmental image. The occlusion detection device 10 only needs to extract the color features and gray-level co-occurrence matrix features within the region of interest of the second region set in the environmental image. It does not need to extract the color features and gray-level co-occurrence matrix features outside the second region set in the environmental image. Therefore, by reducing the area where the occlusion detection device 10 extracts color features and gray-level co-occurrence matrix features, the computational load of the occlusion detection device 10 is greatly reduced, and the feature extraction speed of the occlusion detection device 10 is improved.

[0169] S307. Acquire the detection image.

[0170] The detection image is the image of the detection area captured by camera 20.

[0171] In one embodiment, the camera 20 periodically acquires images of the detection area, which can be used as detection images. In other embodiments, when a user uses the smart lock, the camera 20 receives a trigger signal sent by the occlusion detection device 10 or a trigger signal sent by other components of the smart lock to trigger the camera 20 to acquire images of the detection area and send these images to the occlusion detection device 10 as detection images.

[0172] S308. Extract the second color feature and the second gray-level co-occurrence matrix feature of the second region set of the detection image.

[0173] The extraction methods for the second color feature and the second gray-level co-occurrence matrix feature are similar to those for the first color feature and the first gray-level co-occurrence matrix feature in S306. The difference is that the second color feature and the second gray-level co-occurrence matrix feature in S308 are extracted based on the detected image. The specific process can be found in S306 and will not be repeated here.

[0174] S309. Obtain the first similarity of the first gray-level co-occurrence matrix and the second similarity of the first color feature and the second color feature, and determine whether the camera is occluded based on the first similarity, the second similarity and the threshold range. If the camera is occluded, trigger an occlusion warning.

[0175] The methods for calculating similarity include, but are not limited to, the Minkowski distance method, the Manhattan distance method, and the Euclidean distance method.

[0176] Among them, the first similarity between the first gray-level co-occurrence matrix and the second gray-level co-occurrence matrix; the second similarity between the first color feature and the second color feature, the first similarity and the second similarity can be compared and determined with the threshold range judgment camera respectively.

[0177] In one embodiment, the threshold range includes a texture threshold range and a color threshold range, wherein the texture threshold range includes a first threshold and a second threshold, and the color threshold range includes a third threshold and a fourth threshold.

[0178] The occlusion detection method provided in this application first determines whether the camera 20 is occluded based on the first similarity of the gray-level co-occurrence matrix features of the environmental image and the detection image. When the gray-level co-occurrence matrix cannot accurately determine whether the camera is occluded, it then determines whether the camera 20 is occluded based on the color features of the environmental image and the detection image. It is understandable that since the color features of an image represent the probability of each color appearing but do not specify the exact location of a particular color in the image, relying solely on the second similarity of color features to determine whether the environmental image and the detection image are the same, and thus whether the camera is occluded, can easily lead to bias. Gray-level co-occurrence matrix features, on the other hand, describe the texture features of an image. Texture is the spatial correlation of gray levels or colors of adjacent pixels, and texture features can better describe the detailed texture of the region of interest within the detection area. Compared to texture features, color features are not sensitive to rotation, translation, scale changes, or even various deformations. Therefore, there are cases where the color features are the same, but the objects in the image have changed (e.g., they are occluded). Thus, texture features, compared to color features, can more accurately determine whether the features of the region of interest have changed, and thus determine whether the camera is occluded. Meanwhile, when texture features alone cannot accurately determine whether a camera is obstructed, color features are used for judgment. Combining color and texture features improves the accuracy of occlusion detection.

[0179] S310: If it is impossible to determine whether the camera is blocked based on the first similarity, the second similarity, and the threshold range, obtain the detection information and determine whether there is a human body in the detection area based on the detection information.

[0180] The detection information comes from the detection information generated by the pyroelectric infrared sensor.

[0181] If so, proceed to step S311 to determine that the camera is not obstructed.

[0182] If not, proceed to step S312, determine that the camera is blocked, and trigger an obstruction warning.

[0183] The obstruction warning can be a voice message, email message, or sound and light alarm to remind the user that the camera 20 is obstructed.

[0184] Furthermore, when texture and color features are insufficient to accurately determine whether the camera is obstructed, the presence of a human body in the detection area can be determined based on detection information generated by a passive infrared detector. For example, if a user blocks the camera while using a smart lock, and a human body is present in the detection area, it may be due to normal user operation rather than malicious obstruction, and the obstruction detection device 10 will not trigger an obstruction warning. If no human body is present in the detection area, malicious obstruction may occur, triggering an obstruction warning. Thus, when texture and color features cannot determine whether the camera is obstructed, the obstruction detection device 10 also detects the presence of a human body in the detection area to prevent the camera 20 from being obstructed due to normal user activity within the detection area, thereby reducing the probability of false alarms.

[0185] Please see Figure 15 The specific implementation of S309 provided in one embodiment of this application includes:

[0186] S309a. Obtain the first similarity between the first gray-level co-occurrence matrix and the second gray-level co-occurrence matrix, and determine whether the first similarity is greater than the first threshold or less than the second threshold.

[0187] The threshold range includes a first threshold and a second threshold, with the first threshold being greater than the second threshold.

[0188] It is understandable that the first and second thresholds can be set according to the application environment.

[0189] If the first similarity is greater than the first threshold, proceed to step S309b and determine that the camera is not obstructed.

[0190] If the first similarity is less than the second threshold, proceed to step S309c, determine that the camera is obstructed, and trigger an obstruction warning.

[0191] For example, the first threshold is 90% and the second threshold is 30%. If the first similarity is 91%, the similarity between the detected image and the second region set of the environmental image is high, and it can be determined that the camera is not occluded. When the first similarity is 25%, the similarity between the detected image and the second region set of the environmental image is low, and it can be determined that the camera is occluded.

[0192] If the similarity is greater than the second threshold and less than the first threshold, proceed to step S309d: obtain the second similarity between the first color feature and the second color feature, and determine whether the second similarity is greater than the third threshold or less than the fourth threshold.

[0193] The third threshold is greater than the fourth threshold.

[0194] If the second similarity is greater than the third threshold, proceed to step S309e to determine that the camera is not obstructed.

[0195] If the second similarity is less than the fourth threshold, proceed to step S309f to determine that the camera is obstructed and trigger an obstruction warning.

[0196] If the second similarity is less than the third threshold but greater than the fourth threshold, proceed to step S309g: proceed to step S310. This indicates a situation where the first and second similarities, along with the threshold range, cannot determine whether the camera is obstructed.

[0197] This application first determines whether the camera 20 is occluded based on the relationship between the similarity of the gray-level co-occurrence matrices of the environmental image and the detection image and the threshold range. Since texture features better describe the detailed texture of the region of interest within the detection area, and color features are insensitive to rotation, translation, scale changes, and even various deformations, detecting whether the camera is occluded based on the gray-level co-occurrence matrices of the environmental image and the detection image is more accurate than using color features. This application divides the threshold range into a first threshold and a second threshold, and compares the first similarity with the first and second thresholds respectively, making a judgment based on the relationship between the first similarity and the first and second thresholds. By setting the threshold range in layers, the hierarchical nature of the comparison process is improved, enhancing the accuracy of determining whether the camera is occluded based on the threshold range. Furthermore, when the occlusion of the camera 20 can be determined through the gray-level co-occurrence matrix features of the environmental image and the detection image, the occlusion detection device 10 does not need to calculate the similarity of the gray-level co-occurrence matrix features of the environmental image and the detection image again, reducing the judgment process and improving the efficiency of occlusion detection. Thus, in S306, it is only necessary to "extract the first gray-level co-occurrence matrix features of the environment image", and in S308, it is only necessary to "extract the second gray-level co-occurrence matrix features of the detection image". In S309a, when it is impossible to determine whether the camera is occluded based on the gray-level co-occurrence matrix, it is necessary to "extract the first color features of the environment image and extract the second color features of the detection image".

[0198] Furthermore, in some embodiments, the values ​​of the first threshold and the second threshold are equal. Thus, when making a judgment based on texture features, it is only necessary to determine the relationship between the first similarity and the first or second threshold. That is, if the first similarity is greater than the first threshold, it can be determined that the camera is not obstructed; if the first similarity is less than or equal to the first threshold, it can be determined that the camera is obstructed. In this way, whether the camera is obstructed can be determined solely by texture features.

[0199] In some embodiments, the values ​​of the third threshold and the fourth threshold are equal. Thus, when making a judgment based on color features, it is only necessary to determine the relationship between the second similarity and the third or fourth threshold. That is, if the second similarity is greater than the third threshold, it can be determined that the camera is not obstructed; if the second similarity is less than or equal to the third threshold, it can be determined that the camera is obstructed.

[0200] Please see Figure 16 This is an occlusion detection method provided in the embodiments of this application.

[0201] S1601. Determine a first set of regions, which includes at least one region of interest.

[0202] S1602. Obtain environmental images of the detection area captured by the camera.

[0203] S1603. Determine whether a human face exists in the environmental image.

[0204] If not, proceed to step S1602.

[0205] If so, proceed to step S1604: Determine the target region of the face in the environmental image.

[0206] S1605. Update the first region set based on the target region to form the second region set.

[0207] S1606. Extract the first color feature of the second region set of the environmental image.

[0208] S1607. Acquire the detection image.

[0209] S1608. Extract the second color feature of the second region set of the detection image.

[0210] S1609: Obtain the second similarity between the first color feature and the second color feature, and determine whether the second similarity is greater than the third threshold or less than the fourth threshold.

[0211] If the second similarity is greater than the third threshold, proceed to step S1610 to determine that the camera is not obstructed.

[0212] If the second similarity is less than the fourth threshold, proceed to step S1611 to determine that the camera is obstructed and trigger an obstruction warning.

[0213] If the second similarity is less than the third threshold but greater than the fourth threshold, proceed to step S1612: obtain detection information and determine whether a human body exists in the detection area based on the detection information.

[0214] If so, proceed to step S1613 to determine that the camera is not obstructed.

[0215] If not, proceed to step S1614 to determine that the camera is obstructed and trigger an obstruction warning.

[0216] Compared to Figure 3 The occlusion detection method shown in this application uses an occlusion detection device 10 to extract color features from the environmental image and the detection image, and determines whether the camera is occluded based on the similarity of color features of the regions of interest in the environmental image and the detection image. Since the extraction of gray-level co-occurrence matrix features and the comparison of similarity of the regions of interest are reduced, the computational load of the occlusion detection device can be further reduced, and the real-time performance of occlusion detection can be improved.

[0217] Please see Figure 17 This is an occlusion detection method provided in the embodiments of this application.

[0218] S1701. Determine a first set of regions, which includes at least one region of interest.

[0219] S1702. Acquire environmental images of the detection area captured by the camera.

[0220] S1703. Determine whether a human face exists in the environmental image.

[0221] If not, proceed to step S1702.

[0222] If so, proceed to step S1704: Determine the target region of the face in the environmental image.

[0223] S1705. Update the first region set based on the target region to form the second region set.

[0224] S1706. Extract the first gray-level co-occurrence matrix features of the second region set of the environmental image.

[0225] S1707. Acquire the detection image.

[0226] S1708. Extract the second gray-level co-occurrence matrix features of the second region set of the detected image.

[0227] S1709: Obtain the first similarity between the first gray-level co-occurrence matrix feature and the second gray-level co-occurrence matrix feature, and determine whether the first similarity is greater than the first threshold or less than the second threshold.

[0228] If the first similarity is greater than the first threshold, proceed to step S1710 to determine that the camera is not obstructed.

[0229] If the first similarity is less than the second threshold, proceed to step S1711 to determine that the camera is blocked and trigger an occlusion warning.

[0230] If the first similarity is less than the first threshold and greater than the second threshold, proceed to step S1712: obtain detection information and determine whether there is a human body in the detection area based on the detection information.

[0231] If so, proceed to step S1713 to determine that the camera is not obstructed.

[0232] If not, proceed to step S1714 to determine that the camera is obstructed and trigger an obstruction warning.

[0233] Compared to Figure 3 The occlusion detection method shown in this application uses an occlusion detection device 10 that is only used to extract the gray-level co-occurrence matrix features of the environmental image and the detection image, and to determine whether the camera is occluded based on the similarity of the gray-level co-occurrence matrix features of the regions of interest in the environmental image and the detection image. Since the extraction of color features of the regions of interest and the comparison of similarity are reduced, the computational load of the occlusion detection device can be further reduced, and the real-time performance of occlusion detection can be improved.

[0234] Figure 18 This is a schematic diagram of the structure of an occlusion detection device 10 provided in an embodiment of this application. Figure 18 As shown, the occlusion detection device 10 may include: an RF circuit 901, a memory 902, an input unit 903, a display unit 904, a sensor 905, an audio circuit 906, a Wi-Fi module 907, a processor 908, a power supply 909, and a PIR 910. Those skilled in the art will understand that... Figure 18 The structure shown does not constitute a limitation on the service flow aggregation and forwarding node, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0235] RF circuit 901 can be used to send and receive warning information and warning indication information. It receives and transmits signals, specifically receiving downlink information from the base station and then transferring it to processor 908 for processing. Additionally, it sends uplink data to the base station. Typically, RF circuit 901 includes, but is not limited to: an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc.

[0236] The memory 902 can be used to store software programs and modules. The processor 908 executes various functional applications and data processing of the occlusion detection device by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the occlusion detection device (such as audio data, telephone directory, etc.). In addition, the memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0237] The input unit 903 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the occlusion detection device. Specifically, the input unit 903 may include a touch panel 9031 and other input devices 9032. The touch panel 9031, also known as a touch screen, can collect user touch operations on or near it (such as user operations using fingers, styluses, or other suitable objects or accessories on or near the touch panel 9031 to define a set area), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 9031 may include two parts: a touch detection device and a touch occlusion detection device. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch occlusion detection device; the touch occlusion detection device receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 908, and receives and executes commands from the processor 908. Furthermore, the touch panel 9031 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. In addition to the touch panel 9031, the input unit 903 may also include other input devices 9032. Specifically, these other input devices 9032 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0238] Display unit 904 can be used to display information input by the user or information provided to the user, as well as the area of ​​interest or environmental image of the occlusion detection device. Display unit 904 may include display panel 9041, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 9031 may cover display panel 9041. When touch panel 9031 detects a touch operation on or near it, it transmits the information to processor 908 to determine the type of touch event. Subsequently, processor 908 provides corresponding visual output on display panel 9041 based on the type of touch event. Touch panel 9031 and display panel 9041 can be integrated to realize the input and output functions of the occlusion detection device.

[0239] The occlusion detection device may also include at least one sensor 905, such as a light sensor and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 9041 according to the ambient light level, and the proximity sensor can turn off the display panel 9041 and / or the backlight when the occlusion detection device is moved to the ear; in addition, the occlusion detection device may also be equipped with other sensors such as a barometer, hygrometer, thermometer, and infrared sensor, which will not be described in detail here.

[0240] Audio circuit 906, speaker 9061, and microphone 9062 provide an audio interface between the user and the occlusion detection device. Audio circuit 906 converts received audio data into electrical signals and transmits them to speaker 9061, where speaker 9061 converts them into sound signals for output. On the other hand, microphone 9062 converts collected sound signals into electrical signals, which are received by audio circuit 906, converted into audio data, and then processed by processor 908 before being sent to another occlusion detection device via RF circuit 901, or the audio data can be output to memory 902 for further processing.

[0241] Wi-Fi is a short-range wireless transmission technology. The obstruction detection device, through the Wi-Fi module 907, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 18 Wi-Fi module 907 is shown, but it is understood that it is not a necessary component of the occlusion detection device and can be omitted as needed without changing the nature of the invention.

[0242] The processor 908 is the control center of the occlusion detection device. It connects various parts of the device via interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 902, and by calling data stored in the memory 902, thereby providing overall monitoring of the occlusion detection device. Optionally, the processor 908 may include one or more processing units; preferably, the processor 908 may integrate an application processor and a modem, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem mainly handles wireless communication. It is understood that the aforementioned modem processor may not be integrated into the processor 908.

[0243] The occlusion detection device also includes a power supply 909 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 908 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0244] The occlusion detection device also includes the PIR910, a passive infrared detector used to acquire detection information of the detection area.

[0245] The occlusion detection device may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0246] Figure 18 The occlusion detection device described herein can be used to implement this application. Figure 3 For some or all of the processes described in the method embodiments, please refer to the foregoing. Figure 3 The relevant descriptions in the embodiments will not be repeated here.

[0247] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on the occlusion detection device, the occlusion detection device performs the aforementioned related method steps to implement the occlusion detection method in the above embodiment.

[0248] This embodiment also provides a computer program product that, when run on an occlusion detection device, causes the occlusion detection device to perform the aforementioned related steps to implement the occlusion detection method in the above embodiment.

[0249] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the occlusion detection method in the above method embodiments.

[0250] In this embodiment, the occlusion detection device, computer storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0251] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0252] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0253] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0254] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0255] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0256] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An occlusion detection method, characterized in that, The method includes: Determine a first set of regions, wherein the first set of regions includes at least one region of interest; When the camera is not obstructed, acquire environmental images of the detection area captured by the camera; Determine whether a human face exists in the environmental image; If a face is present in the environmental image, the target region of the face in the environmental image is determined, and the first region set is updated based on the target region to form a second region set; Extract the first gray-level co-occurrence matrix features of the second region set of the environmental image; Acquire the detection image of the detection area captured by the camera; Extract the second gray-level co-occurrence matrix features of the second region set of the detected image; Obtain the first similarity between the features of the first gray-level co-occurrence matrix and the second gray-level co-occurrence matrix; The camera is determined to be occluded based on the first similarity and texture threshold range.

2. The method as described in claim 1, characterized in that, The texture threshold range includes a first threshold and a second threshold, wherein the first threshold is greater than or equal to the second threshold, and the step of determining whether the camera is occluded based on the first similarity and the texture threshold range includes: Determine whether the first similarity is greater than the first threshold or less than the second threshold; If the first similarity is greater than the first threshold, it is determined that the camera is not obstructed; If the first similarity is less than the second threshold, it is determined that the camera is obstructed.

3. The method as described in claim 2, characterized in that, The method further includes: Extract the first color feature of the second region set of the environmental image; Extract the second color features of the second region set of the detected image; Obtain the second similarity between the first color feature and the second color feature; If the first threshold is greater than the second threshold, and the first similarity is greater than the second threshold but less than the first threshold, the camera is determined to be occluded based on the second similarity and the color threshold range.

4. The method as described in claim 3, characterized in that, The color threshold range includes a third threshold and a fourth threshold, wherein the third threshold is greater than or equal to the fourth threshold, and the step of determining whether the camera is occluded based on the second similarity and the color threshold range includes: Determine whether the second similarity is greater than the third threshold or less than the fourth threshold; If the second similarity is greater than the third threshold, it is determined that the camera is not obstructed; If the second similarity is less than the fourth threshold, it is determined that the camera is blocked.

5. The method as described in claim 4, characterized in that, The camera has a pyroelectric infrared sensor, and the method further includes: Acquire detection information, which is generated by the pyroelectric infrared sensor sensing the detection area; If the third threshold is greater than the fourth threshold, and the second similarity is greater than the fourth threshold but less than the third threshold, then the detection area is determined to contain a human body based on the detection information. If the detection information indicates that a human body is present in the detection area, it is determined that the camera is not obstructed. If the detection information indicates that there is no human body in the detection area, the camera is determined to be obstructed.

6. The method according to any one of claims 1 to 5, characterized in that, The step of updating the first region set based on the target region to form a second region set includes: Obtain the union region of the target region and the first region set; The second region set is formed based on the union region.

7. The method as described in claim 6, characterized in that, The process of forming the second region set based on the union region includes: Determine whether the number of regions of interest in the union region is greater than a threshold. If the number of regions of interest in the union region is greater than the number threshold, determine the excess number of regions of interest in the union region that exceed the number threshold; Based on the determination time of the regions of interest in the union region, delete the excess number of regions of interest in the union region with the earliest determination time to form a second region set.

8. The method as described in claim 1, characterized in that, The method further includes: The system receives a first setting signal input by the user and adjusts the second region set according to the first setting signal. The adjustment includes at least one of the following: increasing the number of regions of interest, decreasing the number of regions of interest, changing the shape of the regions of interest, and moving the position of the regions of interest.

9. The method as described in claim 1, characterized in that, Determining the first set of regions includes: Receive a second setting signal input by the user, and form a first region set based on the second setting signal.

10. The method as described in claim 9, characterized in that, The process of forming a first region set based on the second setting signal includes: A first region subset is formed based on the second setting signal, the first region subset including at least one region of interest; Obtain system preset parameters, and form a second region subset based on the system preset parameters. The second region subset includes at least one region of interest. The first region set is formed based on the first region subset and the second region subset.

11. The method as described in claim 1, characterized in that, After determining that the camera is obstructed, the method further includes triggering an obstruction warning.

12. The method as described in claim 1, characterized in that, After determining the first set of regions for the detection region, the method further includes: Extract the third gray-level co-occurrence matrix features of the first region set of the environmental image; Extract the fourth gray-level co-occurrence matrix features of the first region set of the detected image; Obtain the third similarity between the features of the third gray-level co-occurrence matrix and the features of the fourth gray-level co-occurrence matrix; The camera is determined to be occluded based on the third similarity and texture threshold range.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on the occlusion detection device, cause the occlusion detection device to perform the occlusion detection method as described in any one of claims 1 to 12.

14. An occlusion detection device, characterized in that, The occlusion detection device includes a processor and a memory. The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the occlusion detection device to perform the occlusion detection method as described in any one of claims 1 to 12.

15. A chip system applied to an occlusion detection device; the chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected via a line; the interface circuit is configured to receive signals from a memory of the occlusion detection device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the chip system performs the occlusion detection method as described in any one of claims 1 to 12.

Images