Shielding detection method and device based on Hash similarity and feature similarity

Abstract

The invention relates to an occlusion detection method and device based on Hash similarity and feature similarity. According to the occlusion detection method based on the Hash similarity and the feature similarity, the Hash similarity and the Embd feature similarity of the to-be-detected image and the template image and the template updating time are subjected to threshold judgment, so that an occlusion detection result is obtained, and detection is rapid and accurate.

Description

Technical Field

[0001] The present invention relates to camera occlusion detection, and in particular to an occlusion detection method and device based on hash similarity and feature similarity. Background Art

[0002] With the development of cities, the number of elevators has increased year by year. While providing convenience to people, the safety of elevator behavior has also received more and more attention. Some dangerous behaviors need to be detected and prohibited. For example, in recent years, there have been videos of battery vehicles going upstairs in many places, resulting in battery vehicles catching fire and causing injuries. At present, users are also explicitly prohibited from pushing battery vehicles into the elevator to go upstairs. Now many elevators in various places have battery vehicle detection and elevator control functions. If the camera in the elevator cabin detects that a passenger pushes a battery vehicle into the elevator, it will alert and alarm. However, some passengers will cover the camera with items such as umbrellas and hats after entering the elevator, causing the camera to be unable to collect normal pictures, affecting the battery vehicle elevator control function. This will affect the detection and reminder of illegal items and behaviors such as battery vehicles, and there are safety hazards. Safety hazards regarding personal safety are worthy of attention. Therefore, it is very important to detect the obstruction of the elevator cabin camera in a timely manner.

[0003] With the development of intelligence and neural network algorithms, image change detection methods are becoming more mature. Therefore, appropriate methods can be selected to measure image features and the degree of image change, so as to determine whether the camera is blocked for a short period of time and issue timely warnings.

[0004] In the prior art, 1) there is a method for judging camera occlusion based on image color information. Obtain the color information of each pixel unit of the image; count the number of color types of the image based on the color information of each pixel unit of the image; if the number of color types of the image is less than the critical occlusion threshold, it is determined that the camera shooting the image is blocked; if the number of color types of the image is greater than or equal to the critical occlusion threshold, it is determined that the camera shooting the image is not blocked. This method uses color information, which is easily affected by factors such as lighting and environment, resulting in false detection. In addition, the color information is relatively single, and the threshold needs to be adjusted manually, which is suitable for a single scene; 2) Use the area ratio of the connected area to determine whether it is greater than the set threshold to detect camera occlusion. This method often causes false alarms due to large items and other reasons, and the detection results are inaccurate; 3) Use a depth camera or a binocular camera. The corresponding position information can be obtained through this type of camera to determine whether occlusion occurs, but this type of method has certain requirements for hardware and is not applicable to the original camera in the elevator car. Summary of the invention

[0005] The purpose of the present invention and the technical effects that can be achieved are as follows: In order to solve the above-mentioned problems, the present invention uses an occlusion detection method and device based on hash similarity and feature similarity to detect the behavior of occluding the camera.

[0006] An occlusion detection method based on hash similarity and feature similarity comprises the following steps: S1. Collect image data in the elevator car through a camera; S2, processing the image data, when there is a person in the current frame of the image data, adding it to the sequence of images to be detected, and finally obtaining a sequence of n images to be detected arranged in time sequence; S3, taking the first frame in the sequence of the image to be detected as the initial template image, and then subtracting the acquisition time of each frame in the sequence of the image to be detected from the current system time in a time sequence, and if the difference is less than the template update interval threshold, performing a template update operation; the template update operation includes storing the current image as the template image, and recording the current system time point as the template update time; if the difference is greater than the template update interval threshold, not performing a template image update operation; S4, performing hash value calculations on each frame image and the template image in the image sequence to be detected, calculating the Hamming distance according to the hash value, and then determining the hash similarity between the template image and the image to be detected; S5, extracting the Embd feature vectors of the template image and the image to be detected through a convolutional neural network, and performing similarity difference calculation on the Embd feature vector of the template image and the Embd feature vector of the image to be detected, to obtain the Embd feature similarity between the template image and the image to be detected; S6, compare the hash similarity with the hash similarity threshold, and compare the Embd feature similarity with the Embd feature similarity threshold. If both are greater than the threshold, it is determined that the similarity between the image to be detected and the template image is very different, and it is determined that there may be occlusion; otherwise, it is determined that the image to be detected and the template image are not much different, and it is determined that there is no occlusion, that is, no occlusion occurs during the time period of camera acquisition; S7, when it is determined that there may be occlusion, the template update time is subtracted from the current system time. If the absolute value of the difference is less than the template update interval threshold, it means that the current image to be detected is occluded, and it is determined that there is occlusion, that is, the occlusion occurs during the time period of camera acquisition; S8. If the absolute value of the difference is greater than the template update interval threshold, the current image to be detected is used as the template image, and the remaining images in the sequence of images to be detected continue to be subjected to occlusion detection according to steps S4-S8.

[0007] An occlusion detection device based on hash similarity and feature similarity, comprising: A first acquisition module, used for acquiring image data in the elevator car through a camera; The first processing module is used to process the image data, and when there is a person in the current frame of the image data, add the image sequence to be detected, and finally obtain a sequence of n images to be detected arranged in time sequence; The second processing module is used to use the first frame in the sequence of the image to be detected as the initial template image, and then make a difference between the acquisition time of each frame of the image in the sequence of the image to be detected and the current system time in sequence according to the time sequence, and if the absolute value of the difference is less than the template update interval threshold, the template update operation is performed; the template update operation includes storing the current image as the template image and recording the current system time point as the template update time; if the absolute value of the difference is greater than the template update interval threshold, the template image update operation is not performed; The third processing module is used to calculate the hash value of each frame image in the image sequence to be detected and the template image in turn, calculate the Hamming distance according to the hash value, and then determine the hash similarity between the template image and the image to be detected; The fourth processing module extracts the Embd feature vectors of the template image and the image to be detected through a convolutional neural network, and performs similarity difference calculation on the Embd feature vector of the template image and the Embd feature vector of the image to be detected to obtain the Embd feature similarity between the template image and the image to be detected; The first judgment module is used to compare the hash similarity with the hash similarity threshold, and to compare the Embd feature similarity with the Embd feature similarity threshold. If both are greater than the threshold, it is determined that the similarity between the image to be detected and the template image is very different, and it is determined that there may be occlusion; otherwise, it is determined that the image to be detected and the template image are not very different, and it is determined that there is no occlusion, that is, no occlusion occurs during the time period of camera acquisition; The second judgment module is used to make a difference between the template update time and the current system time when it is judged that there may be occlusion. If the absolute value of the difference is less than the template update interval threshold, it means that the current image to be detected is occluded, and it is judged that there is occlusion, that is, the occlusion occurs within the time period of camera acquisition; if the absolute value of the difference is greater than the template update interval threshold, the current image to be detected is used as the template image, and the remaining images in the image sequence to be detected continue to be detected for occlusion according to steps S4-S8.

[0008] An electronic device, comprising: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0009] A computer-readable storage medium stores computer instructions, which implement the steps of the above method when executed by a processor.

[0010] The beneficial effects of the present invention are: This application performs threshold judgment on the hash similarity and Embd feature similarity of the image to be detected and the template image, as well as the template update time, to obtain the occlusion detection result, and the detection is fast and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of the method of the present invention; Figure 2 is a schematic diagram of a device module of the present invention; Among them, there are a first acquisition module 1, a first processing module 2, a second processing module 3, a third processing module 4, a fourth processing module 5, a first judgment module 6, and a second judgment module 7. DETAILED DESCRIPTION

[0012] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0013] This application performs threshold judgment on the hash similarity and Embd feature similarity of the image to be detected and the template image, as well as the template update time, to obtain the occlusion detection result, and the detection is fast and accurate.

[0014] like Figure 1 , a camera occlusion detection method based on hash similarity and feature similarity, comprising the following steps: S1. Collect image data in the elevator car through a camera; S2. Process the image data. When there is a person in the current frame of the image data, add the image sequence to be detected, and finally obtain a sequence of n images to be detected arranged in time series; n = template update interval threshold * number of images that the algorithm can process per second. Specifically, the HiSilicon Hi35xx computing chip can process 10 images per second, and the template update threshold is set to 60 seconds, then the n value is 600.

[0015] S3, taking the first frame in the sequence of the image to be detected as the initial template image, and then subtracting the acquisition time of each frame in the sequence of the image to be detected from the current system time in a time sequence, and if the difference is less than the template update interval threshold, performing a template update operation; the template update operation includes storing the current image as the template image, and recording the current system time point as the template update time; if the difference is greater than the template update interval threshold, not performing a template image update operation; Specifically, when there are 600 images in the image sequence to be detected, the template is dynamically updated starting from the first frame, the template update time is recorded, and then the second frame image is compared with the template to determine whether occlusion occurs, until the last frame.

[0016] Template update interval threshold: This value is used to ensure that occlusion detection is only effective within the update interval threshold time after a person is present. According to actual tests, it is generally set to 60~120 seconds for different detection sensitivities.

[0017] S4, performing hash value calculations on each frame image and the template image in the image sequence to be detected, calculating the Hamming distance according to the hash value, and then determining the hash similarity between the template image and the image to be detected; Specifically, if the current template is the first frame image, hash values ​​are calculated for the first frame image and the second frame image, and the Hamming distance between the two images is calculated, and the Hamming distance value is used as the hash similarity of the two images.

[0018] Hash value calculation: scale the image to a size of 10*10, convert the scaled image to a 64-level grayscale image, and calculate the grayscale average of 10*10 pixels. Compare the grayscale value of each pixel in the scaled image with the grayscale average value, set the pixel value greater than the average value to 1, and set the pixel value less than the average value to 0, and obtain a 10*10 binary matrix. Combine each row together to obtain a 100-bit image hash value.

[0019] Hamming distance calculation: Compare each bit of the hash value and count the number of unequal values, which is the Hamming distance.

[0020] Hash similarity calculation: that is, Hamming distance value.

[0021] S5, extracting the Embd feature vectors of the template image and the image to be detected through a convolutional neural network, and performing similarity difference calculation on the Embd feature vector of the template image and the Embd feature vector of the image to be detected, to obtain the Embd feature similarity between the template image and the image to be detected; The Embd feature vector extraction process is as follows: 1) The input information is a 224*224*3 image pixel matrix. It is first calculated by a convolution layer with a convolution kernel of 3*3*64 and a stride of 1. Then it is calculated by a convolution layer with a convolution kernel of 3*3*64 and a stride of 1, and the data dimension is 224*224*64.

[0022] 2) The data with a dimension of 224*224*64 is calculated through a pooling layer of 2*2 size and stride 2, and the data dimension is 112*112*128; then it is calculated through two convolution layers with the same convolution kernel size of 3*3*128 and stride 1, and the data dimension is 112*112*128.

[0023] 3) The data with a dimension of 112*112*128 is calculated through a pooling layer of 2*2 size and stride 2, and the data dimension is 56*56*256; then it is calculated through three convolution layers with the same convolution kernel size of 3*3*256 and stride 1, and the data dimension is 56*56*256.

[0024] 4) The data with a dimension of 56*56*256 is calculated through a pooling layer of 2*2 size and stride 2, and the data dimension is 28*28*512; then it is calculated through three convolution layers with the same convolution kernel size of 3*3*512 and stride 1, and the data dimension is 28*28*512.

[0025] 5) The data with a dimension of 28*28*512 is calculated through a pooling layer of 2*2 size and stride 2, and the data dimension is 14*14*512; then it is calculated through three convolution layers with the same convolution kernel size of 3*3*512 and stride 1, and the data dimension is 14*14*512.

[0026] 6) The data with a dimension of 14*14*512 is calculated through a pooling layer of 2*2 size and stride 2, and the data dimension is 7*7*512; then it is calculated through three convolution layers with the same convolution kernel size of 3*3*512 and stride 1, and the data dimension is 7*7*512.

[0027] 7) Pass the data with dimension 7*7*512 through two layers of 1*1*4096 fully connected layers to obtain data dimension 1*1*4096.

[0028] 8) Pass the data with dimension 1*1*4096 through the 1*1*1000 fully connected layer to obtain data with dimension 1*1*1000. This data is used as the Embd feature vector of the image.

[0029] The Embd feature vector is used for similarity difference calculation: the Embd feature vector of the template image is subtracted from each corresponding value of the Embd feature vector of the image to be detected, the squares of all the differences are accumulated to obtain the sum of the squares of the differences, the square root of the sum of the squares of the differences is taken, and then multiplied by 100 to obtain the Embd feature similarity difference value.

[0030] Embd feature similarity calculation: that is, Embd feature similarity difference value.

[0031] S6, compare the hash similarity with the hash similarity threshold, and compare the Embd feature similarity with the Embd feature similarity threshold. If both are greater than the threshold, it is determined that the detection image and the template image are very different in similarity, and it is determined that there may be occlusion; otherwise, it is determined that the detection image and the template image are not very different, and it is determined that there is no occlusion; Specifically, according to S4 and S5, the hash similarity value h and feature similarity value f of the first frame image and the second frame image are calculated. The hash similarity threshold is 65, and the Embd feature similarity threshold is 70. Compare h with 65 and f with 70 respectively. If both exceed the threshold, it is determined that the second frame image is blocked; otherwise, it is determined that the second frame image is not blocked, and the first frame image is removed from the image sequence to be detected, and the S3-S7 operations are repeated from the second frame image.

[0032] Hash similarity threshold: According to actual effect tests, the default value is 65, which has the best effect. The smaller the value, the smaller the change in image content will be judged as occlusion, making occlusion detection more sensitive and leading to false detection; on the contrary, the larger the value, the slower the occlusion detection and may lead to missed detection.

[0033] Embd feature similarity threshold: According to actual effect tests, the default value is 70, which has the best effect. The smaller the value, the smaller the change in image content will be judged as occlusion, making occlusion detection more sensitive and leading to false detection; on the contrary, the larger the value, the slower the occlusion detection and may lead to missed detection.

[0034] S7, when it is determined that there may be occlusion, the template update time is subtracted from the current system time. If the absolute value of the difference is less than the template update interval threshold, it means that the current image to be detected is occluded, and it is determined that there is occlusion, that is, the occlusion occurs during the time period of camera acquisition; S8. If the absolute value of the difference is greater than the template update interval threshold, the current image to be detected is used as the template image, and the remaining images in the sequence of images to be detected continue to be subjected to occlusion detection according to steps S4-S8.

[0035] like Figure 2 , a camera occlusion detection device based on hash similarity and feature similarity, comprising: A first acquisition module 1, used to acquire image data in the elevator car through a camera; The first processing module 2 is used to process the image data. When there is a person in the current frame of the image data, the image sequence to be detected is added to finally obtain a sequence of n images to be detected arranged in time sequence; The second processing module 3 is used to use the first frame in the sequence of the image to be detected as the initial template image, and then make a difference between the acquisition time of each frame of the image in the sequence of the image to be detected and the current system time in sequence according to the time sequence. If the absolute value of the difference is less than the template update interval threshold, the template update operation is performed; the template update operation includes storing the current image as the template image and recording the current system time point as the template update time; if the absolute value of the difference is greater than the template update interval threshold, the template image update operation is not performed; The third processing module 4 is used to calculate the hash value of each frame image in the image sequence to be detected and the template image in turn, calculate the Hamming distance according to the hash value, and then determine the hash similarity between the template image and the image to be detected; The fourth processing module 5 extracts the Embd feature vectors of the template image and the image to be detected through a convolutional neural network, and performs similarity difference calculation on the Embd feature vector of the template image and the Embd feature vector of the image to be detected to obtain the Embd feature similarity between the template image and the image to be detected; The first judgment module 6 is used to compare the hash similarity with the hash similarity threshold, and to compare the Embd feature similarity with the Embd feature similarity threshold. If both are greater than the threshold, it is determined that the similarity between the image to be detected and the template image is very different, and it is determined that there may be occlusion; otherwise, it is determined that the image to be detected and the template image are not very different, and it is determined that there is no occlusion, that is, no occlusion occurs during the time period of camera acquisition; The second judgment module 7 is used to make a difference between the template update time and the current system time when it is judged that there may be occlusion. If the absolute value of the difference is less than the template update interval threshold, it means that the current image to be detected is occluded, and it is judged that there is occlusion, that is, the occlusion occurs during the time period of camera acquisition; if the absolute value of the difference is greater than the template update interval threshold, the current image to be detected is used as the template image, and the remaining images in the image sequence to be detected continue to be detected for occlusion according to steps S4-S8.

[0036] An electronic device, comprising: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0037] In the embodiments provided in the present application, it should be understood that the disclosed method and system can also be implemented in other ways. The method and system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the method and system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0038] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0039] On the other hand, a computer-readable storage medium stores computer instructions, which implement the steps of the above method when executed by a processor. The computer program implements the method as described in any one of the first aspects above when executed by a processor. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory 101 (ROM, Read-Only Memory), a random access memory 101 (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes.

[0040] This application performs threshold judgment on the hash values ​​and features in the image, and the occlusion detection is fast and accurate. For example, in the elevator control detection of the battery car entering the elevator, if a passenger pushes the battery car into the elevator and finds that the elevator alarm is sounded, the camera is blocked, resulting in the battery car not being detected, thereby avoiding the battery car elevator control. Therefore, it is necessary to detect the camera occlusion behavior, and to promptly remind and alarm after the camera is blocked. The present invention detects whether the camera is blocked by collecting a single picture, does not require video or manual filtering, saves time and effort, and can timely and effectively feedback the camera occlusion status. The purpose of the present invention: to timely and effectively detect whether the camera in the car is blocked, so as to avoid certain prohibited behaviors of passengers from being detected, and improve the detection effect of safe elevator riding and the internal status of the car.

[0041] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An occlusion detection method based on hash similarity and feature similarity, characterized in that: The steps include: S1. Collect image data in the elevator car through a camera; S2, processing the image data, when there is a person in the current frame of the image data, adding it to the sequence of images to be detected, and finally obtaining a sequence of n images to be detected arranged in time sequence; S3, taking the first frame in the sequence of images to be detected as the initial template image, and then subtracting the acquisition time of each frame in the sequence of images to be detected from the current system time in a time sequence, and if the difference is less than the template update interval threshold, performing a template update operation; the template update operation includes storing the current image as the template image, and recording the current system time point as the template update time; If the difference is greater than the template update interval threshold, the template image update operation is not performed; S4, performing hash value calculations on each frame image and the template image in the image sequence to be detected, calculating the Hamming distance according to the hash value, and then determining the hash similarity between the template image and the image to be detected; S5, extracting the Embd feature vectors of the template image and the image to be detected through a convolutional neural network, and performing similarity difference calculation on the Embd feature vector of the template image and the Embd feature vector of the image to be detected, to obtain the Embd feature similarity between the template image and the image to be detected; S6, compare the hash similarity with the hash similarity threshold, and compare the Embd feature similarity with the Embd feature similarity threshold. If both are greater than the threshold, it is determined that the similarity between the image to be detected and the template image is very different, and it is determined that there may be occlusion; otherwise, it is determined that the image to be detected and the template image are not much different, and it is determined that there is no occlusion, that is, no occlusion occurs during the time period of camera acquisition; S7, when it is determined that there may be occlusion, the template update time is subtracted from the current system time. If the absolute value of the difference is less than the template update interval threshold, it means that the current image to be detected is occluded, and it is determined that there is occlusion, that is, the occlusion occurs during the time period of camera acquisition; S8. If the absolute value of the difference is greater than the template update interval threshold, the current image to be detected is used as the template image, and the remaining images in the sequence of images to be detected continue to be subjected to occlusion detection according to steps S4-S8.

2. An occlusion detection device based on hash similarity and Embd feature similarity, characterized in that: include: A first acquisition module, used for acquiring image data in the elevator car through a camera; The first processing module is used to process the image data, and when there is a person in the current frame of the image data, add the image sequence to be detected, and finally obtain a sequence of n images to be detected arranged in time sequence; The second processing module is used to use the first frame in the sequence of images to be detected as the initial template image, and then make a difference between the acquisition time of each frame in the sequence of images to be detected and the current system time in sequence according to the time sequence, and if the absolute value of the difference is less than the template update interval threshold, perform a template update operation; the template update operation includes storing the current image as the template image and recording the current system time point as the template update time; If the absolute value of the difference is greater than the template update interval threshold, the template image update operation is not performed; The third processing module is used to calculate the hash value of each frame image in the image sequence to be detected and the template image in turn, calculate the Hamming distance according to the hash value, and then determine the hash similarity between the template image and the image to be detected; The fourth processing module extracts the Embd feature vectors of the template image and the image to be detected through a convolutional neural network, and performs similarity difference calculation on the Embd feature vector of the template image and the Embd feature vector of the image to be detected to obtain the Embd feature similarity between the template image and the image to be detected; The first judgment module is used to compare the hash similarity with the hash similarity threshold, and to compare the Embd feature similarity with the Embd feature similarity threshold. If both are greater than the threshold, it is determined that the similarity between the image to be detected and the template image is very different, and it is determined that there may be occlusion; otherwise, it is determined that the image to be detected and the template image are not very different, and it is determined that there is no occlusion, that is, no occlusion occurs during the time period of camera acquisition; The second judgment module is used to make a difference between the template update time and the current system time when it is judged that there may be occlusion. If the absolute value of the difference is less than the template update interval threshold, it means that the current image to be detected is occluded, and it is judged that there is occlusion, that is, the occlusion occurs during the time period of camera acquisition; If the absolute value of the difference is greater than the template update interval threshold, the current image to be detected is used as the template image, and the remaining images in the sequence of images to be detected continue to be subjected to occlusion detection according to steps S4-S8.

3. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in claim 1.

4. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method as claimed in claim 1 are implemented.

Images