A deep learning surveillance video occlusion recognition system and method

Abstract

The present invention relates to the field of image processing technology, and specifically discloses a deep learning surveillance video occlusion recognition system and method, including an image acquisition device, an image processing server and an image analysis platform, wherein the image acquisition device is provided with a plurality of cameras, the cameras are connected to a network module, and are used to transmit real-time surveillance videos and intercepted pictures to the image processing server, an image preprocessing platform is installed on the image processing server, the image preprocessing platform compares, deletes and classifies and stores the captured intercepted pictures, and an image analysis platform is also installed on the image processing server, which is used to analyze and determine the occlusion rate of the intercepted pictures after image preprocessing; the present invention performs dual occlusion rate detection on the pictures in the memory through a histogram and a training deep learning model, firstly detecting the proportion of the occluded pictures, and then detecting the proportion of the pixel values ​​in the histogram, so that the detection is more accurate, and the accuracy of the camera occlusion rate detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a deep learning surveillance video occlusion recognition system and method. Background Art

[0002] Nowadays, the requirements for land use are high, and the nature of land use cannot be changed at will. Therefore, video surveillance of the land is required. Many cameras for land monitoring have been produced, and some cameras need to be installed in more remote areas, which puts higher requirements on the maintenance of the cameras. In addition, the current cameras directly store videos and pictures, which increases the pressure on the memory. If they are not cleaned in time, it will affect the transmission of new data. Cameras in the wild are also easily interfered by the external environment, causing obstructions to the cameras. It is difficult for cameras to detect the existence of obstructions when there are too many cameras. If they are not cleaned in time, it will affect the use of the cameras. Therefore, it is necessary to add a recognition system for camera obstruction.

[0003] There are two methods for detecting video occlusion in the prior art. One is to detect the area of ​​the largest connected area of ​​black in the image. The ratio of this area to the area of ​​the entire image is the occlusion rate. This method can only detect occlusions that are close to the lens or black, and there are misjudgments for occlusions that are far away from the lens or other colors. The other is scene switching judgment, which requires setting a reference scene image and judging whether it is occluded based on the comparison between the current scene and the reference scene. Since the reference scene changes with the season, day / night, construction and other objective factors, the reference scene image needs to be updated frequently, which is a very large workload for the current large number of monitoring equipment. Since the reference scene cannot be updated in time, the accuracy of occlusion detection is still low. Therefore, it is necessary to design a deep learning monitoring video occlusion recognition system and method to solve the problems of large storage capacity of existing cameras, high misjudgment rate of occlusion rate detection, and inaccurate detection. Summary of the invention

[0004] In view of the problems existing in the prior art, the object of the present invention is to provide a deep learning surveillance video occlusion recognition system and method.

[0005] The technical solution adopted by the present invention to solve its technical problem is: a deep learning surveillance video occlusion recognition system, including an image acquisition device, an image processing server and an image analysis platform, the image acquisition device is provided with a plurality of cameras, the cameras are connected to a network module, and are used to transmit real-time monitoring video and intercepted pictures to the image processing server, an image preprocessing platform is installed on the image processing server, the image preprocessing platform compares, deletes and classifies and stores the captured intercepted pictures, and an image analysis platform is also installed on the image processing server, which is used to analyze and determine the occlusion rate of the intercepted pictures after image preprocessing.

[0006] Specifically, the image processing server is provided with a memory for storing monitoring videos and captured images captured by the camera, and the storage time of the video files and image files is set to 1 month in the memory.

[0007] Specifically, video capture software is also installed on the image processing server. Preset points are set in the video capture software. A picture is captured at each preset point. The time for capturing a picture is 15-20 seconds. The preset point setting interval is 2 minutes. It is used to capture the surveillance video collected by the camera to form a captured picture.

[0008] A deep learning surveillance video occlusion recognition method, characterized by comprising the following steps:

[0009] 1) The cameras are numbered and entered into the image preprocessing platform and the image analysis platform. The surveillance videos and captured images within 7 days after the camera is installed are all saved in the memory and used as materials for training the atlas and training the deep learning model.

[0010] 2) The real-time monitoring video of the camera is transmitted to the image processing server through the network module to form a short video of two minutes. The video screenshot software captures the short video into a picture of two minutes, and the short video is directly stored in the memory;

[0011] 3) The captured image is transmitted to the image processing server through the network module and enters the image preprocessing platform;

[0012] 4) The image preprocessing platform compares the captured images in chronological order with the images in the training atlas. If the similarity between the captured images and the images in the training atlas reaches 100%, the captured images are deleted into the storage box, which is automatically deleted after 7 days. If the similarity between the captured images and the images in the training atlas is less than 100%, the captured images are stored in the memory.

[0013] 5) The image analysis platform analyzes the images in the memory to determine the occlusion rate. The captured images in the memory are compared with the trained deep learning model to determine the position and color of the difference area in the captured image. If the difference area is black, and the proportion of the black area in the captured image exceeds the set threshold of 10%, it means that close-range occlusion has occurred. If it is other colors, the position and color of the difference area on the captured image are determined;

[0014] 6) After the intercepted image in the memory is quantized, a histogram is calculated in the RGB color space. The pixel value ratios of different areas in the intercepted image are calculated through the histogram, and compared with the histogram of the image in the training set to determine the position of the different areas in the intercepted image;

[0015] 7) If the captured image and the image in the training atlas have the same position but different colors, the land use status in the monitored area has changed;

[0016] 8) If the position of the different area of ​​the captured image is different from that of the image in the training set, the color is not black, and the proportion of the pixel value of the different area exceeds the set threshold of 10%, then occlusion occurs due to other colored objects.

[0017] Specifically, the deep learning model in step 1) uses pictures taken by the camera in the early stage as a training atlas, and the training atlas is trained by PPYOLO v2 software. The initial model of the software is trained using the COCO data set to obtain a deep learning model of the pictures taken by the camera.

[0018] Specifically, the similarity between the captured image in step 4) and the image in the training atlas is matched using the directly generated original histogram, and the histograms of the two images, HistA and HistB, are calculated respectively, and then the normalized correlation coefficient of the two histograms is calculated, and the normalized correlation coefficient includes the Bhattacharyya distance and the histogram intersection distance.

[0019] Specifically, the RGB color space calculation histogram in step 6) is Figure 1 It is generally composed of three channels: R, G, and B. Each channel consists of 8 bits, with a maximum of 255. If the histogram is constructed directly according to each different value of the three channels, the amount of data will be very large, which is 256*256*256=256 cubed bins. For simplicity, 8 bins are set for each channel. In this way, the maximum value of each channel is 256 / 8=32, that is, each channel is divided into 8 bins, and each bin stores 32 numbers, 0-31, 32-63, 64-127,...224-255. The RGB color space is converted into one-dimensional 8*8*8=512 bins in total.

[0020] The present invention has the following beneficial effects:

[0021] Deep learning surveillance video occlusion recognition system and method designed by the present invention

[0022] 1. Through the histogram and training deep learning model, the images in the memory are double-occluded. The proportion of occluded images is first detected, and then the proportion of pixel values ​​in the histogram is detected, making the detection more accurate and improving the accuracy of camera occlusion rate detection;

[0023] 2. Through the pre-processing of pictures, the same pictures and surveillance at different times can be directly deleted. The deleted content will be automatically deleted after 7 days. The storage time is also set for the memory, which improves the storage capacity and service life of the memory, avoids the loss of recent surveillance videos and pictures caused by long-term full loading of the memory, and maintains the integrity of the data;

[0024] 3. The present invention realizes automatic processing of pictures and videos, reduces manual monitoring, reduces usage costs, and improves work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the deep learning surveillance video occlusion recognition method. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely further describe the technical solutions in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, a deep learning surveillance video occlusion recognition system includes an image acquisition device, an image processing server and an image analysis platform. The image acquisition device is provided with a plurality of cameras, and the cameras are connected to a network module. The network module adopts an STM32 gateway processor for transmitting real-time surveillance video and intercepted pictures to the image processing server. An image preprocessing platform is installed on the image processing server. The image preprocessing platform compares, deletes and classifies and stores the captured intercepted pictures. An image analysis platform is also installed on the image processing server for analyzing and judging the occlusion rate of the intercepted pictures after image preprocessing.

[0028] The image processing server is provided with a memory for storing the surveillance videos and pictures collected by the camera. The storage time of the video files and picture files is set to 1 month. The image processing server is also installed with video capture software. The video capture software is provided with preset points. Each time a preset point is reached, a picture is captured. The time for capturing a picture is 15-20 seconds. The preset point is set at an interval of 2 minutes. It is used to capture the surveillance video collected by the camera to form a captured picture.

[0029] A deep learning surveillance video occlusion recognition method comprises the following steps:

[0030] 1) The cameras are numbered and entered into the image preprocessing platform and the image analysis platform. The surveillance videos and captured pictures within 7 days after the camera is installed are all saved in the memory. The camera can be manually controlled to capture multiple pictures within 7 days, which are used as the material for the training atlas and the deep learning model. The deep learning model uses the pictures taken by the camera in the early stage as the training atlas. The training atlas is trained by the PPYOLO v2 software. The initial model of the software is trained using the COCO data set to obtain a deep learning model of the pictures taken by the camera.

[0031] 2) The real-time monitoring video of the camera is transmitted to the image processing server through the network module, forming a short video of two minutes. The video screenshot software captures the short video into a picture of two minutes, and the short video directly enters the memory.

[0032] 3) The captured image is transmitted to the image processing server through the network module and enters the image preprocessing platform.

[0033] 4) The image preprocessing platform compares the captured images in chronological order with the images in the training atlas. If the similarity between the captured images and the images in the training atlas reaches 100%, the captured images are deleted into a storage box, which is automatically deleted after 7 days. If the similarity between the captured images and the images in the training atlas is less than 100%, the captured images are stored in the memory.

[0034] The similarity between the captured image and the image in the training set is matched using the directly generated original histogram. The histograms of the two images, HistA and HistB, are calculated respectively, and then the normalized correlation coefficient of the two histograms is calculated. The normalized correlation coefficient includes the Bhattacharyya distance and the histogram intersection distance.

[0035] Below is a Matlab Demo and experimental results of image similarity calculation based on histogram distance:

[0036] M = imread('1.jpg');

[0037] N = imread('2.jpg');

[0038] I = rgb2gray(M);

[0039] J = rgb2gray(N);

[0040] [Count1,x] = imhist(I);

[0041] [Count2,x] = imhist(J);

[0042] Sum1=sum(Count1); Sum2=sum(Count2);

[0043] Sumup=sqrt(Count1.*Count2);

[0044] SumDown = sqrt(Sum1 * Sum2);

[0045] Sumup = sum(Sumup);

[0046] figure(1);

[0047] subplot(2,2,1);imshow(I);

[0048] subplot(2,2,2);imshow(J);

[0049] subplot(2,2,3);imhist(I);

[0050] subplot(2,2,4);imhist(J);

[0051] HistDist=1-sqrt(1-Sumup / SumDown).

[0052] 5) The image analysis platform analyzes the images in the memory to determine the occlusion rate. The captured images in the memory are compared with the trained deep learning model to determine the position and color of the difference area in the captured image. If the difference area is black and the proportion of the black area in the image exceeds the set threshold of 10%, it means that close-range occlusion occurs. If it is other colors, the position and color of the difference area on the captured image are determined;

[0053] 6) After the intercepted image in the memory is quantized, a histogram is calculated in the RGB color space. The pixel value ratios of different areas in the intercepted image are calculated through the histogram, and compared with the histogram of the image in the training set to determine the position of the different areas in the intercepted image;

[0054] RGB color space histogram calculation Figure 1It is generally composed of three channels: R, G, and B. Each channel consists of 8 bits, with a maximum of 255. If the histogram is constructed directly according to each different value of the three channels, the amount of data will be very large, which is 256*256*256=256 cubed bins. For simplicity, 8 bins are set for each channel. In this way, the maximum value of each channel is 256 / 8=32, that is, each channel is divided into 8 bins, and each bin stores 32 numbers, 0-31, 32-63, 64-127,...224-255. The RGB color space is converted into one-dimensional 8*8*8=512 bins in total.

[0055] R=image[(y*W+x)*3]>>R_SHIFT;

[0056] G=image[(y*W+x)*3+1]>>G_SHIFT;

[0057] B=image[(y*W+x)*3+2]>>B_SHIFT;

[0058] The above R_SHIFT, G_SHIFT, B_SHIFT macros define 5, right shift 5 bits, and each R, G, B value is divided by 32 and mapped to the corresponding 8 bins. 0-31 is mapped to bins1, 32-63 is mapped to bins2... 224-255 is mapped to bins8.

[0059] 7) If the captured image and the image in the training atlas have the same position but different colors, the land use status in the monitored area has changed;

[0060] 8) If the position of the different area of ​​the captured image is different from that of the image in the training set, the color is not black, and the proportion of the pixel value of the different area exceeds the set threshold of 10%, then occlusion occurs due to other colored objects.

[0061] The present invention is not limited to the above-mentioned implementation modes, and anyone should be aware of the structural changes made under the enlightenment of the present invention, and all those having the same or similar technical solutions as the present invention fall within the protection scope of the present invention.

[0062] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.

Claims

1. A deep learning surveillance video occlusion recognition system, characterized in that: It includes an image acquisition device, an image processing server and an image analysis platform. The image acquisition device is provided with a plurality of cameras. The cameras are connected to a network module and are used to transmit real-time monitoring videos and intercepted pictures to the image processing server. An image preprocessing platform is installed on the image processing server. The image preprocessing platform compares, deletes and classifies and stores the captured intercepted pictures. An image analysis platform is also installed on the image processing server to analyze and determine the occlusion rate of the intercepted pictures after image preprocessing. The image processing server is equipped with a memory for storing the surveillance videos and captured pictures collected by the camera; the image processing server is also equipped with video capture software; The recognition method adopted by the deep learning surveillance video occlusion recognition system includes the following steps: 1) The cameras are numbered and entered into the image preprocessing platform and the image analysis platform. The surveillance videos and captured images within 7 days after the camera is installed are all saved in the memory and used as materials for training the atlas and training the deep learning model. 2) The real-time monitoring video of the camera is transmitted to the image processing server through the network module to form a short video of two minutes. The video screenshot software captures the short video into a picture of two minutes, and the short video is directly stored in the memory; 3) The captured image is transmitted to the image processing server through the network module and enters the image preprocessing platform; 4) The image preprocessing platform compares the captured images in chronological order with the images in the training atlas. If the similarity between the captured images and the images in the training atlas reaches 100%, the captured images are deleted into the storage box, which is automatically deleted after 7 days. If the similarity between the captured images and the images in the training atlas is less than 100%, the captured images are stored in the memory. 5) The image analysis platform analyzes the images in the memory to determine the occlusion rate. The captured images in the memory are compared with the trained deep learning model to determine the position and color of the difference area in the captured image. If the difference area is black, and the proportion of the black area in the captured image exceeds the set threshold of 10%, it means that close-range occlusion has occurred. If it is other colors, the position and color of the difference area on the captured image are determined; 6) After the intercepted image in the memory is quantized, a histogram is calculated in the RGB color space. The pixel value ratios of different areas in the intercepted image are calculated through the histogram, and compared with the histogram of the image in the training set to determine the position of the different areas in the intercepted image; 7) If the captured image and the image in the training atlas have the same position but different colors, the land use status in the monitored area has changed; 8) If the position of the different area of ​​the captured image is different from that of the image in the training set, the color is not black, and the proportion of the pixel value of the different area exceeds the set threshold of 10%, then occlusion occurs due to other colored objects.

2. The deep learning surveillance video occlusion recognition system according to claim 1 is characterized in that: The storage time of the video files and picture files is set to 1 month in the memory.

3. The deep learning surveillance video occlusion recognition system according to claim 1 is characterized in that: The video capture software is provided with preset points, and a picture is captured at each preset point. The time for capturing a picture is 15-20 seconds, and the preset point setting interval is 2 minutes. It is used to capture the monitoring video collected by the camera to form a captured picture.

4. The deep learning surveillance video occlusion recognition system according to claim 1 is characterized in that: The deep learning model in step 1) uses the pictures taken by the camera in the early stage as the training atlas, and the training atlas is trained by PPYOLO v2 software. The initial model of the software is trained using the COCO data set to obtain a deep learning model of the pictures taken by the camera.

5. The deep learning surveillance video occlusion recognition system according to claim 1 is characterized in that: The similarity between the captured image in step 4) and the image in the training atlas is matched using the directly generated original histogram, and the histograms of the two images, HistA and HistB, are calculated respectively, and then the normalized correlation coefficient of the two histograms is calculated, and the normalized correlation coefficient includes the Bhattacharyya distance and the histogram intersection distance.

6. The deep learning surveillance video occlusion recognition system according to claim 1 is characterized in that: The RGB color space calculation histogram in the step 6) is composed of three channels, R, G, and B. Each channel is composed of 8 bits, with a maximum of 255. If the histogram is constructed directly according to each different value of the three channels, the amount of data is very large, which is 256*256*256=256 cubed bins. For simplicity, 8 bins are set for each channel. In this way, the maximum value of each channel is 256 / 8=32, that is, each channel is divided into 8 bins, and each bin stores 32 numbers, 0-31, 32-63, 64-127, ... 224-255. The RGB color space is converted into a one-dimensional total of 8*8*8=512 bins.

Images