Unmanned aerial vehicle-based night patrol method and device, and electronic device

Abstract

The application provides a kind of night patrol method, device and electronic equipment based on unmanned aerial vehicle, the method includes the same block method with preset reference base map to block visible light image, the block of visible light image is compared with the block of preset reference base map one by one, determine the block of first block image as abnormal block image;Abnormal block image is projected to infrared image, determine the abnormal area on infrared image;Calculate the gray level co-occurrence matrix of the abnormal area of infrared image, whether there is interference source is calculated according to the gray level co-occurrence matrix;Using frame difference method, detect whether there is moving object in the abnormal area of infrared image;According to the detection result of interference source and the detection result of frame difference method, determine whether there is abnormal personnel.Can avoid the problem that abnormal personnel cannot be accurately detected due to low visibility and low brightness, achieve the purpose of accurate detection at night, effectively improve the accuracy.

Description

Technical Field

[0001] This invention relates to the field of environmental monitoring technology, and in particular to a method, apparatus, and electronic device for nighttime patrol based on unmanned aerial vehicles (UAVs). Background Technology

[0002] Currently, nighttime patrols are required in various scenarios such as public security management, traffic inspection, and anti-smuggling. Most nighttime patrols currently rely on single methods such as fixed cameras to record on-site footage, using the naked eye or simple image recognition technology to identify suspicious individuals. However, due to the low brightness and visibility at night, both visual and image recognition methods are prone to errors and cannot accurately identify suspicious individuals. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, device, and electronic device for night patrol based on drones to address the aforementioned technical problems.

[0004] Night patrol methods, devices, and electronic equipment based on drones, including the following patrol methods:

[0005] Collect visible light and infrared images of each waypoint along the preset patrol route;

[0006] The visible light image and the preset reference base image are divided into blocks using the same block division method. The blocks of the visible light image are compared with the blocks of the preset reference base image one by one. When the difference between the blocks of the visible light image and the blocks of the preset reference base image is greater than a preset difference threshold, the blocks of the visible light image are determined to be abnormal blocks.

[0007] The abnormal segmented image is projected onto the infrared image of the same waypoint, and the region corresponding to the abnormal segmented image on the infrared image is determined to be an abnormal region;

[0008] Calculate the gray-level co-occurrence matrix of the abnormal region in the infrared image, calculate the contrast energy information of the abnormal region in the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information;

[0009] The frame difference method is used to detect whether there are moving objects in the abnormal areas of the infrared image;

[0010] Based on the detection results of the interference source and the detection results of the frame difference method, it is determined whether there are any abnormal personnel.

[0011] In one embodiment, the step of dividing the visible light image and the preset reference base image into blocks using the same block division method, comparing each block of the visible light image with the block of the preset reference base image, and determining that a block of the visible light image is an abnormal block image when the difference between the block of the visible light image and the block of the preset reference base image is greater than a preset difference threshold includes:

[0012] The visible light image and the preset reference base map are divided into blocks using the same block division method to obtain multiple first block images of the visible light image and multiple second block images of the preset reference base map;

[0013] Calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively. Based on the Hu moment vector of each first block image and the Hu moment vector of each second block image, calculate the vector distance between the corresponding first block images and second block images.

[0014] When the vector distance between the first block image and the second block image that have a corresponding relationship is greater than a preset distance threshold, the first block image is determined to be an abnormal block image.

[0015] In one embodiment, the step of dividing the visible light image and the preset reference base map into blocks using the same block division method to obtain a plurality of first block images of the visible light image and a plurality of second block images of the preset reference base map includes:

[0016] An image edge detection algorithm is used to extract the contour information of the visible light image and the preset reference base image to obtain a first contour image of the visible light image and a second contour image of the preset reference base image;

[0017] The first contour image of the visible light image and the second contour image of the preset reference base image are divided into blocks using the same block division method to obtain multiple first block images of the visible light image and multiple second block images of the preset reference base image;

[0018] The steps of calculating the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively include:

[0019] Based on the contour information of the first contour image and the contour information of the second contour image, calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image.

[0020] In one embodiment, the step of using an image edge detection algorithm to extract the contour information of the visible light image and the preset reference base image to obtain a first contour image of the visible light image and a second contour image of the preset reference base image includes:

[0021] The visible light image is enhanced using an image enhancement algorithm to obtain the enhanced visible light image.

[0022] An image edge detection algorithm is used to extract the contour information of the enhanced visible light image and the preset reference base image to obtain a first contour image of the visible light image and a second contour image of the preset reference base image.

[0023] In one embodiment, the step of calculating the gray-level co-occurrence matrix of the abnormal regions in the infrared image includes:

[0024] Histogram equalization is performed on the abnormal regions of the infrared image to improve the contrast of the abnormal regions.

[0025] Calculate the gray-level co-occurrence matrix of the abnormal regions in the infrared image after histogram equalization.

[0026] In one embodiment, the step of detecting whether an interference source exists in the abnormal region based on the contrast energy information includes:

[0027] The contrast energy information is input into a preset SVM classifier, and the preset SVM classifier is used to detect whether there is an interference source in the abnormal region.

[0028] In one embodiment, the step of acquiring visible light and infrared images of each waypoint along a preset patrol route includes:

[0029] A preset patrol path is obtained, and the drone is controlled to fly according to the preset patrol path. The image sensor on the drone is controlled to collect visible light and infrared images of each waypoint on the preset patrol path. The preset patrol path records the flight sequence of each waypoint and the shooting posture of the drone at each waypoint.

[0030] A drone-based night patrol device includes:

[0031] The image acquisition module is used to acquire visible light and infrared images of each waypoint along the preset patrol route;

[0032] The abnormal segmentation determination module is used to segment the visible light image and the preset reference base image into blocks using the same segmentation method, compare the blocks of the visible light image with the blocks of the preset reference base image one by one, and determine that the blocks of the visible light image are abnormal segmented images when the difference between the blocks of the visible light image and the blocks of the preset reference base image is greater than a preset difference threshold.

[0033] An infrared anomaly region determination module is used to project the anomaly block image onto the infrared image of the same waypoint, and determine the region corresponding to the anomaly block image on the infrared image as an anomaly region;

[0034] An interference source detection module is used to calculate the gray-level co-occurrence matrix of the abnormal region of the infrared image, calculate the contrast energy information of the abnormal region of the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information.

[0035] The moving object detection module is used to detect whether there are moving objects in abnormal areas of the infrared image using the frame difference method.

[0036] The abnormal personnel detection module is used to determine whether there are abnormal personnel based on the detection results of the interference source and the detection results of the frame difference method.

[0037] An electronic device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to perform the following steps:

[0038] Collect visible light and infrared images of each waypoint along the preset patrol route;

[0039] The visible light image and the preset reference base image are divided into blocks using the same block division method. The blocks of the visible light image are compared with the blocks of the preset reference base image one by one. When the difference between the blocks of the visible light image and the blocks of the preset reference base image is greater than a preset difference threshold, the blocks of the visible light image are determined to be abnormal blocks.

[0040] The abnormal segmented image is projected onto the infrared image of the same waypoint, and the region corresponding to the abnormal segmented image on the infrared image is determined to be an abnormal region;

[0041] Calculate the gray-level co-occurrence matrix of the abnormal region in the infrared image, calculate the contrast energy information of the abnormal region in the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information;

[0042] The frame difference method is used to detect whether there are moving objects in the abnormal areas of the infrared image;

[0043] Based on the detection results of the interference source and the detection results of the frame difference method, it is determined whether there are any abnormal personnel.

[0044] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0045] Collect visible light and infrared images of each waypoint along the preset patrol route;

[0046] The visible light image and the preset reference base image are divided into blocks using the same block division method. The blocks of the visible light image are compared with the blocks of the preset reference base image one by one. When the difference between the blocks of the visible light image and the blocks of the preset reference base image is greater than a preset difference threshold, the blocks of the visible light image are determined to be abnormal blocks.

[0047] The abnormal segmented image is projected onto the infrared image of the same waypoint, and the region corresponding to the abnormal segmented image on the infrared image is determined to be an abnormal region;

[0048] Calculate the gray-level co-occurrence matrix of the abnormal region in the infrared image, calculate the contrast energy information of the abnormal region in the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information;

[0049] The frame difference method is used to detect whether there are moving objects in the abnormal areas of the infrared image;

[0050] Based on the detection results of the interference source and the detection results of the frame difference method, it is determined whether there are any abnormal personnel.

[0051] The aforementioned drone-based night patrol method, device, and electronic equipment collect visible light and infrared images at each waypoint along the patrol route. The visible light images are compared block by block with a preset reference map to detect abnormal blocks, thereby identifying abnormal areas in the infrared images. These abnormal areas are then used to detect the presence of interference sources and moving objects. This avoids the problem of inaccurate detection of abnormal personnel due to low visibility and brightness, achieving the goal of accurate detection at night and effectively improving accuracy. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a night patrol method based on unmanned aerial vehicles (UAVs) in one embodiment.

[0053] Figure 2 This is a flowchart illustrating some steps of a drone-based night patrol method in one embodiment;

[0054] Figure 3 This is a flowchart illustrating some steps of a drone-based night patrol method in one embodiment;

[0055] Figure 4 This is a flowchart illustrating some steps of a drone-based night patrol method in one embodiment;

[0056] Figure 5 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Example 1

[0058] In this embodiment, please refer to Figure 1 This paper provides a method for nighttime patrol based on drones, which includes:

[0059] Step 110: Collect visible light and infrared images of each waypoint along the preset patrol route;

[0060] In this embodiment, visible light and infrared images are acquired using image sensors on the drone. The image sensors include a visible light camera and an infrared camera, and the patrol images include both visible light and infrared images. In one embodiment, the image sensor includes a dual-light camera.

[0061] In this embodiment, the step of acquiring visible light and infrared images of each waypoint along the preset patrol route includes:

[0062] A preset patrol path is obtained, and the drone is controlled to fly according to the preset patrol path. The image sensor on the drone is controlled to collect visible light and infrared images of each waypoint on the preset patrol path. The preset patrol path records the flight sequence of each waypoint and the shooting posture of the drone at each waypoint.

[0063] In this embodiment, the preset patrol path is pre-determined based on the patrol area. In one embodiment, waypoints are determined according to requirements, and the patrol path is planned starting from the starting point, traversing all waypoints along the shortest path to obtain the preset patrol path. Multiple waypoints are set on this preset patrol path; these waypoints are locations where the image sensor needs to capture images at specific points. The preset patrol path not only records the UAV's flight path, altitude, and speed, but also the UAV's flight sequence to each waypoint and its shooting posture at each waypoint. This allows the UAV to capture images of the waypoints at preset angles, altitudes, and tilts according to the preset shooting posture, ensuring that the scene in the acquired visible light image matches the preset reference map.

[0064] In this embodiment, the UAV is controlled to fly according to the preset patrol path. Upon reaching each waypoint on the preset patrol path, the attitude of the UAV is controlled according to the shooting attitude of each waypoint, and the image sensor on the UAV is controlled to acquire patrol images at that waypoint. This ensures that the scene of the acquired visible light image can be matched with a preset reference base map. In this embodiment, at one waypoint, one visible light image and multiple infrared images are acquired. Specifically, a visible light image is captured vertically downward using a dual-light camera, and then several infrared images are captured at preset time intervals, thus completing the data acquisition for one waypoint.

[0065] After acquiring visible light and infrared images of waypoints, these images are numbered to ensure they correspond to a pre-defined reference map at the respective waypoint. Furthermore, both visible light and infrared images are acquired using the same shooting posture to ensure that the range and angle of the images captured at the same waypoint are identical.

[0066] Step 120: Divide the visible light image and the preset reference base image into blocks using the same block division method. Compare each block of the visible light image with the block of the preset reference base image. When the difference between the block of the visible light image and the block of the preset reference base image is greater than a preset difference threshold, determine that the block of the visible light image is an abnormal block image.

[0067] In this embodiment, the preset reference base map is an image pre-collected at waypoints and used as a reference for images collected during patrols. This preset reference base map can be taken during the day or at night, and it contains no anomalies. It is worth noting that each waypoint corresponds to one preset reference base map. In this way, after acquiring the visible light image of the waypoint, the visible light image and the preset reference map are evenly divided into multiple image blocks. For example, the visible light image and the preset reference map are divided into 9 image blocks in a 3*3 manner. Then, the divided images are compared one by one according to their corresponding positions. For example, the visible light image block with coordinates (1,1) is compared with the preset reference map block with coordinates (1,1), and the visible light image block with coordinates (2,3) is compared with the preset reference map block with coordinates (2,3). When the difference between the visible light image block and the preset reference map block is greater than the preset difference threshold, it indicates that there is an anomaly in that block. For example, if there are abnormal personnel, then the visible light image block is determined to be an abnormal block image.

[0068] In this embodiment, abnormal segmented images are identified and merged to obtain abnormal block images.

[0069] Step 130: Project the abnormal block image onto the infrared image of the same waypoint, and determine that the area corresponding to the abnormal block image on the infrared image is an abnormal area.

[0070] In this embodiment, the abnormal block image is projected onto the infrared image of the same waypoint according to the corresponding position, so that the area on the abnormal block image coincides with the corresponding area on the infrared image, and the area on the infrared image corresponding to the abnormal block image is determined to be an abnormal area.

[0071] Step 140: Calculate the gray-level co-occurrence matrix of the abnormal region of the infrared image, calculate the contrast energy information of the abnormal region of the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information.

[0072] In this embodiment, the contrast energy information includes contrast information, energy information, etc. In this embodiment, by analyzing the infrared image and calculating the gray-level co-occurrence matrix of the abnormal regions of the infrared image, the contrast energy information of the abnormal regions can be obtained. Therefore, the presence of interference sources in the abnormal regions can be detected based on the contrast energy information. Common interference sources include heat reflection, water accumulation, or animals.

[0073] Step 150: Use the frame difference method to detect whether there are moving objects in the abnormal areas of the infrared image.

[0074] In this embodiment, the frame difference method is used to analyze the abnormal regions of infrared images taken continuously at the same waypoint. The frame difference method can quickly determine whether there are moving objects in the abnormal regions by calculating the differences between consecutive frames of infrared images in the abnormal regions.

[0075] Step 160: Based on the detection results of the interference source and the detection results of the frame difference method, determine whether there are any abnormal personnel.

[0076] In this embodiment, the detection results of the interference source and the frame difference method are combined to comprehensively determine whether there are abnormal personnel. For example, if the interference source detection results detect and eliminate the interference source, and detect personnel movement, it is determined that there are abnormal personnel. Then, the waypoint position is located, the abnormal position is reported, and a warning is issued.

[0077] In the above embodiments, visible light images and infrared images are collected at each waypoint on the patrol route. The visible light images are compared with a preset reference base map block by block to detect abnormal blocks, thereby determining abnormal areas on the infrared images. The abnormal areas on the infrared images are used to detect whether there are interference sources and whether there are moving objects. This avoids the problem of not being able to accurately detect the presence of abnormal personnel due to low visibility and brightness, achieving the goal of accurate detection at night and effectively improving the accuracy rate.

[0078] In one embodiment, the step of dividing the visible light image and the preset reference base image into blocks using the same block division method, comparing each block of the visible light image with the block of the preset reference base image one by one, and determining that a block of the visible light image is an abnormal block image when the difference between the block of the visible light image and the block of the preset reference base image is greater than a preset difference threshold includes:

[0079] The visible light image and the preset reference base map are divided into blocks using the same block division method to obtain multiple first block images of the visible light image and multiple second block images of the preset reference base map;

[0080] Calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively. Based on the Hu moment vector of each first block image and the Hu moment vector of each second block image, calculate the vector distance between the corresponding first block images and second block images.

[0081] When the vector distance between the first block image and the second block image that have a corresponding relationship is greater than a preset distance threshold, the first block image is determined to be an abnormal block image.

[0082] In this embodiment, the Hu moment vector consists of a set of feature vector values ​​describing the overall shape of an image region. The Hu moment is used to quantize the contour shape of each image patch into a feature vector, while the vector distance is used to measure the similarity between two feature vectors (i.e., two shapes). The Hu moment vector is invariant to translation, scaling, and rotation. Regardless of how the position, size, or angle of an object in the image patch changes, as long as the shape itself remains unchanged, its Hu moment vector remains essentially constant. Therefore, the Hu moment vector is suitable for shape matching.

[0083] In this embodiment, the visible light image and the preset reference base image are first uniformly divided into blocks using the same segmentation method to obtain a first block image and a second block image. Then, the Hu moment vectors of each first block image and each second block image are calculated. The Hu moment vectors of the first and second block images located at the same segmentation position are compared, and the vector distance (Euclidean distance) between them is calculated, thus achieving one-to-one calculation of the vector distance between each first block image and each second block image. The smaller the vector distance between two Hu moment vectors in space, the more similar the two shapes are. When the vector distance between the Hu moment vectors of the first and second block images at the same segmentation position is greater than a preset distance threshold, it indicates a significant difference between the first and second block images at that position. Therefore, the first block image is determined to be an abnormal block image, thereby determining the location of the abnormal person in the image.

[0084] In this embodiment, after determining that the first segmented image is an abnormal segmented image, the abnormal segmented images are merged to obtain an abnormal block image. The abnormal block image is then projected onto the infrared image of the same waypoint, and the region corresponding to the abnormal segmented image on the infrared image is determined to be an abnormal region.

[0085] In one embodiment, the step of dividing the visible light image and the preset reference base map into blocks using the same block division method to obtain a plurality of first block images of the visible light image and a plurality of second block images of the preset reference base map includes:

[0086] An image edge detection algorithm is used to extract the contour information of the visible light image and the preset reference base image to obtain a first contour image of the visible light image and a second contour image of the preset reference base image;

[0087] The first contour image of the visible light image and the second contour image of the preset reference base image are divided into blocks using the same block division method to obtain multiple first block images of the visible light image and multiple second block images of the preset reference base image;

[0088] The steps of calculating the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively include:

[0089] Based on the contour information of the first contour image and the contour information of the second contour image, calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image.

[0090] In this embodiment, to accurately calculate the Hu moment vector of the image, contour information is first extracted from the image to obtain a contour image. Then, the Hu moment vector is calculated using the contour information in the contour image. It is worth noting that the extracted contour image contains less image information and less data, and can more accurately represent the main information of objects in the image. Therefore, calculating the Hu moment vector through the contour makes the calculation more efficient and accurate. In this embodiment, the image edge detection algorithm uses the Canny operator algorithm. The Canny operator is an edge detection algorithm that uses a "hysteresis threshold" strategy to find the complete and continuous contour lines of the main objects in the image while effectively suppressing noise. In other embodiments, the edge detection algorithm used to extract contour lines can also be implemented using the Sobel / Prewitt operator, Laplacian of Gaussian, or HED, CED, etc., which will not be described in detail in this embodiment.

[0091] In one embodiment, the step of extracting the contour information of the visible light image and the preset reference base image using an image edge detection algorithm to obtain a first contour image of the visible light image and a second contour image of the preset reference base image includes:

[0092] The visible light image is enhanced using an image enhancement algorithm to obtain the enhanced visible light image.

[0093] An image edge detection algorithm is used to extract the contour information of the enhanced visible light image and the preset reference base image to obtain a first contour image of the visible light image and a second contour image of the preset reference base image.

[0094] In this embodiment, the visible light image is enhanced to accurately extract the image contours. In one embodiment, the image enhancement algorithm is the Retinex algorithm, specifically a single-scale Retinex algorithm, also known as the SSR algorithm. In this embodiment, the single-scale Retinex algorithm can separate the illumination component and the reflection component of the object itself in the image. Through processing, it can effectively improve the overall visual effect of nighttime or unevenly lit images, enhance details and contrast, and lay a good foundation for subsequent contour extraction. In other embodiments, the image enhancement algorithm used for image enhancement can also employ histogram equalization (CLAHE), homomorphic filtering / wavelet transform, etc., which will not be described in detail in this embodiment.

[0095] In one embodiment, the step of calculating the gray-level co-occurrence matrix of the abnormal regions in the infrared image includes:

[0096] Histogram equalization is performed on the abnormal regions of the infrared image to improve the contrast of the abnormal regions.

[0097] Calculate the gray-level co-occurrence matrix of the abnormal regions in the infrared image after histogram equalization.

[0098] In this embodiment, infrared images can reflect temperature information in the image. Therefore, the average temperature and maximum temperature of abnormal areas in the infrared image are statistically analyzed. At the same time, histogram equalization processing is performed on the abnormal areas to improve the contrast of the abnormal areas in the infrared image and reduce the impact of low thermal contrast on the detection effect at night. In this way, the improved contrast is beneficial to the subsequent calculation of the gray-level co-occurrence matrix of the abnormal areas, making the subsequent calculation results more accurate.

[0099] In one embodiment, the step of detecting whether an interference source exists in the abnormal region based on the contrast energy information includes:

[0100] The contrast energy information is input into a preset SVM classifier, and the preset SVM classifier is used to detect whether there is an interference source in the abnormal region.

[0101] In this embodiment, contrast information and energy information are input into a preset SVM classifier. The preset SVM classifier is used to detect whether there is an interference source in the abnormal region. When there is an interference source in the abnormal region, the interference source is excluded. Example 2

[0102] In this embodiment, the preliminary steps of the drone-based night patrol method include: Figure 2As shown, the patrol area is determined, and patrol points (waypoints) are selected. Patrol routes are planned based on these waypoints, and images of the waypoints are taken at night as reference base maps. Specifically, in the preliminary preparation work, the monitoring and patrol area is first selected based on the jurisdiction and the performance of the drone. Then, locations with frequent past abnormal behavior, or locations analyzed based on the experience of patrol personnel, are selected as the initial patrol waypoints for the drone, for example, if the abnormal behavior is smuggling. After recording the GPS coordinates of the waypoints, the shortest path from the drone's takeoff point to traverse all waypoints is set, and this path is used as the drone's patrol path. After setting the path, preliminary data collection is carried out. The drone takes visible light images of each waypoint at night, with the camera pointing vertically downwards to reduce the difficulty of identification. The flight altitude is controlled at approximately 50 meters to ensure that the image area is within the effective range of the drone's infrared imaging equipment. After taking images of each waypoint, the corresponding photo numbers are recorded as reference base maps for subsequent comparisons.

[0103] like Figure 3 As shown, after the drone reaches the waypoint, it adjusts its attitude to maintain the previously set flight path, adjusts the gimbal, and takes a visible light photo vertically downwards. Keeping the gimbal stationary, it then takes a series of infrared photos at equal time intervals. This process represents data collection during actual patrol. Unlike the initial preparation, during patrol, the drone needs to collect both visible light and infrared images simultaneously. The specific procedure is as follows: when the drone reaches the corresponding waypoint and maintains the same attitude as the flight path, it uses a dual-light camera to take a visible light photo vertically downwards. Subsequently, it takes several infrared photos at equal time intervals, thus completing the data collection for one waypoint.

[0104] like Figure 4 As shown, after acquiring visible light and infrared images, data processing is performed. The specific process is as follows: Data processing: First, the reference base map and the visible light photos taken during patrol are processed. Specifically, the Retinex algorithm (specifically the single-scale Retinex algorithm, i.e., the SSR algorithm) is used to process the above images in the same way to obtain enhanced visible light images. This step can effectively improve the information content of visible light images taken at night. Then, the Canny operator is used to extract the contour information of the enhanced image. After obtaining the contour maps of the base map and the patrol visible light photos, the two images are evenly divided into blocks according to the same block division method. Then, the Hu moments of each block are calculated, and the vector distance between corresponding blocks of the base map and the patrol photos is calculated. A threshold is set, and blocks whose vector distance exceeds the threshold are marked as abnormal blocks. After obtaining the abnormal blocks with different contours, adjacent abnormal blocks are merged to obtain the processed block information.

[0105] After identifying the anomalous regions, these regions are projected onto a set of infrared images taken at the same waypoint. The average and maximum temperatures of the anomalous regions are then statistically analyzed. Simultaneously, histogram equalization is applied to these regions to improve infrared image contrast and reduce the impact of low thermal contrast at night on detection performance. The gray-level co-occurrence matrix (GLCM) of the processed infrared image regions is calculated, and contrast and energy information are derived from this matrix. This information is then fed into an SVM classifier to eliminate the influence of common interference sources. Furthermore, frame difference analysis is used to detect anomalous regions within the infrared image set to check for moving objects. Finally, based on the results of the SVM classifier and the detection of moving objects, the anomalous location is reported to the personnel in charge, and a warning is issued.

[0106] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps. Example 3

[0107] In this embodiment, a night patrol device based on a drone is provided, comprising:

[0108] The image acquisition module is used to acquire visible light and infrared images of each waypoint along the preset patrol route;

[0109] The abnormal segmentation determination module is used to segment the visible light image and the preset reference base image into blocks using the same segmentation method, compare the blocks of the visible light image with the blocks of the preset reference base image one by one, and determine that the blocks of the visible light image are abnormal segmented images when the difference between the blocks of the visible light image and the blocks of the preset reference base image is greater than a preset difference threshold.

[0110] An infrared anomaly region determination module is used to project the anomaly block image onto the infrared image of the same waypoint, and determine the region corresponding to the anomaly block image on the infrared image as an anomaly region;

[0111] An interference source detection module is used to calculate the gray-level co-occurrence matrix of the abnormal region of the infrared image, calculate the contrast energy information of the abnormal region of the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information.

[0112] The moving object detection module is used to detect whether there are moving objects in abnormal areas of the infrared image using the frame difference method.

[0113] The abnormal personnel detection module is used to determine whether there are abnormal personnel based on the detection results of the interference source and the detection results of the frame difference method.

[0114] In one embodiment, the abnormal block determination module includes:

[0115] The segmented image acquisition unit is used to segment the visible light image and the preset reference base map into blocks according to the same segmentation method to obtain multiple first segmented images of the visible light image and multiple second segmented images of the preset reference base map;

[0116] The vector distance calculation unit is used to calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively, and calculate the vector distance between the first block image and the second block image that have a corresponding relationship based on the Hu moment vector of each first block image and the Hu moment vector of each second block image.

[0117] The abnormal segmentation determination unit is used to determine that the first segmentation image is an abnormal segmentation image when the vector distance between the first segmentation image and the second segmentation image, which have a corresponding relationship, is greater than a preset distance threshold.

[0118] In one embodiment, the segmented image acquisition unit includes:

[0119] The contour extraction subunit is used to extract the contour information of the visible light image and the preset reference base image using an image edge detection algorithm, so as to obtain a first contour image of the visible light image and a second contour image of the preset reference base image;

[0120] The segmentation subunit is used to segment the first contour image of the visible light image and the second contour image of the preset reference base image into blocks using the same segmentation method, so as to obtain multiple first segmented images of the visible light image and multiple second segmented images of the preset reference base image.

[0121] The vector distance calculation unit is further configured to calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image based on the contour information of the first contour image and the contour information of the second contour image.

[0122] The contour extraction subunit is further configured to perform image enhancement on the visible light image using an image enhancement algorithm to obtain the enhanced visible light image; and to extract the contour information of the enhanced visible light image and the preset reference base image using an image edge detection algorithm to obtain a first contour image of the visible light image and a second contour image of the preset reference base image.

[0123] In one embodiment, the interference source detection module includes:

[0124] The equalization processing unit is used to perform histogram equalization processing on the abnormal regions of the infrared image to improve the contrast of the abnormal regions of the infrared image.

[0125] The gray-level co-occurrence matrix calculation unit is used to calculate the gray-level co-occurrence matrix of abnormal regions in the infrared image after histogram equalization processing.

[0126] In one embodiment, the interference source detection module is further configured to input the contrast energy information into a preset SVM classifier, and use the preset SVM classifier to detect whether there is an interference source in the abnormal region.

[0127] In one embodiment, the image acquisition module is further configured to acquire a preset patrol path, control the drone to fly according to the preset patrol path, and control the image sensor on the drone to acquire visible light and infrared images of each waypoint on the preset patrol path, wherein the preset patrol path records the flight sequence of each waypoint and the shooting posture of the drone at each waypoint.

[0128] Specific limitations regarding the drone-based night patrol device can be found in the limitations of the drone-based night patrol method described above, and will not be repeated here. Each unit in the aforementioned drone-based night patrol device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in the electronic device, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each unit. Example 4

[0129] In this embodiment, an electronic device is provided. Its internal structure diagram can be shown as follows: Figure 5As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs, and also contains a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with other electronic devices that have deployed application software. When the computer program is executed by the processor, it implements a drone-based nighttime patrol method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0130] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0131] In one embodiment, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the drone-based night patrol method of any of the above embodiments. Example 5

[0132] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps in the drone-based night patrol method of any of the above embodiments.

[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for nighttime patrol based on unmanned aerial vehicles (UAVs), characterized in that, include: Collect visible light and infrared images of each waypoint along the preset patrol route; The visible light image and the preset reference base image are divided into blocks using the same block division method. The blocks of the visible light image are compared with the blocks of the preset reference base image one by one. When the difference between the blocks of the visible light image and the blocks of the preset reference base image is greater than a preset difference threshold, the blocks of the visible light image are determined to be abnormal blocks. The abnormal segmented image is projected onto the infrared image of the same waypoint, and the region corresponding to the abnormal segmented image on the infrared image is determined to be an abnormal region; Calculate the gray-level co-occurrence matrix of the abnormal region in the infrared image, calculate the contrast energy information of the abnormal region in the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information; The frame difference method is used to detect whether there are moving objects in the abnormal areas of the infrared image; Based on the detection results of the interference source and the detection results of the frame difference method, determine whether there are any abnormal personnel; The step of dividing the visible light image and the preset reference base image into blocks using the same block division method, comparing each block of the visible light image with the block of the preset reference base image, and determining that a block of the visible light image is an abnormal block image when the difference between the block of the visible light image and the block of the preset reference base image is greater than a preset difference threshold includes: The visible light image and the preset reference base map are divided into blocks using the same block division method to obtain multiple first block images of the visible light image and multiple second block images of the preset reference base map; Calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively. Based on the Hu moment vector of each first block image and the Hu moment vector of each second block image, calculate the vector distance between the corresponding first block images and second block images. When the vector distance between the first block image and the second block image that have a corresponding relationship is greater than a preset distance threshold, the first block image is determined to be an abnormal block image; After determining that the first segmented image is an abnormal segmented image, the abnormal segmented images are merged to obtain the abnormal block image.

2. The method according to claim 1, characterized in that, The step of dividing the visible light image and the preset reference base image into blocks using the same block division method to obtain multiple first block images of the visible light image and multiple second block images of the preset reference base image includes: An image edge detection algorithm is used to extract the contour information of the visible light image and the preset reference base image to obtain a first contour image of the visible light image and a second contour image of the preset reference base image; The first contour image of the visible light image and the second contour image of the preset reference base image are divided into blocks using the same block division method to obtain multiple first block images of the visible light image and multiple second block images of the preset reference base image; The steps of calculating the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively include: Based on the contour information of the first contour image and the contour information of the second contour image, calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image.

3. The method according to claim 2, characterized in that, The step of extracting the contour information of the visible light image and the preset reference base image using an image edge detection algorithm to obtain a first contour image of the visible light image and a second contour image of the preset reference base image includes: The visible light image is enhanced using an image enhancement algorithm to obtain the enhanced visible light image. An image edge detection algorithm is used to extract the contour information of the enhanced visible light image and the preset reference base image to obtain a first contour image of the visible light image and a second contour image of the preset reference base image.

4. The method according to claim 1, characterized in that, The step of calculating the gray-level co-occurrence matrix of the abnormal regions in the infrared image includes: Histogram equalization is performed on the abnormal regions of the infrared image to improve the contrast of the abnormal regions. Calculate the gray-level co-occurrence matrix of the abnormal regions in the infrared image after histogram equalization.

5. The method according to claim 1, characterized in that, The step of detecting whether there is an interference source in the abnormal region based on the contrast energy information includes: The contrast energy information is input into a preset SVM classifier, and the preset SVM classifier is used to detect whether there is an interference source in the abnormal region.

6. The method according to any one of claims 1-5, characterized in that, The steps of acquiring visible light and infrared images of each waypoint along the preset patrol route include: A preset patrol path is obtained, and the drone is controlled to fly according to the preset patrol path. The image sensor on the drone is controlled to collect visible light and infrared images of each waypoint on the preset patrol path. The preset patrol path records the flight sequence of each waypoint and the shooting posture of the drone at each waypoint.

7. A night patrol device based on unmanned aerial vehicles (UAVs), characterized in that, include The image acquisition module is used to acquire visible light and infrared images of each waypoint along the preset patrol route; The abnormal segmentation determination module is used to segment the visible light image and the preset reference base image into blocks using the same segmentation method, compare the blocks of the visible light image with the blocks of the preset reference base image one by one, and determine that the blocks of the visible light image are abnormal segmented images when the difference between the blocks of the visible light image and the blocks of the preset reference base image is greater than a preset difference threshold. An infrared anomaly region determination module is used to project the anomaly block image onto the infrared image of the same waypoint, and determine the region corresponding to the anomaly block image on the infrared image as an anomaly region; An interference source detection module is used to calculate the gray-level co-occurrence matrix of the abnormal region of the infrared image, calculate the contrast energy information of the abnormal region of the infrared image based on the gray-level co-occurrence matrix, and detect whether there is an interference source in the abnormal region based on the contrast energy information. The moving object detection module is used to detect whether there are moving objects in abnormal areas of the infrared image using the frame difference method. An abnormal personnel detection module is used to determine whether there are abnormal personnel based on the detection results of the interference source and the detection results of the frame difference method. The abnormal block determination module includes: The segmented image acquisition unit is used to segment the visible light image and the preset reference base map into blocks according to the same segmentation method to obtain multiple first segmented images of the visible light image and multiple second segmented images of the preset reference base map; The vector distance calculation unit is used to calculate the Hu moment vector of each first block image and the Hu moment vector of each second block image respectively, and calculate the vector distance between the first block image and the second block image that have a corresponding relationship based on the Hu moment vector of each first block image and the Hu moment vector of each second block image. The abnormal block determination unit is used to determine that the first block image is an abnormal block image when the vector distance between the first block image and the second block image that have a corresponding relationship is greater than a preset distance threshold. After determining that the first block image is an abnormal block image, the abnormal block images are merged to obtain an abnormal block image.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images