Visual monitoring method for illegal building machine based on low-altitude aircraft

Abstract

The invention discloses an illegal building machine vision monitoring method based on a low-altitude aircraft, and belongs to the technical field of unmanned aerial vehicle monitoring and computer vision, and the method comprises the steps: collecting video data of a target area through the low-altitude aircraft carrying a multispectral camera device and an IMU / BDS module; processing original video data by adopting a multi-sensor fusion dynamic image stabilization preprocessing method; the video quality is improved by using a double-stage image enhancement technology based on deep learning; recognizing illegal buildings through a detection algorithm integrating unsupervised learning and supervised learning; and outputting a detection result in combination with time sequence analysis and multi-source data verification. According to the method, the problem of poor image quality in aerial photography monitoring can be effectively solved, the recognition precision and real-time performance of illegal buildings are improved, and the manual recognition cost is reduced. Meanwhile, the low-altitude aircraft is used for data acquisition, a wider monitoring area can be covered, and effective monitoring and management of illegal buildings are realized.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) monitoring and computer vision technology, and in particular to a machine vision monitoring method for illegal buildings based on a low-altitude aircraft. Background Art

[0002] Currently, monitoring illegal urban structures primarily relies on a combination of manual inspections and regular aerial photography, which presents significant efficiency bottlenecks and technical deficiencies. According to statistics, traditional manual inspections require an average of approximately three working days per square kilometer, and the rate of missed detections is high. Existing drone monitoring systems generally use a single visible light sensor, which dramatically degrades image quality under complex lighting conditions (such as backlight and haze), with measured PSNR values ​​often falling below 25dB. More critically, existing algorithms have a recognition rate of less than 60% for small illegal structures (such as rooftop additions and enclosed balconies). These technical shortcomings have directly contributed to the national average annual rate of new illegal building detections hovering around 68%.

[0003] At the technical implementation level, existing monitoring systems face three core challenges. First, dynamic image stabilization technologies often rely on single-sensor compensation. For example, pure IMU solutions can achieve a compensation residual error of ±15 pixels in strong winds (wind speeds >10 m / s). Traditional LK optical flow-based methods have a computational complexity of O(n²), failing to meet real-time requirements. Second, there is a disconnect between image enhancement and object detection algorithms. Mainstream solutions, such as the cascaded architecture of SRCNN + Faster R-CNN, lead to cumulative error amplification, resulting in a 12.6 percentage point drop in measured mean average approach (MAP). More importantly, existing systems lack time series analysis capabilities, resulting in an average lag of 17.6 days in identifying illegal structures during construction. A 2023 IEEE Journal of Remote Sensing study indicated that the current state-of-the-art multi-temporal detection algorithm, A^3Net, achieved a change detection F1-score of only 0.71 over a three-month time span. These technical shortcomings severely hinder the timely management of illegal structures, resulting in significant economic losses nationwide annually due to illegal construction.

[0004] Regarding machine vision methods for identifying illegal buildings using low-altitude aircraft, scholar Chen Wanwei, in his paper "Research on Building Illegal Information Collection Based on UAV Photogrammetry," proposed improving the efficiency of illegal building detection by integrating modern remote sensing imagery with two-dimensional and three-dimensional model data. While the two-dimensional and three-dimensional linkages yield low-error vector data for illegal buildings, the area of ​​buildings collected from two-dimensional images is significantly affected by eaves. Even with three-dimensional data, the influence of eaves on area measurement cannot be eliminated. Inventor Zeng Cheng et al. achieved automated identification in their patent "A Method for Automatic Identification of Illegal Buildings Based on a Fully Convolutional Neural Network," but the multi-step process (including feature point detection, homography transformation, change detection, and deep learning classification) can result in a long processing time.

[0005] Therefore, a machine vision method for illegal buildings based on low-altitude aircraft is needed, which can significantly solve the problems of low image signal-to-noise ratio, insufficient labeled samples, and high delay in timely processing in aerial monitoring. Summary of the Invention

[0006] The main purpose of the present invention is to provide a machine vision monitoring method for illegal buildings based on a low-altitude aircraft, which can effectively solve the problems mentioned in the background technology.

[0007] To achieve the above object, the technical solution adopted by the present invention is: A method for machine vision monitoring of illegal buildings based on a low-altitude aircraft comprises the following steps: S1: Video data of the target area is collected by a low-altitude aircraft equipped with a multispectral camera and an IMU / BDS module. The multispectral camera includes at least visible light and near-infrared band sensors. As a preferred technical solution of the present invention, in S1, the multispectral imaging system adopts a five-lens synchronous acquisition architecture, covering the following bands: the visible light band (450-650 nanometers) has a resolution of 3840×2160 pixels and a frame rate of 30 frames per second; the near-infrared band (700-900 nanometers) has a resolution of 1280×720 pixels and a frame rate of 30 frames per second. In addition, an optional polarized light channel is provided to eliminate interference from specular reflections. S11; As a preferred technical solution of the present invention: In S1, the IMU module includes a six-axis inertial measurement unit, the gyroscope zero bias stability is <0.5° / h, and the sampling frequency is 200Hz; S12: As a preferred technical solution of the present invention: In S1, the BDS module features high-precision dual-frequency RTK, supports SBAS / QZSS enhanced signals, has a horizontal positioning accuracy of better than 2 cm, a vertical positioning accuracy of better than 5 cm, and an update frequency of up to 10 times per second, ensuring accurate positioning and navigation of the aircraft in complex environments; S13: As a preferred technical solution of the present invention: In S1, the flight platform requirements for the low-altitude aircraft are as follows: the flight altitude should be controlled within the range of 150 meters to 500 meters, and this altitude range can be dynamically adjusted according to the specific requirements of ground resolution in actual missions; the flight speed must not exceed 12 meters per second to ensure that the longitudinal and lateral overlap rates of aerial images are both higher than 80%; at the same time, the onboard gimbal system must have a jitter compensation accuracy of ±0.01 degrees to ensure the stability of high-precision imaging. The coordinated optimization of these parameters effectively balances the requirements of operational efficiency and data acquisition quality. S2: Process the raw video data using a multi-sensor fusion dynamic image stabilization preprocessing method, including motion compensation based on IMU data and electronic image stabilization combined with SIFT feature point matching; S21: As a preferred technical solution of the present invention: In S2, the dynamic image stabilization preprocessing includes: motion compensation based on IMU data, using six-axis IMU (gyroscope + accelerometer) data to establish a drone motion model and calculate inter-frame attitude changes: ; in, is the angular velocity, is the gyro bias, For noise.

[0008] Motion compensation uses the extended Kalman filter (EKF) to fuse IMU data with BDS position information, estimate camera position changes, and perform affine transformation compensation to eliminate large-scale motion blur.

[0009] S22: As a preferred technical solution in the present invention: in S2, the electronic image stabilization of the SIFT feature point matching uses accelerated robust features to extract key points, which is adapted to low-texture areas (such as roofs and roads) of low-altitude aerial images.

[0010] S3: Process the video frames using a two-stage image enhancement technique based on deep learning. The first stage uses a U-Net network for deblurring, and the second stage uses an SRGAN network to achieve super-resolution reconstruction. S31: As a preferred technical solution of the present invention: in S3, the first stage U-Net performs deblurring processing, the network architecture adopts an improved U-Net++ structure, and adds dense skip connections between the encoder and decoder to enhance feature reuse; S32: As a preferred technical solution of the present invention: in S3, the second stage SRGAN super-resolution reconstruction, the network architecture adopts the improved ESRGAN, the generator is based on RRDB (residual dense block), and the discriminator adopts spectral normalization PatchGAN; S4: Identify illegal buildings by integrating unsupervised and supervised learning detection algorithms, including: constructing an initial feature extractor using self-supervised pre-training and supervised fine-tuning based on the Faster R-CNN framework; S41: As a preferred technical solution of the present invention: In S4, the self-supervised pre-training framework adopts MoCo v3, constructs a pre-training task through visual transformation, applies random cropping, rotation (±30°), and color jitter to the same image to generate positive sample pairs, uses ResNet-50 as the encoder, and dynamically updates the momentum queue; optimizes the feature extractor, and uses InfoNCE Loss as the loss function to maximize the similarity of positive samples; S42: As a preferred technical solution of the present invention: In S4, the supervised fine-tuning described above can improve the network by fusing multi-scale features through feature pyramid (FPN) to adapt to illegal buildings of different sizes; it can also use a decoupled detection head to separate the classification and regression branches to reduce task conflicts; its difficult sample mining is to sort the negative samples by prediction confidence, and retain the top 30% as difficult samples; the ratio of positive and negative samples is controlled at 1:3 to avoid class imbalance; S5: Combine time series analysis and multi-source data verification to output detection results, including: using optical flow method to associate targets across frames, and performing spatial coordinate matching verification between detection results and GIS map data; S51: As a preferred technical solution in the present invention: In the S5, in the time series analysis, the cross-frame tracking adopts the Farneback dense optical flow algorithm to calculate the pixel motion field between adjacent frames; a motion trajectory model is established for the detected illegal construction target, and the position is predicted based on the Kalman filter; the detection frames of N consecutive frames (default N=5) are clustered using the DBSCAN algorithm; false targets that appear instantaneously (such as flying birds, cloud shadows) are eliminated; the detection frame coordinates (pixel level) are converted to the WGS84 geographic coordinate system through a perspective projection model; compared with the building approval database of the land planning department, and unregistered illegal construction areas are marked; GIS spatial matching converts the detection frame coordinates (pixel level) to the WGS84 geographic coordinate system through a perspective projection model, and compares with the building approval database of the land planning department, and marks unregistered illegal construction areas.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The present invention collects data through a low-altitude aircraft equipped with a multispectral camera and an IMU / BDS module, and adopts dynamic image stabilization preprocessing of multi-sensor fusion (Kalman filtering, SIFT feature point matching), two-stage image enhancement technology based on U-Net and SRGAN, a detection algorithm that integrates MoCov3 self-supervised pre-training and FasterR-CNN supervised fine-tuning, and optical flow cross-frame tracking combined with time series analysis and multi-source verification of GIS data and building approval database. Its beneficial effects are to effectively solve the problems of low signal-to-noise ratio, insufficient samples and high processing delay of aerial images, improve the accuracy and real-time performance of illegal building identification, reduce labor costs, expand monitoring coverage, and provide quantitative evidence through three-dimensional point cloud reconstruction to achieve efficient monitoring and precise management of illegal buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0013] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0014] Reference Figure 1 A machine vision method for detecting illegal buildings based on a low-altitude aircraft includes the following steps: S1. Collect video data of the target area using a low-altitude aircraft equipped with a multispectral camera and an IMU / BDS module. The multispectral camera includes at least visible light and near-infrared band sensors. S2. Process the raw video data using a multi-sensor fusion dynamic image stabilization preprocessing method, including motion compensation based on IMU data and electronic image stabilization combined with SIFT feature point matching; S3. Process the video frames using a two-stage image enhancement technique based on deep learning. The first stage uses a U-Net network for deblurring, and the second stage uses an SRGAN network for super-resolution reconstruction. S4. Identify illegal buildings by integrating unsupervised and supervised learning detection algorithms, including: constructing an initial feature extractor using self-supervised pre-training and performing supervised fine-tuning based on the Faster R-CNN framework; S5. Combine time series analysis and multi-source data verification to output detection results, including: using optical flow method to associate targets across frames, and verifying spatial coordinate matching between detection results and GIS map data.

[0015] In step S1, video data of the target area is collected by a low-altitude aircraft equipped with a multispectral camera and an IMU / BDS module, wherein the multispectral camera includes at least visible light and near-infrared band sensors; the IMU module includes a six-axis inertial measurement unit, a gyroscope zero bias stability of <0.5° / h, and a sampling frequency of 200Hz; The multispectral imaging system uses a five-lens synchronous acquisition architecture, covering the following bands: visible light band (450-650 nm) with a resolution of 3840 × 2160 pixels and a frame rate of 30 frames per second; near-infrared band (700-900 nm) with a resolution of 1280 × 720 pixels and a frame rate of 30 frames per second. In addition, an optional polarization channel is provided to eliminate interference from specular reflections. In step S2, the original video data is processed using a dynamic image stabilization preprocessing method of multi-sensor fusion. The dynamic image stabilization preprocessing step includes a motion compensation model: ; The weight coefficient ; Using six-axis IMU (gyroscope + accelerometer) data, we build a drone motion model and calculate the inter-frame attitude changes: ; in, is the angular velocity, is the gyro bias, For noise; Electronic image stabilization based on SIFT feature point matching uses accelerated robust feature extraction to extract key points, adapting to low-texture areas (such as roofs and roads) in low-altitude aerial images; For example, we use a feature pyramid to construct a scale space, detect feature points at different scales, and establish correspondences between adjacent frames through feature point matching to achieve electronic image stabilization. To improve matching accuracy, we use the RANSAC algorithm to eliminate mismatched points and obtain stable matching results.

[0016] In step S3, a two-stage image enhancement technique based on deep learning is used to process the video frames, which is performed in sequence: the first stage uses a U-Net network for deblurring, and the second stage uses an SRGAN network to achieve super-resolution reconstruction; In the first stage, U-Net performs deblurring. The network architecture uses an improved U-Net++ structure, adding dense skip connections between the encoder and decoder to enhance feature reuse. Skip connections enable shallow-layer features to be directly transferred to deep layers, improving information flow efficiency and feature expression capabilities. During training, the mean squared error (MSE) is used as the loss function, and the network parameters are optimized through the back-propagation algorithm to minimize the error between the output deblurred image and the clear image. The second stage, SRGAN super-resolution reconstruction, uses a modified ESRGAN network architecture, with a generator based on RRDB and a spectrally normalized PatchGAN discriminator. By introducing a relative discriminator and perceptual loss, the realism and detail of the reconstructed image are further enhanced, ensuring that the image quality meets the high requirements of subsequent detection tasks. During the training phase, a combination of adversarial loss and content loss is used as the optimization objective. By alternating between the generator and discriminator, the distribution of high-resolution images is gradually approached. In step S4, illegal buildings are identified by integrating unsupervised and supervised learning detection algorithms, including: constructing an initial feature extractor using self-supervised pre-training and performing supervised fine-tuning using Faster R-CNN as a framework; Self-supervised pre-training utilizes the MoCo v3 framework, constructing pre-training tasks through visual transformations. Positive sample pairs are generated by applying random cropping, flipping (horizontally or vertically), and color jittering to the same image. ResNet-101 is used as the encoder to capture richer feature representations, and the momentum queue is dynamically updated to maintain the timeliness and diversity of feature representations. During the pre-training phase, the feature extractor is continuously optimized, and the InfoNCE loss function is used, aiming to maximize the similarity between positive samples while minimizing the similarity between negative samples, thereby enhancing the model's generalization ability. Faster R-CNN performs supervised fine-tuning on the framework. During the supervised fine-tuning phase, a hard example mining strategy is used to focus on samples with low classification confidence or large bounding box regression errors. By backpropagating the gradients of these hard examples, the model is more focused on accurately detecting illegal buildings. Combine time series analysis and multi-source data verification to output detection results, including: using optical flow to associate targets across frames, and verifying spatial coordinate matching between detection results and GIS map data; For example, the Farneback dense optical flow algorithm is used to calculate the pixel motion field between adjacent frames. This algorithm compares the grayscale value changes of pixels in adjacent frames to estimate the pixel motion vector, thereby achieving accurate tracking of the target area. For detected illegal structures, a motion trajectory model is established and its position is predicted using Kalman filtering to smooth the motion trajectory and reduce noise interference.

[0017] The detection frame coordinates (pixel level) are converted to the WGS84 geographic coordinate system using a perspective projection model for comparison with the land planning department's building approval database. The perspective projection model takes into account the camera's intrinsic and extrinsic parameters, accurately converting image coordinates to geographic coordinates. By comparing with the building approval database, unregistered illegal construction areas are marked, enabling accurate identification of illegal buildings. GIS spatial matching is the process of verifying the spatial coordinates of the detection frame against GIS map data. By converting the detection frame coordinates to the WGS84 geographic coordinate system and comparing them with the building information in the GIS map, the accuracy of the detection results can be further verified. GIS spatial matching also provides the specific geographic location of illegal construction areas, providing strong support for subsequent law enforcement actions.

[0018] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A machine vision monitoring method for illegal buildings based on a low-altitude aircraft, characterized by: The following steps are involved: S1: Video data of the target area is collected by a low-altitude aircraft equipped with a multispectral camera and an IMU / BDS module. The multispectral camera includes at least visible light and near-infrared band sensors. S2: Process the raw video data using a multi-sensor fusion dynamic image stabilization preprocessing method, including motion compensation based on IMU data and electronic image stabilization combined with SIFT feature point matching; S3: Process video frames using a two-stage image enhancement technique based on deep learning. The first stage is to use the U-Net network for deblurring. The second stage is to achieve super-resolution reconstruction through the SRGAN network. S4: Identify illegal buildings by integrating unsupervised and supervised learning detection algorithms, including: constructing an initial feature extractor using self-supervised pre-training and supervised fine-tuning based on the Faster R-CNN framework; S5: Combine time series analysis and multi-source data verification to output detection results, including: using optical flow method to associate targets across frames, and verifying spatial coordinate matching between detection results and GIS map data.

2. The method for machine vision monitoring of illegal buildings based on a low-altitude aircraft according to claim 1, characterized in that: The low-altitude aircraft in S1 is a six-rotor drone with an operating altitude of 200-500 meters. The positioning accuracy of the IMU / BDS module is not less than 0.1 meter, the gyroscope zero bias stability of the IMU module is less than 0.5° / h and the sampling frequency is 200Hz. The BDS module uses dual-frequency RTK technology and supports SBAS / QZSS enhanced signals, with horizontal positioning accuracy better than 2 cm and vertical positioning accuracy better than 5 cm.

3. The method for machine vision monitoring of illegal buildings based on low-altitude aircraft according to claim 1, characterized in that: The dynamic image stabilization preprocessing in S2 includes: using Kalman filtering to fuse the IMU angular velocity and the motion vector of the image feature point; and establishing an affine transformation model to compensate for the motion between consecutive frames.

4. The method for machine vision monitoring of illegal buildings based on a low-altitude aircraft according to claim 1, characterized in that: The loss function of the U-net network in S3 adopts a weighted combination of Charbonnier loss and perceptual loss; the generator of the SRGAN network adopts the ResNet architecture, the discriminator adopts the PatchGAN structure, the U-Net network adopts the U-Net++ structure and adds dense skip connections between the encoder and decoder; the SRGAN network adopts the ESRGAN architecture, its generator is based on the residual dense block (RRDB) and the discriminator adopts the spectral normalization PatchGAN structure.

5. The method for machine vision monitoring of illegal buildings based on low-altitude aircraft according to claim 1, characterized in that: The self-supervised pre-training in S4 uses the MoCo v3 framework, and the pretext task is image puzzle reconstruction. During supervised fine-tuning, a difficult sample mining strategy is adopted, and the negative sample ratio is no less than 1:

3. The self-supervised pre-training applies random cropping, ±30° rotation, and color jittering to the input image through the MoCo v3 framework to generate positive sample pairs. The supervised fine-tuning uses a feature pyramid (FPN) to fuse multi-scale features, configures a decoupled detection head, and retains the top 30% of difficult negative samples in prediction confidence.

6. The method for machine vision monitoring of illegal buildings based on low-altitude aircraft according to claim 1, characterized in that: The multi-source data verification in S5 includes: accessing the building approval database of the land planning department for legality verification; calculating the volume parameters of illegal buildings through three-dimensional point cloud reconstruction; the time series analysis uses the Farneback dense optical flow algorithm to establish the target motion trajectory and integrates the Kalman filter to predict the position; multi-source verification uses the DBSCAN algorithm to cluster the detection frames of 5 consecutive frames to eliminate instantaneous false targets, and converts the pixel coordinates to the WGS84 coordinate system through perspective projection to verify the legality with the building approval database.

7. A system for implementing the method according to any one of claims 1 to 6, characterized in that: include: The onboard module includes a multispectral gimbal camera and a high-precision IMU / BDS integrated navigation system. The edge computing unit includes a Jetson AGX Orin processor deployed on the aircraft for real-time image stabilization. The ground analysis platform includes a GPU server cluster that runs deep learning detection and verification algorithms. The multispectral gimbal camera uses a five-lens synchronous acquisition architecture: the visible light band (450-650 nanometers) has a resolution of 3840×2160@30fps, and the near-infrared band (700-900 nanometers) has a resolution of 1280×720@30fps. The edge computing unit is deployed on the Jetson AGX Orin platform to execute the dynamic image stabilization algorithm in real time.

8. The low-altitude aircraft-based machine vision monitoring system for illegal buildings according to claim 7, characterized in that: The multispectral pan-tilt camera has a polarization light filtering function, and the operating waveband covers 450-900nm; the edge computing unit and the ground platform transmit data via a 5G private network, with an end-to-end delay of less than 200ms; the multispectral pan-tilt camera includes a polarization light channel to eliminate mirror reflection interference; the edge computing unit and the ground platform transmit data via a 5G private network, using the RTP protocol to encapsulate the video stream and appending a BDS timestamp and drone ID code to the data packet header.

Images