Illegal building monitoring method and device based on oblique photography

Abstract

The invention discloses an illegal building monitoring method and device based on oblique photography, and the device comprises a feature extraction module, a multi-modal data collection module, a data fusion and analysis module, and a dynamic early warning threshold setting module. Comprising the following steps of S1, auxiliary data collection, S2, image semantic segmentation, S3, multi-modal data collection and transmission, and S4, data fusion analysis and early warning. According to the method, building semantic areas are divided carefully through a semantic segmentation technology, illegal building parts can be accurately positioned, balcony extension and roof storey adding of old communities, illegal building of industrial park plants or illegal building behaviors of various buildings of suburban joint parts can be accurately identified, misjudgment can be effectively reduced, and by means of multi-modal data fusion, the construction efficiency can be improved. By combining the advantages of various sensors such as visible light, thermal imaging and millimeter wave radar, quick discovery and early warning can be realized at the early stage of construction violation or when obvious construction violation signs appear.

Description

Technical Field

[0001] The present invention relates to the technical field of illegal construction detection, and in particular to an illegal construction monitoring method and device based on oblique photography. Background Art

[0002] Urban planning is a blueprint for the orderly development of a city, which stipulates key elements such as the layout, use, height, density, etc. of various types of buildings. The emergence of illegal buildings disrupts this carefully planned layout and causes an imbalance in the urban spatial structure.

[0003] In the process of urbanization, illegal construction control is crucial, but traditional monitoring methods have many limitations. Manual inspections are inefficient, prone to omissions, and their accuracy is affected by personnel experience; although satellite remote sensing has a wide coverage, its resolution is limited and is greatly affected by weather, making it difficult to identify small hidden illegal buildings. Drone inspections currently rely on a single visible light camera, and data processing is simple and prone to misjudgment. With the development of computer vision and artificial intelligence, deep learning algorithms have achieved remarkable results, semantic segmentation can accurately identify illegal construction sites, multimodal data fusion can improve monitoring accuracy, and dynamic warning threshold setting can reduce false alarms and missed reports. Therefore, there is an urgent need for illegal construction monitoring systems and methods that combine these innovative technologies to overcome traditional deficiencies and improve monitoring efficiency. Accordingly, the present invention proposes a method and device for monitoring illegal buildings based on oblique photography. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an illegal construction monitoring method and device based on oblique photography.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A device for monitoring illegal buildings based on oblique photography, comprising a feature extraction module, a multimodal data acquisition module, a data fusion and analysis module, and a dynamic warning threshold setting module, wherein the feature extraction module is used to process the acquired building image data, and divide the building image into a plurality of semantic sub-regions to accurately identify the features of illegal building parts, wherein the semantic sub-regions include but are not limited to roofs, walls, and balconies;

[0007] The data acquisition module is used to process the collected building image data, integrating a visible light camera, a thermal imager and a millimeter wave radar. The visible light camera is used to obtain the visual image of the building appearance, the thermal imager is used to monitor the temperature distribution of the building surface, and the millimeter wave radar is used to detect the information of dynamic objects around the building. Each modal data is synchronously collected and transmitted;

[0008] The data fusion and analysis module is used to fuse multimodal data, combine the building semantic area features, and comprehensively judge whether there is illegal construction. The data fusion and analysis module receives data from the feature extraction module and the multimodal data acquisition module, performs fusion analysis on the multimodal data according to the preset algorithm, and judges whether there is illegal construction in combination with the building semantic area features;

[0009] The dynamic warning threshold setting module is used to dynamically adjust the warning threshold based on the history of illegal construction in the area, seasonal factors and time period factors to ensure timely and effective warnings. The dynamic warning threshold setting module dynamically adjusts the warning threshold based on the historical data of illegal construction in the area, seasonal factors and time period factors. When the probability of illegal construction is higher than the dynamic warning threshold, the warning signal is triggered.

[0010] Preferably, the feature extraction module adopts a deep learning model and is pre-trained with a large number of annotated building image samples so that it can accurately distinguish the feature differences between legal buildings and illegal buildings in each semantic sub-region, and the training samples cover a variety of architectural styles and illegal building types.

[0011] Preferably, in the application scenario of illegal construction monitoring in old residential areas, the thermal imager in the multimodal data acquisition module can monitor the temperature changes in the construction hotspot area, and the millimeter-wave radar can detect signs of frequent material handling around the building. Once abnormal thermal imaging hotspots appear in the building area identified by semantic segmentation and the millimeter-wave radar captures the corresponding dynamics around the building, combined with the dynamic warning threshold setting module, a rapid warning can be issued within 1-2 days.

[0012] Preferably, in the application scenario of illegal construction monitoring in industrial parks, the data fusion and analysis module compares the surface temperature data of the factory building collected by the infrared thermal imager with the standard temperature distribution, combined with the material handling frequency monitored by the millimeter wave radar. When semantic segmentation finds that the appearance semantic features of a specific semantic area of ​​a factory building show signs of new building structures, and the temperature and material handling indicators exceed the set thresholds, an early warning of illegal construction and expansion risks is issued 3-5 days in advance.

[0013] Preferably, in the application scenario of illegal construction monitoring in the urban-rural junction, the dynamic warning threshold setting module is based on the local illegal construction historical frequency, the construction characteristics of the current season, and the concealment of the time period through the formula P dyn =P base +ω×H hist +τ×S season +ξ×T time Dynamically adjust the warning threshold P base , where P base is the basic warning threshold, H hist is the frequency factor of illegal construction in the region, S season is the seasonal influence factor, T timeis the time period influencing factor, ω, τ, ξ are weight coefficients, and when ω+τ+ξ=1, a timely warning is issued based on the adjusted threshold value in the early stage of illegal construction.

[0014] Preferably, during the execution of the illegal building monitoring method, new illegal building samples and legal building samples are regularly collected, and the deep learning model in the feature extraction module is updated and trained to adapt to changes in building morphology and improve the accuracy of illegal building monitoring.

[0015] Preferably, a method for monitoring illegal construction comprises the following steps:

[0016] S1. Collect auxiliary data. Use an aircraft equipped with imaging equipment to collect multi-angle, high-resolution images of the monitoring area according to a preset route. At the same time, collect auxiliary data such as the aircraft's position information, attitude information, and shooting time;

[0017] S2, image semantic segmentation, transferring the collected image data to the feature extraction module for processing and dividing the semantic sub-areas of the building;

[0018] S3, multimodal data acquisition and transmission, the multimodal data acquisition module synchronously collects visible light, thermal imaging, and millimeter wave radar data, and transmits them to the data fusion and analysis module;

[0019] S4. Data fusion analysis and early warning. The data fusion and analysis module combines the semantic sub-region features with multimodal data, and determines whether there is any illegal construction based on the threshold set by the dynamic early warning threshold setting module. If so, an early warning is triggered, and a report containing the illegal construction location, evidence pictures, and early warning time is generated.

[0020] The present invention has the following beneficial effects:

[0021] 1. Through the semantic segmentation technology, the semantic areas of buildings are divided in detail, and the illegal construction parts can be accurately located. Whether it is the balcony expansion and roof addition in the old community, the illegal construction of the factory in the industrial park, or the illegal construction of various buildings in the urban-rural fringe, they can be accurately identified and misjudgment can be effectively reduced. With the help of multimodal data fusion, combined with the advantages of multiple sensors such as visible light, thermal imaging, millimeter wave radar, etc., it can quickly discover and warn in the early stage of illegal construction or when there are obvious signs of illegal construction.

[0022] 2. Automatic monitoring systems replace or assist manual inspections, greatly reducing manpower input and improving monitoring efficiency. For example, old residential areas no longer rely solely on manual labor, industrial parks can flexibly adjust drone inspection plans based on production characteristics and illegal construction risk areas, and urban-rural fringe areas can overcome terrain difficulties to achieve comprehensive monitoring, optimize monitoring resource allocation, and improve overall monitoring efficiency.

[0023] 3. The monitoring system has good adaptability to the characteristics of different regions. Although the industrial park has a complex environment, it can still be effectively monitored through technical means; although the urban-rural fringe area has complex terrain and scattered buildings, the combination of high-altitude drones and ground distributed millimeter-wave radars and semantic segmentation ensures that there are no blind spots in monitoring, adapts to various complex environments and building distribution conditions, and ensures the comprehensiveness and effectiveness of monitoring.

[0024] 4. Although the initial equipment investment is large, such as the purchase of drones, multimodal sensors and supporting software, in the long run, timely discovery and prevention of illegal construction avoids subsequent high governance costs. For example, the long-term monitoring efficiency of urban-rural fringe areas has been significantly improved, and compared with the costs of demolition and rectification after the illegal buildings are built, the initial investment costs are effectively balanced, achieving cost-effectiveness optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of an illegal construction monitoring method based on oblique photography proposed by the present invention;

[0026] Figure 2 It is a structural schematic diagram of an illegal construction monitoring device based on oblique photography proposed in the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0028] An illegal construction monitoring device based on oblique photography includes a feature extraction module: a deep learning model is used to train with a large number of labeled samples covering various architectural styles and illegal construction types. By learning the feature differences of different semantic sub-regions when they are legal and illegal, the building image data is accurately processed, and the semantic sub-regions are divided to identify the illegal construction parts, such as distinguishing between normal facilities on the roof and illegal construction, legal reinforcement of the wall and illegal expansion, and legal renovation of the balcony and over-limit closure.

[0029] Multimodal data acquisition module: integrated with visible light camera, thermal imager and millimeter wave radar. Visible light camera obtains building appearance details; thermal imager monitors building surface temperature changes and captures construction hotspots; millimeter wave radar detects information of surrounding dynamic objects. Data of each modality is collected and transmitted synchronously to provide clues for illegal construction judgment, such as monitoring material handling and temperature anomalies in old residential areas, comparing factory temperature and material handling in industrial parks, and monitoring the characteristics of the adaptation environment in urban-rural fringe areas.

[0030] Data fusion and analysis module: Receive data from the above modules, use advanced algorithms for fusion analysis, and judge illegal buildings based on the semantic regional characteristics of buildings. Judge in different scenarios based on corresponding indicators, such as judging illegal construction in old residential areas based on regional information, early warning expansion of industrial parks based on factory data, and timely discovering illegal buildings in urban-rural fringe areas based on dynamic thresholds.

[0031] Dynamic warning threshold setting module: Analyze the history of illegal construction in the region, seasons and time period factors, and dynamically adjust the threshold through complex modeling. When the probability of illegal construction exceeds the threshold, multiple forms of warnings such as SMS, warning windows, sound and light alarms are triggered to ensure timely information transmission.

[0032] A method for monitoring illegal constructions comprises the following steps:

[0033] S1. Use professional aircraft to collect images of the monitoring area and auxiliary data such as location, attitude, time, etc. according to preset routes. Route planning takes into account terrain, building distribution and illegal construction hotspots to ensure comprehensive coverage and data accuracy.

[0034] Collect auxiliary data:

[0035] Aircraft preparation and route planning: According to the characteristics of the monitoring area, select a suitable aircraft equipped with imaging equipment, such as a drone. Before flying, conduct a comprehensive inspection of the drone, including whether the battery power is sufficient, whether the flight control system is normal, and the clarity and stability of the imaging equipment. At the same time, according to the scope of the monitoring area, the topography and the distribution of buildings, use professional geographic information system (GIS) software or flight planning applications to carefully plan the preset route of the aircraft. The route design must ensure that it can cover the entire monitoring area and avoid monitoring blind spots. Considering the height differences of different buildings, the flight altitude should be set reasonably to obtain clear and comprehensive building images. For areas with dense high-rise buildings, the flight altitude should be appropriately increased to ensure that the top of the building can be photographed; in areas with complex terrain, such as mountainous or hilly areas, the route should be adjusted according to the terrain to ensure that the aircraft is safe and the image collection is complete. In terms of shooting angles, multi-angle shooting methods are used, such as horizontal shooting and tilted shooting (tilting the camera from different angles) to obtain detailed information on each facade of the building, including the building's appearance (such as the shape and outline of the building), color, texture (wall material texture, roof tile texture, etc.).

[0036] Image acquisition and auxiliary data recording: The aircraft flies according to the preset route. During the flight, the imaging equipment continues to work and shoots the buildings in the monitoring area from multiple angles. When shooting, adjust the camera parameters in real time according to the actual situation of the building, such as focal length (adjust according to the distance between the building and the aircraft to ensure clear imaging of the building), aperture (adjust according to light conditions and depth of field requirements to ensure the overall clarity of the picture and highlight the key parts), shutter speed (when the light is dark or there are many moving targets in the building, choose a suitable shutter speed to avoid blur), etc., to obtain high-quality visual images of the building's appearance. At the same time, through the built-in global positioning system (GPS) or Beidou navigation system of the aircraft, the position information of the aircraft is collected in real time, and the latitude and longitude coordinates of the aircraft when each image is taken are recorded to a certain accuracy (such as a few decimal places) so as to determine the geographical location of the building later. The attitude information of the aircraft, including heading angle (the angle between the flight direction of the aircraft and the true north direction), pitch angle (the angle between the longitudinal axis of the aircraft body and the horizontal plane) and roll angle (the rotation angle of the aircraft around the longitudinal axis of the fuselage), is obtained through sensors such as the inertial measurement unit (IMU). These attitude information helps to determine the shooting angle and the spatial position relationship of the image. Accurately record the shooting time, use a unified time standard (such as Coordinated Universal Time, UTC), record to the second or even millisecond level accuracy, provide an accurate time series reference for subsequent data processing and analysis, and facilitate comparison and correlation analysis of data collected at different times.

[0037] S2. The image data is transmitted to the feature extraction module for processing after verification and error correction, and the building semantic sub-areas are divided.

[0038] Image semantic segmentation:

[0039] Data transmission and preprocessing: After the acquisition is completed, the image data is transmitted in real time to the processing platform where the feature extraction module is located through high-speed wireless communication technology (such as 4G / 5G network or dedicated high-speed data transmission link). During the transmission process, the data is preliminarily checked to check whether the data is complete, whether there are any transmission errors or lost data packets, etc. At the same time, the image data is sorted and classified and stored according to information such as shooting time and location to ensure the orderliness of the data. After receiving the image data, it is preprocessed. First, the image format is converted to convert the original image format (which may vary depending on the imaging device) into a common format suitable for deep learning model processing, such as JPEG or PNG format, to ensure the compatibility of the data in subsequent processing. Then the image is cropped to remove the invalid area at the edge of the image (such as part of the sky or ground area caused by the shooting angle), highlight the main part of the building, reduce data redundancy, and improve processing efficiency. Then the image enhancement operation is performed, and the corresponding algorithm is used to adjust according to the brightness, contrast, color balance, etc. of the image. For example, if the image is dark overall, the building details can be made clearer by increasing the brightness and adjusting the contrast; if the color saturation is insufficient, the color balance can be appropriately enhanced to highlight the color characteristics of the building, so that the subsequent semantic segmentation model can better identify the building structure and features.

[0040] Semantic segmentation model processing: The preprocessed image data is input into a pre-trained semantic segmentation model based on deep learning. The model uses an advanced neural network architecture, such as the U-Net architecture based on convolutional neural network (CNN) or its improved version. The U-Net architecture has an encoding-decoding structure. In the encoding part, the feature information of the image is gradually extracted through the convolution layer and the pooling layer, the image resolution is reduced, and high-level semantic features are obtained; in the decoding part, the image resolution is gradually restored through the upsampling layer and the convolution layer, and the feature information of the encoding part is integrated to achieve pixel-level classification of the image. During the training phase, the model is trained using a large number of annotated architectural image samples. These samples cover a variety of architectural styles (such as European, Chinese, modern minimalist, etc.), different building types (residential, commercial buildings, industrial plants, etc.) and various types of illegal buildings (such as roof additions, balcony extensions, wall protrusions, etc.). During the training process, the model learns the feature representations of different semantic areas (such as roofs, walls, doors and windows, balconies, ancillary facilities, etc.) in the image, and can accurately distinguish the feature differences between legal building structures and illegal construction parts. When processing image data, the model classifies the input image pixel by pixel and assigns a semantic category label to each pixel, thus clearly dividing the semantic sub-areas of the building. For example, the roof area is marked as one category, and the wall is marked as another category. Different colors or marks are used to intuitively display each semantic sub-area in the output image, providing an accurate basis for the division of building structures for subsequent illegal construction judgments.

[0041] S3. The multimodal data acquisition module synchronously collects data and transmits it to the fusion analysis module. Each mode works according to the set parameters and maintains time synchronization.

[0042] Multimodal data acquisition and transmission:

[0043] Multimodal sensors work synchronously: The visible light camera, thermal imager and millimeter wave radar in the multimodal data acquisition module start working at the same time to collect data from the buildings in the monitoring area. The visible light camera continuously captures the appearance of the building, and its shooting frequency matches the image acquisition frequency of the aircraft to ensure that multimodal data is obtained at the same time or at a similar time. The shooting parameters of the visible light camera (such as focal length, aperture, shutter speed, etc.) are optimized and adjusted according to the lighting conditions and building characteristics to obtain clear and accurate visual information of the building appearance, complementing the aircraft imaging equipment, providing more comprehensive and multi-angle building appearance details, which can be used to further verify and supplement the building appearance feature details, such as building decorative lines, facade material texture, etc. The thermal imager monitors the temperature distribution of the building surface in real time. Its working principle is based on the infrared radiation characteristics of the object and can detect tiny temperature differences on the building surface. The temperature resolution and spatial resolution of the thermal imager are selected according to the monitoring needs. Generally, the temperature resolution can reach 0.1℃ or even higher, and the spatial resolution can clearly distinguish different areas on the building surface. During the monitoring process, the thermal imager can capture the heat changes generated during the construction process, such as the hydration heat after cement pouring, the high temperature generated by metal processing, etc. These abnormal temperature areas may be related to illegal construction activities. Millimeter wave radar transmits millimeter wave signals and receives reflected waves. Its operating frequency is usually in the millimeter wave frequency band (such as 30GHz-300GHz), and it can penetrate a certain degree of obstruction (such as smoke, dust, light vegetation, etc.). The detection range and angle of the millimeter wave radar are adjusted according to the size of the monitoring area and the distribution of buildings to ensure that the entire monitoring area can be covered. When working, the millimeter wave radar continuously detects information about dynamic objects around the building, including the distance between the object and the radar (the accuracy can reach centimeters or even higher), relative speed (accuracy depends on the performance of the radar) and direction of movement, etc., effectively monitoring the handling of people, vehicles and materials around the building, and providing an important basis for judging the abnormality of construction activities.

[0044] Data transmission and integration preparation: The data collected by each modal sensor is transmitted to the data fusion and analysis module in real time through wired or wireless communication (select appropriate communication technology according to actual conditions, such as Ethernet, Wi-Fi, Bluetooth or dedicated data transmission line, etc.). During the transmission process, the data is packaged, encoded and processed to ensure the stability and accuracy of data transmission and prevent data loss or errors. For visible light image data, image compression technology (such as lossless or lossy compression algorithms such as JPEG2000) is used to reduce the amount of data and improve transmission efficiency while ensuring image quality; thermal imaging temperature data is digitally encoded and transmitted in a certain data format (such as binary format); millimeter wave radar dynamic data is signal processed and converted into data format, and the original radar echo signal is converted into digital format data containing information such as distance, speed, angle, etc. In the data fusion and analysis module, the received multi-modal data is synchronized and formatted uniformly. Since the frequency and time base of the data collected by each sensor may be different, the visible light image data, thermal imaging temperature data and millimeter wave radar dynamic data are accurately synchronized in time through technical means such as timestamp alignment, so that they can accurately correspond to the building status at the same time. At the same time, data in different formats are converted into a unified data format, such as converting all data into a matrix form or a specific data structure, to facilitate subsequent fusion analysis operations, ensure that each modal data can be accurately matched and associated in space and time, and provide a basic guarantee for data fusion analysis.

[0045] S4. The fusion analysis module combines semantic regions with multimodal data and determines illegal buildings based on dynamic thresholds. If any, an early warning is triggered and a report containing the location of the illegal building, evidence images, and early warning time is generated and passed on to relevant personnel.

[0046] Data fusion analysis and early warning:

[0047] Multimodal data fusion: The synchronously processed visible light image data, thermal imaging temperature data and millimeter wave radar dynamic data are fused. Based on the semantic sub-area division of the building, the data of each modality are spatially registered and associated so that the data collected by different sensors can accurately correspond to the specific parts of the building. First, the visible light image is accurately matched with the building area after semantic segmentation through the image registration algorithm to determine the position of each pixel in the building semantic area; then, the temperature anomaly area detected by the thermal imager is matched with the roof, wall and other areas after semantic segmentation to determine the specific building part where the temperature anomaly occurs; at the same time, the movement trajectory of the surrounding dynamic objects detected by the millimeter wave radar is associated with the entrance and exit of the building, the construction area (determined according to semantic segmentation), etc., to analyze the relationship between object movement and building activities. Use appropriate data fusion algorithms (such as data fusion methods based on Bayesian estimation, Kalman filtering or deep learning) to fuse multimodal data. The Bayesian estimation method calculates the posterior probability based on the prior probability and likelihood function, comprehensively considers the information provided by each modal data, and determines the possibility of illegal construction behavior; the Kalman filter makes the best estimate of the dynamic system (such as the movement of objects around the building and the change of the building state) through prediction and update steps, and integrates multimodal data to achieve real-time tracking and analysis of the building state; the deep learning method (such as the convolutional neural network fusion architecture) automatically extracts features and fuses them by learning a large number of multimodal data samples to improve the fusion effect. The algorithm comprehensively considers the characteristics and reliability of each modal data, integrates the information provided by different modal data according to the preset weight distribution scheme, and generates a comprehensive description of the building state and activities. For example, in the construction area determined by semantic segmentation, the visible light image shows a newly built building structure frame, the thermal imager detects the temperature rise in the area, and the millimeter wave radar detects frequent material handling vehicle activities in the surrounding area. The data fusion algorithm comprehensively analyzes this information to improve the accuracy of the judgment of illegal construction behavior.

[0048] Illegal construction judgment and early warning decision-making: The data fusion and analysis module judges illegal construction behavior based on the threshold set by the dynamic early warning threshold setting module. The dynamic early warning threshold setting module calculates the real-time early warning threshold based on the historical data of illegal construction in the area, seasonal factors, time period factors, etc. through a pre-established mathematical model (such as a weighted sum model, a fuzzy logic model, etc.). The weighted sum model assigns corresponding weights to each factor according to its importance and calculates the threshold, such as P dyn =P base +ω×H hist +τ×S season +ξ×T time , where P dyn is the dynamic warning threshold, P base is the basic warning threshold, H histis the frequency factor of illegal construction in the region, S season is the seasonal influence factor, T time is the time period influencing factor, ω, τ, ξ are weight coefficients and ω+τ+ξ=1. The fuzzy logic model considers the fuzziness of the factors and calculates the threshold through fuzzy rules and membership functions. In different scenarios, the weight coefficients and model parameters are adjusted according to the actual situation. In areas with a high incidence of illegal construction in history, increasing H hist The weight of S season At night or during concealed construction, the T time The illegal construction possibility score calculated by the fusion data is compared with the dynamic warning threshold. If the illegal construction possibility score exceeds the threshold, it is determined that there is illegal construction and the early warning mechanism is triggered. There are various early warning methods, including sending SMS notifications to relevant law enforcement departments or management personnel (including information such as the location of the illegal construction and a brief situation), popping up a warning window on the monitoring system platform (displaying pictures and detailed information of illegal construction evidence), issuing sound and light alarms (sounding alarms and flashing lights at the monitoring site or control center), etc., to ensure that the early warning information can be conveyed to relevant personnel in a timely and effective manner so that timely measures can be taken to stop illegal construction.

[0049] Report generation and recording: When the warning is triggered, the system automatically generates a report containing detailed illegal construction information. The report content includes the location of the illegal construction, and the specific building semantic area where the illegal construction occurs is accurately marked through semantic segmentation (such as the privately built part on the east side of the roof of a building, the specific coordinate range; the area of ​​the wall of a unit building that is expanded outward, the floors and wall positions involved, etc.), using precise geographic coordinates or building relative position descriptions to facilitate law enforcement personnel to accurately find the location of the illegal construction. Evidence images integrate key evidence images related to illegal construction in multimodal data such as visible light images, thermal imaging images, and millimeter wave radar data visualization images, and intuitively display the characteristics of illegal construction from different angles (such as the appearance of illegal construction, abnormal temperature areas, and the association between the trajectory of surrounding dynamic objects and illegal construction) and related activity signs, providing strong visual evidence for law enforcement. The warning time records the exact time point when the illegal construction is discovered and the warning is triggered, accurate to seconds or even milliseconds, providing time clues for subsequent law enforcement actions, and facilitating the tracing of the discovery process and time sequence of illegal construction. At the same time, the report will be stored in the database and classified according to the location, time, type, etc. of the illegal construction, which is convenient for historical query and data analysis, and provides data support for subsequent illegal construction management and monitoring system optimization. Through statistical analysis of historical illegal construction data, the monitoring strategy and algorithm parameters are continuously adjusted to improve the performance of the monitoring system.

[0050] Embodiment 1:

[0051] In a project to monitor illegal buildings in old residential areas with diverse architectural styles, an intelligent monitoring system was deployed. The system uses feature extraction technology based on semantic segmentation to process high-definition images collected by drones.

[0052] Assume that the pixel set of the building in the image is B, and divide it into n semantic sub-regions through the deep learning semantic segmentation model, such as roof region R, wall region W, balcony region Y, etc., satisfying Where Si represents the i-th semantic sub-region.

[0053] At the same time, multimodal sensors, including visible light cameras, thermal imagers, and millimeter wave radars, are installed at fixed monitoring points on the ground. For thermal imaging data, the normal surface temperature range of the building is defined as [Tmin, Tmax]. When the temperature T of a certain area of ​​the building is monitored by a thermal imager, When the area is determined to be a semantic sub-area under construction (such as a roof covering area) through semantic segmentation, it is initially marked as a suspected illegal construction hotspot.

[0054] Millimeter wave radar is used to monitor the dynamics around the building and set the speed threshold V of normal moving objects. th , if the speed of the object approaching the building is detected to be V>Vth, and the object is approaching the semantic area corresponding to the suspected illegal construction hotspot, combined with the semantic segmentation information, the probability P of the building being illegal is comprehensively judged:

[0055] P=α×P thermal +β×P radar +γ×P seg

[0056] Where P thermal is the probability of illegal construction determined by thermal imaging (high value when temperature is abnormal), P radar is the probability of illegal construction determined by the millimeter-wave radar (high value when the object is abnormal in dynamics), P seg is the probability of illegal construction determined by semantic segmentation (high value when the semantic features of the building appearance change). α, β, and γ are weight coefficients, which are determined based on experience or training data and satisfy α+β+γ=1. threshold (preset warning probability threshold), the system immediately sends a warning message to the monitoring center.

[0057] Table 1: Data comparison table of old residential quarters and Example 1

[0058]

[0059] Embodiment 2:

[0060] In a large industrial park, an innovative illegal construction monitoring solution was adopted to prevent enterprises from expanding their factories without permission. Drones were used to carry multi-lens tilt cameras and infrared thermal imagers for regular inspections.

[0061] Firstly, the oblique photography images are processed based on semantic segmentation, and the image pixels of the factory buildings in the park are divided into different semantic parts. The total building pixel area is set as P, which is divided into production area A, storage area B, office area B, etc., that is, Q j is the jth semantic sub-region.

[0062] The infrared thermal imager collects the surface temperature data of the factory. Since different production process factories have relatively stable temperature distribution patterns during normal operation, the standard temperature distribution function is set to T(x,y), where (x,y) is the image coordinate. The actual monitored temperature distribution function is T′(x,y), and the temperature difference function ΔT(x,y)=T′(x,y)-T(x,y) is calculated.

[0063] At the same time, the millimeter-wave radar of the ground monitoring station monitors the dynamics of vehicles and material handling around the building. The material handling frequency function F(t) is defined to represent the number of material handling times near the factory building within a unit time t. When semantic segmentation is used to find that the appearance semantic features of the storage area B of a factory building have signs of new building structures, and ∫ s |ΔT(x,y)|dxdy>T int (S is the image area corresponding to the storage area, T int is the temperature integration threshold) and F(t)>F th (F th is the handling frequency threshold), the factory building is judged to have the risk of illegal construction and expansion, and an early warning is triggered. The early warning level L can be determined according to the following formula:

[0064] L = λ × L seg +μ×L thermal+ν ×L radar

[0065] Where L seg is the warning level corresponding to the severity of illegal construction judged by semantic segmentation (for example, the greater the proportion of new structures, the higher the level), L thermal is the warning level determined by thermal imaging (the higher the temperature anomaly, the higher the level), L radar is the warning level determined by the millimeter-wave radar (the higher the frequency of handling, the higher the level), λ, μ, υ are weight coefficients, and λ+μ+ν=1. Different warning levels correspond to different law enforcement response processes to ensure accurate handling of illegal construction risks.

[0066] Table 2: Data comparison table of industrial park comparative example and embodiment 2

[0067]

[0068] Embodiment three:

[0069] In the suburban area, where illegal construction is prevalent and the building types are mixed, a set of high-precision illegal construction monitoring equipment is used. It includes high-altitude drones equipped with high-definition visible light and thermal imaging cameras, and ground-distributed millimeter-wave radar arrays.

[0070] The feature extraction technology based on semantic segmentation is used to process UAV images, and the building pixel area M is subdivided into semantic sub-areas such as residential houses H, agricultural sheds G, and temporary buildings T, namely Where N k is the kth semantic sub-region.

[0071] Thermal imaging cameras monitor the surface temperature of buildings, and define the normal day and night temperature fluctuation range of buildings as [ΔT min ,ΔT max ], the actual monitored temperature fluctuation of a building is ΔT′, if And the building is located in the semantic sub-area T of temporary building after semantic segmentation and is marked as a building with abnormal temperature.

[0072] The ground millimeter-wave radar array monitors the activities of people and vehicles around the building. The population density function is ρ(x, y, t), which represents the density of people at the coordinate (x, y) at time t. The vehicle traffic flow function is When a temporary building T semantic sub-area is found around ∫ A ρ(x,y,t)dxdy>ρth (A is the surrounding area, ρth is the threshold of population density) and (B is the surrounding channel area, is the vehicle traffic flow threshold), combined with semantic segmentation to determine whether the building has illegal construction activities.

[0073] Dynamic warning threshold P dyn It is set based on historical data of illegal construction in the area, current season, time period and other factors, and is dynamically adjusted through the following formula:

[0074] P dyn =P base +ω×H hist +τ×S season +ξ×T time

[0075] Where P base is the basic warning threshold, H hist is the frequency factor of illegal construction in the region, S season is the seasonal influencing factor (e.g. the early warning threshold should be appropriately raised for construction in summer), T timeis the time period influencing factor (e.g., the threshold of concealed construction at night is raised), ω, τ, ξ are weight coefficients, and ω+τ+ξ=1. dyn When a construction project is underway, accurate early warning information will be sent to relevant departments to curb illegal construction activities in a timely manner.

[0076] Table 3: Data comparison table of the comparative example and embodiment 3 in the urban-rural junction

[0077]

[0078] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An illegal construction monitoring device based on oblique photography, comprising a feature extraction module, a multimodal data acquisition module, a data fusion and analysis module, and a dynamic warning threshold setting module, characterized in that: The feature extraction module is used to process the collected building image data and divide the building image into multiple semantic sub-regions to accurately identify the features of the illegal building parts. The semantic sub-regions include but are not limited to roofs, walls, and balconies. The multi-mode data acquisition module is used to process the collected building image data, integrating a visible light camera, a thermal imager and a millimeter wave radar. The visible light camera is used to obtain the visual image of the building appearance, the thermal imager is used to monitor the temperature distribution of the building surface, and the millimeter wave radar is used to detect the information of dynamic objects around the building. Each modal data is synchronously collected and transmitted; The data fusion and analysis module is used to fuse multimodal data, combine the building semantic area features, and comprehensively judge whether there is illegal construction. The data fusion and analysis module receives data from the feature extraction module and the multimodal data acquisition module, performs fusion analysis on the multimodal data according to the preset algorithm, and judges whether there is illegal construction in combination with the building semantic area features; The dynamic warning threshold setting module is used to dynamically adjust the warning threshold based on the history of illegal construction in the area, seasonal factors and time period factors to ensure timely and effective warnings. The dynamic warning threshold setting module dynamically adjusts the warning threshold based on the historical data of illegal construction in the area, seasonal factors and time period factors. When the probability of illegal construction is higher than the dynamic warning threshold, the warning signal is triggered.

2. The illegal construction monitoring device based on oblique photography according to claim 1 is characterized in that: The feature extraction module adopts a deep learning model and is pre-trained with a large number of labeled building image samples, so that it can accurately distinguish the feature differences between legal buildings and illegal buildings in each semantic sub-area. The training samples cover a variety of architectural styles and illegal building types.

3. The illegal construction monitoring device based on oblique photography according to claim 1 is characterized in that: In the application scenario of illegal construction monitoring in old communities, the thermal imager in the multimodal data acquisition module can monitor temperature changes in construction hotspots, and the millimeter-wave radar can detect signs of frequent material handling around the building. Once abnormal thermal imaging hotspots appear in the building area identified by semantic segmentation and the millimeter-wave radar captures the corresponding dynamics around the building, combined with the dynamic warning threshold setting module, a rapid warning can be issued within 1-2 days.

4. The illegal construction monitoring device based on oblique photography according to claim 1 is characterized in that: In the application scenario of illegal construction monitoring in industrial parks, the data fusion and analysis module compares the surface temperature data of the factory building collected by the infrared thermal imager with the standard temperature distribution, combined with the material handling frequency monitored by the millimeter wave radar. When semantic segmentation finds that the appearance semantic features of a specific semantic area of ​​a factory building show signs of new building structures, and the temperature and material handling indicators exceed the set thresholds, the risk of illegal construction expansion will be warned 3-5 days in advance.

5. The illegal construction monitoring device based on oblique photography according to claim 1 is characterized in that: In the application scenario of illegal construction monitoring in the urban-rural junction, the dynamic warning threshold setting module is based on the local illegal construction historical frequency, the construction characteristics of the current season, and the concealment of the time period through the formula P dyn =P base +ω×H hist +τ×S season +ξ×T time Dynamically adjust the warning threshold P base , where P base is the basic warning threshold, H hist is the frequency factor of illegal construction in the region, S season is the seasonal influence factor, T time is the time period influencing factor, ω, τ, ξ are weight coefficients, and when ω+τ+ξ=1, a timely warning is issued based on the adjusted threshold value in the early stage of illegal construction.

6. The illegal construction monitoring device based on oblique photography according to claim 1 is characterized in that: During the execution of the illegal building monitoring method, new illegal building samples and legal building samples are regularly collected, and the deep learning model in the feature extraction module is updated and trained to adapt to changes in building morphology and improve the accuracy of illegal building monitoring.

7. A method for monitoring illegal construction, comprising the following steps: S1. Collect auxiliary data. Use an aircraft equipped with imaging equipment to collect multi-angle, high-resolution images of the monitoring area according to a preset route. At the same time, collect auxiliary data such as the aircraft's position information, attitude information, and shooting time; S2, image semantic segmentation, transferring the collected image data to the feature extraction module for processing and dividing the semantic sub-areas of the building; S3, multimodal data acquisition and transmission, the multimodal data acquisition module synchronously collects visible light, thermal imaging, and millimeter wave radar data, and transmits them to the data fusion and analysis module; S4. Data fusion analysis and early warning. The data fusion and analysis module combines the semantic sub-region features with multimodal data, and determines whether there is any illegal construction based on the threshold set by the dynamic early warning threshold setting module. If so, an early warning is triggered, and a report containing the illegal construction location, evidence pictures, and early warning time is generated.

Images