An illegal building intelligent identification method based on space-air-ground integrated collaborative perception and a medium

Abstract

The application discloses an illegal building intelligent identification method based on space-air-ground integrated collaborative sensing and a medium, mainly relates to the illegal building identification technical field, and is used to solve the problems that the existing scheme depends on a single data source or mode, adopts simple splicing or summation operation, pays insufficient attention to local geometric structure, and puts the positioning and classification tasks in the same branch. Including: obtaining a pixel-level illegal building probability graph by using multi-modal fusion features and local-global interactive attention features; obtaining classification probability data by using the pixel-level illegal building probability graph and the features of the data of each mode; obtaining a structure loss and a positioning loss by using the pixel-level illegal building probability graph; obtaining a classification loss by using the classification probability data; further obtaining a total loss by weighting; iteratively updating the model by using the total loss until a preset iteration condition is met, to obtain a trained model; and inputting a preprocessed real-time multi-source collaborative sensing array into the trained model to obtain an illegal building identification result.

Description

Technical Field

[0001] This application relates to the field of intelligent identification technology for illegal construction, and in particular to an intelligent identification method and medium for illegal construction based on integrated air-space-ground collaborative perception. Background Technology

[0002] With the continuous evolution of remote sensing and ground-based sensing technologies, the fusion of multi-source heterogeneous data, such as satellite remote sensing data, UAV imagery, and ground-based laser point clouds, provides multi-dimensional information support for the identification of illegal buildings. These data sources are complementary in terms of spatial resolution, spectral characteristics, and three-dimensional morphology, which theoretically can significantly improve the comprehensiveness and accuracy of illegal building identification.

[0003] However, the existing technical solutions still have the following key problems that urgently need to be solved:

[0004] 1. Existing methods typically rely on a single data source or modality, making it difficult to fully utilize the advantages of data from different sources, resulting in the failure to effectively address spatiotemporal differences and modal heterogeneity issues.

[0005] 2. Traditional feature fusion methods often use simple splicing or summation operations without considering the differences in feature levels and semantic importance, resulting in poor fusion effects and easy loss of feature information.

[0006] 3. Existing deep learning models do not pay enough attention to local geometric structures (such as illegal walls and roofs), making it difficult to capture detailed information about illegal areas, resulting in low recognition accuracy.

[0007] 4. Existing illegal construction identification models usually place the localization and classification tasks in the same branch, which leads to feature coupling between tasks and affects the model's recognition accuracy and generalization ability. Summary of the Invention

[0008] This application provides a method and medium for intelligent identification of illegal buildings based on integrated air-space-ground perception to solve the above-mentioned technical problems.

[0009] Firstly, this application provides an intelligent identification method for illegal constructions based on integrated air-space-ground collaborative perception, the method comprising:

[0010] Collect an initial multi-source collaborative sensing array for a preset historical time period, and obtain the multi-source collaborative sensing array after preprocessing; obtain the illegal construction labeling data of the multi-source collaborative sensing array;

[0011] The satellite remote sensing data, UAV data and ground point cloud data in the multi-source collaborative sensing array are fused to obtain the registration parameter matrix; the registration parameter matrix is ​​used to obtain the features of each modality in the multi-source collaborative sensing array; the features of each modality are used to quantify the weight of each modality; and the features and weights of each modality are used to obtain the multi-modal fusion features.

[0012] The multimodal fusion feature transformation generates a query matrix, a key matrix, and a value matrix; the Transformer model introduces a pre-defined local constraint mask matrix that constrains the scope of attention calculation in the standard attention mechanism that includes the query matrix, key matrix, and value matrix, and outputs local-global interactive attention features;

[0013] By utilizing multimodal fusion features and local-global interactive attention features, a pixel-level illegal construction probability map is obtained; by utilizing the pixel-level illegal construction probability map and the features of each modality, classification probability data is obtained.

[0014] Using pixel-level illegal construction probability maps, structural loss and localization loss are obtained; using classification probability data, classification loss is obtained; then weighted summation is used to obtain the overall loss; the overall loss is used to iteratively update the Transformer model until the preset iteration conditions are met, and a trained Transformer model is obtained.

[0015] The preprocessed real-time multi-source collaborative sensing array is input into the trained Transformer model to obtain the illegal construction identification results.

[0016] In one implementation of this application, the multi-source collaborative sensing array includes: satellite remote sensing data, UAV data, and ground point cloud data;

[0017] The initial multi-source collaborative sensing array is acquired, and after preprocessing, the resulting multi-source collaborative sensing array includes:

[0018] Acquire satellite remote sensing data through satellite platforms;

[0019] By using a drone equipped with an RGB camera to take aerial photos at a preset low altitude, high spatial resolution drone data can be obtained.

[0020] Ground point cloud data containing three-dimensional coordinates and reflection intensity is generated by collecting data through ground-based lidar scanning or oblique photogrammetry.

[0021] Radiometric calibration and atmospheric correction are performed on satellite remote sensing data; image stitching and geometric correction are performed on UAV data; and noise reduction and filtering are performed on ground point cloud data.

[0022] Geographic Information System (GIS) tools are used to align processed satellite remote sensing data, UAV data, and ground point cloud data to the same coordinate system and the same timestamp.

[0023] In one implementation of this application, satellite remote sensing data, UAV data, and ground point cloud data in a multi-source collaborative sensing array are fused to obtain a registration parameter matrix, specifically including:

[0024] The point cloud data features are transformed into dense feature map data using a multilayer perceptron.

[0025] Learnable upsampling is performed on satellite remote sensing data to match the spatial size of UAV imagery, and learnable downsampling is performed on UAV data to keep the spatial size of UAV data unchanged. The upsampled satellite remote sensing data, downsampled UAV data, and dense feature map data are then stitched together, and the number of channels is compressed by 1×1 convolution to output the registration parameter matrix.

[0026] In one implementation of this application, the modal data features include satellite enhancement features, UAV enhancement features, and ground fusion features;

[0027] Using the registration parameter matrix, the features of each modality in the multi-source collaborative sensing array are obtained, specifically including:

[0028] The registration parameter matrix is ​​input into the ResNet-50 backbone network, and cross-modal shared feature maps are extracted through convolutional residual blocks and skip connections. The output size is reduced to 1 / 4 of the original UAV data shared feature map.

[0029] A 3×3 convolution transformation is performed on the shared feature map, and the weight vector generated by the channel attention mechanism of the satellite remote sensing data is broadcast to the same spatial dimension. The shared feature map after convolution transformation is weighted by the Hadamard product operation to obtain satellite enhanced features.

[0030] A 3×3 convolution transformation is performed on the shared feature map. The multi-scale spatial features extracted from the UAV data through dilated spatial pyramid pooling are then concatenated with the shared feature map after the convolution transformation to obtain UAV enhanced features.

[0031] A 3×3 convolution transformation is performed on the shared feature map, and the rasterized point cloud features are downsampled to the same size as the shared feature map through a 4×4 convolution. Then, the ground fusion features are obtained by adding elements one by one.

[0032] In one implementation of this application, the weights of each modal data include: satellite augmentation feature weights, UAV augmentation feature weights, and ground fusion feature weights;

[0033] By utilizing the features of each modal data, the weights of each modal data are quantified, specifically including:

[0034] The current modal data features and the shared feature map are concatenated along the channel dimension. After layer normalization and 1×1 convolution compression, the modal position encoding tensor corresponding to the current modal data features is superimposed. The dynamic gating coefficients corresponding to the current modal data features are generated by the Sigmoid function. The modal data weights corresponding to the current modal data features are quantized by the dynamic gating coefficients.

[0035] In one implementation of this application, multimodal fusion features are obtained by utilizing the features and weights of each modality's data, specifically including:

[0036] Group normalization and depthwise separable convolution are performed on each modal data feature to extract the preset local features of each modal data feature. The preset local features of the modal data feature are multiplied element-wise with the modal data weights corresponding to the modal data features to obtain the product results. Then, the weighted sum is obtained to obtain the multimodal fusion features.

[0037] In one implementation of this application, the Transformer model introduces a pre-defined local constraint mask matrix to constrain the scope of attention computation within the standard attention mechanism, which includes a query matrix, a key matrix, and a value matrix. The output is a local-global interactive attention feature, specifically including:

[0038] Through the formula:

[0039] Obtain local-global interactive attention features;

[0040] in, This represents the local-to-global interactive attention features;

[0041] Q represents the query matrix;

[0042] K represents the bond matrix;

[0043] V represents the value matrix;

[0044] This represents the transpose of the key matrix K. This represents the product of the query matrix and the transpose of the key matrix;

[0045] Indicates the dimension of K;

[0046] This represents a preset local constraint mask matrix;

[0047] This represents the normalized exponential function.

[0048] In one implementation of this application, a pixel-level illegal construction probability map is obtained using multimodal fusion features and local-global interactive attention features; classification probability data is obtained using the pixel-level illegal construction probability map and the features of each modality, specifically including:

[0049] The local-global interactive attention features and the multimodal fusion features aligned by 1×1 convolution are residually added, and the addition result is normalized by layer to output the attention fusion transformation features;

[0050] After performing two 3×3 convolution operations on the attention fusion transformation features, data of a preset size is obtained by upsampling by a preset multiple, and then a pixel-level illegal construction probability map is output through the Sigmoid activation function.

[0051] Through the formula:

[0052]

[0053] Calculate the classification probability data;

[0054] in, Represents categorical probability data;

[0055] This indicates the satellite enhancement features in the data of each modality. Global average pooling is performed, and a channel attention weight vector of satellite features with a size of 1×1×256 is generated using a fully connected layer;

[0056] This represents the UAV augmentation features in the data features of each modality. The multi-scale global vector of UAV features with a size of 1×1×256 is extracted by the global average pooling branch of the void space pyramid pooling module.

[0057] It is a fully connected neural network for the classification layer;

[0058] This indicates a vector concatenation operation that aggregates multimodal global features into a comprehensive feature vector;

[0059] This indicates a global average pooling operation;

[0060] This represents the attention fusion transformation feature;

[0061] This represents the shared feature map extracted using the registration parameter matrix;

[0062] This represents the ground fusion feature among the features of each modality of data;

[0063] This represents the normalized exponential function.

[0064] In one implementation of this application, the Transformer model is iteratively updated using the overall loss until a preset iteration condition is met to obtain a trained Transformer model, specifically including:

[0065] The multi-source collaborative sensing array and labeled data for a preset historical time period are split into training set and validation set;

[0066] According to the preset batch, the multi-source collaborative sensing array and labeled data in the training set are input into the model in sequence. When the overall loss of the validation set no longer decreases within a preset number of consecutive cycles or reaches the preset maximum number of iterations, the trained Transformer model is obtained.

[0067] Secondly, this application provides a non-volatile computer storage medium storing computer instructions, which, when executed, implement an intelligent identification method for illegal construction based on integrated air-space-ground perception, as described above.

[0068] As can be seen from the above technical solutions, this application has the following advantages:

[0069] Multi-source data fusion addresses the issues of spatiotemporal differences and modal heterogeneity.

[0070] By integrating satellite remote sensing, UAV aerial photography, and ground point cloud data, a multi-source collaborative sensing array is established and registered. A registration parameter matrix is ​​then used to achieve spatial unification of cross-modal data. Compared to single data sources, this method effectively eliminates spatiotemporal scale differences (such as the macroscopic coverage of satellite imagery versus the microscopic details of ground point clouds). Simultaneously, by quantifying the weights of each modality, it dynamically balances the contributions of data from different sources (e.g., satellite data provides macroscopic layout, UAV data captures mesoscopic structure, and point cloud data characterizes microscopic geometry). This solves the information fragmentation problem caused by modal heterogeneity in traditional methods, improving the comprehensiveness and accuracy of illegal construction identification.

[0071] Multimodal feature fusion optimizes feature hierarchy and semantic importance:

[0072] A dynamic weighting mechanism is employed to fuse multimodal features, avoiding simple splicing or summation operations. By quantifying the weights of each modality, this method can adaptively adjust feature contributions according to the task requirements of illegal construction identification (such as positioning accuracy or classification reliability). For example, in illegal wall detection, high-frequency geometric features of ground point clouds are enhanced, while in roof identification, texture information from satellite imagery is emphasized. This hierarchical fusion strategy preserves the semantic integrity of the original features, reduces information loss, and makes the fused features more reflective of the essential attributes of illegal construction, thereby improving the input quality of subsequent models.

[0073] Local constraint attention mechanism enhances the ability to capture geometric structures:

[0074] By introducing a pre-defined local constraint mask matrix into the Transformer model, the scope of attention computation is limited, enabling the model to focus on the local geometry of the illegal construction area (such as the vertical edges of walls and the sloping surfaces of roofs). Through local-global interactive attention features, this method simultaneously considers the overall layout and detailed features of the illegal construction, solving the problem of insufficient attention to local structures in traditional deep learning models. For example, in complex urban environments, this mechanism can accurately identify the irregular shapes of illegal walls or abnormal protrusions on roofs, significantly improving the detail sensitivity of illegal construction detection.

[0075] Dual-task decoupling design improves localization and classification accuracy:

[0076] The illegal construction location and classification tasks are decoupled into independent branches: the location branch generates structural and localization losses based on pixel-level illegal construction probability maps, while the classification branch uses multimodal features to output classification probability data and calculate classification loss. By iteratively optimizing the model through weighted overall loss, this method avoids the problem of feature coupling between tasks in traditional single-branch models (such as localization errors interfering with classification decisions). The decoupling design allows the location branch to focus on spatial location optimization, while the classification branch strengthens semantic feature extraction, thereby improving the model's generalization ability in complex scenarios, such as maintaining stable output of illegal construction category and location even under changes in lighting or background interference. Attached Figure Description

[0077] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a flowchart of an intelligent identification method for illegal buildings based on integrated air-space-ground perception, provided in an embodiment of this application.

[0079] Figure 2This is a robustness analysis diagram of a method under different environmental complexities provided in the embodiments of this application.

[0080] Figure 3 This is a comparison chart of the boundary recognition quality of different methods provided in the embodiments of this application.

[0081] Figure 4 This is a comparison chart of detection accuracy under different scales of illegal construction provided in the embodiments of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0084] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0085] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0086] The embodiment provides a method for intelligent identification of illegal buildings based on integrated air-space-ground collaborative perception, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps:

[0087] Step 110: Collect the initial multi-source collaborative sensing array for a preset historical time period, and obtain the multi-source collaborative sensing array after preprocessing; obtain the illegal construction labeling data of the multi-source collaborative sensing array.

[0088] In some embodiments, the multi-source collaborative sensing array includes: satellite remote sensing data, UAV data, and ground point cloud data;

[0089] The initial multi-source collaborative sensing array is acquired, and after preprocessing, the resulting multi-source collaborative sensing array includes:

[0090] Satellite remote sensing data is acquired through satellite platforms (specifically, satellite remote sensing data is acquired through satellite platforms such as Gaofen series, Landsat or Sentinel, collecting multi-band raster data, including spectral information such as visible light and near infrared, with a spatial resolution of generally 10 to 30 meters, covering a large area but lacking detail).

[0091] By using drones equipped with RGB cameras to conduct low-altitude aerial photography at preset levels, high spatial resolution drone data is obtained (specifically, drone data is collected by drones equipped with RGB cameras at low altitudes to obtain high spatial resolution RGB images, with resolution reaching the centimeter level, focusing on covering areas suspected of illegal construction).

[0092] Ground point cloud data containing three-dimensional coordinates and reflection intensity is generated by ground-based lidar scanning or oblique photogrammetry. Specifically, ground point cloud data is generated by ground-based lidar scanning or oblique photogrammetry to generate sparse point clouds containing three-dimensional coordinates (XYZ) and reflection intensity. LiDAR directly acquires the point cloud and records the echo intensity, while oblique photogrammetry generates point clouds by reconstructing multi-view images. Its reflection intensity can be derived from multispectral sensors (such as the near-infrared band).

[0093] To construct the training dataset, the raw data is first preprocessed, including radiometric calibration and atmospheric correction of satellite data, image stitching and geometric correction of UAV data, and denoising and filtering of ground point clouds to ensure data quality. Then, spatiotemporal benchmark unification is performed by using geographic information system tools to align multi-source data to the same coordinate system (such as WGS84) and the same timestamp to resolve initial spatiotemporal biases.

[0094] It should be noted that the specific implementation methods of the above preprocessing process are all implemented by existing technologies, and this application does not limit them here.

[0095] In addition, the annotation work can be done manually by professionals. For each set of multi-source data, pixel-level illegal construction area masks are labeled as location labels (two-dimensional matrix, illegal construction area value is 1, non-illegal construction is 0), and image-level classification labels are labeled at the same time (binary classification: illegal construction or non-illegal construction) to form ground truth data.

[0096] When dividing the dataset, it is randomly divided into training set, validation set and test set in a ratio of 7:2:1 to avoid spatiotemporal overlap and ensure generalization. The training set is used for model parameter learning, the validation set is used for hyperparameter tuning and early stopping judgment, and the test set is used for final evaluation.

[0097] Step 120: Perform fusion processing on satellite remote sensing data, UAV data and ground point cloud data in the multi-source collaborative sensing array to obtain a registration parameter matrix; use the registration parameter matrix to obtain the features of each modality in the multi-source collaborative sensing array; use the features of each modality to quantify the weight of each modality; use the features of each modality and the weight of each modality to obtain multi-modal fusion features.

[0098] It should be noted that multi-source data from air, space, and ground have significant differences in spatiotemporal reference and resolution. Specifically, satellite remote sensing data, although possessing multispectral information, has low spatial resolution; UAV data has high spatial resolution but lacks spectral dimensions; and ground sensor data, as a sparse point cloud representation, is incomplete. Conventional registration methods typically use a single spatiotemporal transformation matrix for processing, which is insufficient to effectively address feature misalignment and modal heterogeneity caused by resolution differences, resulting in inadequate registration accuracy.

[0099] This application dynamically adjusts the spatial resolution of multi-source data through upsampling and downsampling operations, and integrates features from satellites, UAVs, and ground point clouds to output a registration parameter matrix of uniform size, thereby achieving adaptive resolution registration of multi-source data and solving the problem of spatiotemporal inconsistency.

[0100] Specifically, satellite remote sensing data, UAV data, and ground point cloud data in the multi-source collaborative sensing array are fused to obtain a registration parameter matrix, as detailed below:

[0101] 1) Point cloud data rasterization processing:

[0102] By performing feature transformation on ground point cloud data using a multilayer perceptron, sparse point cloud data containing XYZ coordinates and reflection intensity is projected onto a two-dimensional grid, outputting a dense rasterized feature map with the same size as the UAV image. This solves the point cloud sparsity problem and realizes the conversion from ground point cloud data to a dense feature map, represented as:

[0103] ;

[0104] In the formula, This represents the point cloud rasterization function, which transforms sparse point clouds into dense feature maps. Characterizing point cloud data The rasterized feature map obtained after multilayer perceptron transformation has a dimension of [dimensionality missing]. ;

[0105] Represents ground point cloud data, with dimensions of N represents the number of point clouds, and the point cloud data. It includes 4-dimensional features of XYZ coordinates and reflection intensity, which are derived from lidar or oblique photogrammetry. The reflection intensity is derived from lidar echo intensity or multispectral sensor.

[0106] This represents a multilayer perceptron, used to learn feature representations of point clouds;

[0107] This represents the height of the drone image, which is used as the height dimension of the output feature map.

[0108] This represents the width of the drone image, which is used as the width dimension of the output feature map.

[0109] The number of channels in the output feature map is determined by the network structure of the multilayer perceptron.

[0110] In one implementation, the reflection intensity in the point cloud data collected by lidar comes from the laser echo intensity, reflecting the material properties of ground objects, such as high reflectivity of concrete and low reflectivity of vegetation. The point cloud data from oblique photogrammetry may derive intensity values ​​from multispectral sensors, such as near-infrared reflectivity.

[0111] In one implementation, the structure of the multilayer perceptron is as follows: input layer (4-dimensional), fully connected layer (64-dimensional, ReLU activation function), fully connected layer (128-dimensional, ReLU activation function), output layer (… (Dimension), preferably setting the number of channels in the output feature map. To balance computational efficiency and feature representation capability, the multilayer perceptron learns the mapping from point cloud to raster features through linear transformation and activation function.

[0112] 2) Calculation of adaptive resolution registration parameters:

[0113] A learnable upsampling operation is performed on satellite remote sensing data to match its spatial size with that of UAV imagery, and a learnable downsampling operation is performed on UAV data to maintain its spatial size. The upsampled satellite data, downsampled UAV data, and rasterized point cloud features are then concatenated by channel stitching. The number of channels is compressed using a 1×1 convolution, and a registration parameter matrix is ​​output. This achieves dynamic adjustment of the spatial resolution and feature fusion of multi-source data, as shown below:

[0114] ;

[0115] In the formula, This represents the registration parameter matrix, with an output size of [missing information]. This generates spatial transformation parameters for each pixel. For example, it dynamically adjusts the pixel position using three methods: horizontal translation, vertical translation, and rotation, to align with multi-source data, corresponding to the registration parameter matrix. 3 channels;

[0116] This indicates a 1×1 convolution operation that performs cross-modal compression on the concatenated features.

[0117] This indicates an upsampling operation, which converts satellite remote sensing data... Space dimensions from Upsampling ;

[0118] Indicates the altitude of the satellite image;

[0119] Indicates the width of the satellite image;

[0120] Indicates the number of bands in the satellite imagery;

[0121] This indicates a downsampling operation on UAV data. Perform feature transformation while maintaining spatial dimensions. ;

[0122] This indicates a channel splicing operation;

[0123] This represents satellite remote sensing data, acquired via satellite, and is in the format of... Multiband raster data;

[0124] This represents drone data, with dimensions of [size missing]. Data collected via drone aerial photography is in the following format: RGB image.

[0125] It should be noted that by aligning the resolution through upsampling and downsampling, splicing and fusing multimodal information, and using 1×1 convolution to compress features, dynamic resolution adjustment and feature fusion are achieved, thereby improving registration accuracy and solving the problem of spatiotemporal inconsistency of multi-source data.

[0126] In addition, there is a semantic gap between multi-source data. Conventional methods use independent branches for feature extraction, resulting in insufficient synergy and an inability to fully utilize cross-modal information. A single feature extraction network is difficult to adapt to the spectral characteristics of satellite remote sensing data, the spatial details of UAV data, and the geometric information of ground point clouds at the same time, thus resulting in insufficient feature representation.

[0127] A hybrid structure combining a shared backbone network and modality-specific branches is constructed. Cross-modal knowledge transfer is achieved through shared feature maps, and modality-specific operations are combined to enhance the advantageous features of each data source, thereby realizing the collaborative extraction of multi-scale features and improving feature representation capabilities. Specifically, using the registration parameter matrix, the features of each modality of data in the multi-source collaborative sensing array (each modality of data features includes satellite augmentation features, UAV augmentation features, and ground fusion features) are obtained. The specific steps are as follows:

[0128] 1) Shared feature map extraction:

[0129] The registration parameter matrix is ​​input into the ResNet-50 backbone network, and cross-modal shared feature maps are extracted through convolutional residual blocks and skip connections. The output is a shared feature map with a size of 1 / 4 of the original UAV image, providing a general spatial semantic foundation for modality-specific branches.

[0130] ;

[0131] In the formula, Represents a shared feature map, with a size of It is a basic feature shared across modalities and contains general spatial semantic information.

[0132] This represents the ResNet-50 backbone network, which contains 50 convolutional residual blocks and mitigates gradient vanishing through skip connections, used to extract deep shared feature maps.

[0133] In practical implementation, the shared feature map is the feature processed by the ResNet-50 backbone network, and its size is [size missing]. By using the convolution stride of the ResNet-50 backbone network, a 4x downsampling is achieved, reducing the feature map size to 1 / 4 of its original size, thereby reducing computational cost and expanding the receptive field.

[0134] 2) Satellite remote sensing data feature enhancement:

[0135] A 3×3 convolution transformation is performed on the shared feature map. The weight vector generated from the satellite remote sensing data via a channel attention mechanism is broadcast to the same spatial dimension. The shared feature map after the convolution transformation is then weighted using a Hadamard product operation. The spectral sensitivity of the satellite enhanced features is expressed as:

[0136] ;

[0137] In the formula, This represents satellite-enhanced features, improves spectral sensitivity, and maintains the output size by using 3×3 convolutions to share feature maps. Consistent, the number of channels has been adjusted to 256;

[0138] This represents a 3×3 convolution operation, used for feature transformation.

[0139] This represents the Hadamard product, which is an element-wise multiplication operation used for feature weighting.

[0140] This represents the channel attention mechanism, which generates a feature weight map.

[0141] In practice, the channel attention mechanism is implemented by processing satellite remote sensing data. Spatial downsampling is performed using convolution with a stride of 4. The dimension is then determined by global average pooling, resulting in an output dimension of [dimensionality]. The feature vectors are then processed through two fully connected layers for feature mapping. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid activation function. The number of output channels is set to 256, generating channel weight vectors. These channel weight vectors are then broadcast to... The spatial dimension is multiplied channel-by-channel with the downsampled feature map.

[0142] 3) Enhanced features of drone data:

[0143] A 3×3 convolution transformation is performed on the shared feature map. The multi-scale spatial features extracted from the UAV data through dilated spatial pyramid pooling are then concatenated with the shared feature map after the convolution transformation to fuse multi-scale spatial detail information, as shown below:

[0144] ;

[0145] In the formula, This indicates that the drone's enhanced features are fused with multi-scale spatial details, and the number of output channels for the hollow spatial pyramid pooling is set to 256, preserving and sharing feature maps. Consistent;

[0146] This represents the pyramid pooling of void spaces, extracting multi-scale spatial features.

[0147] In its implementation, the dilated spatial pyramid pooling is based on deformable convolutional operations to adapt to the irregular shapes of illegal structures. Specifically, it extracts multi-scale features through parallel convolutional layers, including three dilated convolutions (dilation rates of 1, 6, and 12) and a global average pooling layer. The dilation rates of the three dilated convolutions are set to 1, 6, and 12, respectively, and the size of each convolutional kernel is [size missing]. The output feature maps are spliced ​​together and multi-scale information is fused.

[0148] In practical implementation, The convolution operation used in this project employs 128 filters. The number of channels was compressed from 256 to 128. The number of output channels for each parallel convolutional layer in the item is set to 32, and the concatenation result is... The channel is then used to obtain the stitched-together drone augmentation features. The number of channels is 256.

[0149] 4) Ground data feature fusion:

[0150] A 3×3 convolution transformation is performed on the shared feature map, and the rasterized point cloud features are downsampled to the same size as the shared feature map through a 4×4 convolution. The ground geometric information and the shared feature map are then fused through element-wise addition, as shown below:

[0151] ;

[0152] In the formula, Representing ground fusion features, injecting point cloud geometric information, and rasterizing feature maps. Performing 4×4 convolution downsampling with a stride of 4, the spatial size then changes from... Become The number of channels was adjusted to 256 to maintain the shared feature map. Consistent.

[0153] It should be noted that by reducing parameters through a shared backbone network and preserving data advantages in modality-specific branches, and based on shared feature maps, each branch specifically enhances spectral, spatial, and geometric information, thereby achieving complementary enhancement of multimodal features. By reducing parameter redundancy through a shared backbone network and strengthening spectral, spatial, and geometric information in modality-specific branches, the accuracy and generalization ability of illegal construction identification are improved.

[0154] In addition, conventional feature fusion methods usually use simple splicing or addition operations, which fail to consider the hierarchical differences and semantic importance between multi-source features. This results in important information being diluted or ignored during the fusion process, and the inability to dynamically focus on key features related to illegal construction, thus leading to feature hierarchical mismatch problems.

[0155] This step employs a gated cross-scale attention mechanism. By dynamically calculating the gating coefficients of each modality feature and performing a weighted summation of the features after depthwise separable convolution, adaptive feature fusion is achieved, improving the accuracy of feature fusion. Specifically, it utilizes the features of each modality data to quantify the weights of each modality data; and uses the features and weights of each modality data to obtain multimodal fusion features. The specific steps are as follows:

[0156] 1) Calculation of gating coefficient:

[0157] Modal-specific features and shared feature maps are concatenated along the channel dimension, compressed by layer normalization and 1×1 convolution, and then superimposed with modality position encoding tensors. Dynamic gating coefficients are generated using the Sigmoid function to quantify the weights of each modality in feature fusion, as shown below:

[0158] ;

[0159] In the formula, The position encoding tensor for modality m has an output dimension of ;

[0160] This represents the Sigmoid activation function, with an output range between 0 and 1. The term represents the adaptive weights of mode m in the fusion;

[0161] The representation layer normalization operation is used to stabilize the training process and accelerate convergence;

[0162] m represents the modal index, and its value range is... These correspond to satellite, drone, and ground data, respectively. This represents a set of three modes: satellite data, UAV data, and ground data. Sat represents the satellite data mode, dro represents the UAV data mode, and gro represents the ground data mode.

[0163] z represents the gated feature tensor of mode m, with dimension . Each spatial location corresponds to a weight value, and each mode is calculated independently;

[0164] This represents a specific feature corresponding to mode m. When m is 'sat', it corresponds to a satellite augmentation feature; when m is 'dro', it corresponds to a UAV augmentation feature. When m is gro, it corresponds to the ground fusion feature. All sizes , The number of channels for each modality feature is set to 256.

[0165] In the specific implementation, the position encoding tensor PosEnc(m) of mode m is used to distinguish between three modes: satellite data, UAV data, and ground data. The position encoding vector corresponding to the satellite data mode is defined as [1,0,0], the position encoding vector corresponding to the UAV data mode is [0,1,0], and the position encoding vector corresponding to the ground data mode is [0,0,1]. Then, the vector is mapped to a scalar through a fully connected layer, and the scalar is broadcast to the size. The dimension of generation and Tensors of the same dimension, i.e., the position encoding tensor PosEnc(m) of mode m.

[0166] 2) Feature fusion calculation:

[0167] For each modality feature, group normalization and depthwise separable convolution are performed to extract local features. The results are then multiplied element-wise with the corresponding gating coefficients, and the weighted sum of all modality features is obtained to obtain the multimodal fusion feature, expressed as:

[0168] ;

[0169] In the formula, This represents multimodal fusion features, aggregating key multimodal information, with dimensions of [missing information].

[0170] ;

[0171] Group-Normalization involves dividing the channels into multiple groups, and normalizing within each group. For example, the channel dimension of the input feature is divided into 32 groups.

[0172] This represents a summation operation, which accumulates the weighted features of all modalities;

[0173] This indicates a depthwise separable convolution, which consists of depthwise convolution and pointwise convolution. It is used to extract local features of each modality. Specifically, depthwise convolution is performed by independently convolving each channel, and then pointwise convolution is performed by fusing the channels through 1×1 convolution. This achieves the preservation of spatial information with a small number of parameters. The number of output channels is set to 512.

[0174] like Figure 2 As shown, in one embodiment, robustness analysis is performed under different environmental complexities to verify the stability and adaptability of the method under complex environmental conditions. The experiment simulates different environmental scenarios from simple to complex, and systematically analyzes the relationship between detection accuracy and environmental complexity. Figure 2The vertical axis represents detection accuracy, ranging from 0 to 1, representing the proportion of correct detections. The horizontal axis represents environmental complexity, a continuous value between zero and one; a higher value indicates a more complex environment, such as increased occlusion, lighting variations, or terrain undulations. The scatter plot shows that as environmental complexity increases, the detection accuracy of all methods decreases, but the decrease in accuracy for this technique is significantly less than that of conventional methods. The difference in the slope of the fitted line indicates that this technique is less sensitive to environmental complexity, demonstrating the adaptive nature of the cross-scale attention fusion module. This module dynamically adjusts the weights of each modality feature through a gating mechanism, maintaining effective fusion of key information even in complex environments. In the scatter plot, the data points for this technique are more concentrated in the higher-precision region with less dispersion, indicating that it maintains stable performance output under various complex conditions.

[0175] Step 130: Transform the multimodal fusion features to generate a query matrix, a key matrix, and a value matrix; The Transformer model introduces a preset local constraint mask matrix that constrains the scope of attention calculation in the standard attention mechanism that includes the query matrix, key matrix, and value matrix, and outputs local-global interactive attention features.

[0176] It should be noted that conventional Transformers often ignore local structural features during long-distance modeling, resulting in insufficient sensitivity to local geometric features such as illegal walls and roofs, making it difficult to capture subtle structural changes in illegal construction areas.

[0177] This step employs a local-global interactive encoder. By utilizing local constraint masks and feature dimensionality reduction operations, it forces the preservation of structural relationships within the local window when calculating global attention, thereby enhancing the sensitivity to the local geometric features of illegal structures. The specific steps are as follows:

[0178] 1) Querying, key and value generation:

[0179] A linear transformation is performed on the multimodal fusion features to generate a query matrix. After dimensionality reduction via 4×4 convolution, the multimodal fusion features are then subjected to linear transformations to generate key and value matrices, reducing the computational complexity of the attention layer. This is represented as follows:

[0180] ;

[0181] ;

[0182] ;

[0183] In the formula, Q represents the query matrix, which is used to calculate the attention weights;

[0184] K represents the key matrix, which is used to calculate similarity with the query matrix;

[0185] V represents the value matrix, used for weighted summation to generate output features;

[0186] The weight matrix representing the query transformation is a trainable parameter;

[0187] The weight matrix representing the key transformation is a trainable parameter;

[0188] The weight matrix representing the value transformation is a trainable parameter;

[0189] This represents a feature dimensionality reduction operation, implemented through 4×4 convolution. It reduces the spatial size of the feature map and increases the number of channels, reducing the number of key-value pairs by 16 times, thereby reducing the computational cost of subsequent attention calculations.

[0190] 2) Definition of local constraint mask (preset local constraint mask matrix):

[0191] A mask matrix is ​​constructed by comparing the Euclidean distance of pixel coordinates with a local neighborhood threshold. When the distance is less than the threshold, the mask value is 0, allowing attention calculation; otherwise, it is negative infinity, suppressing attention and forcibly preserving the structural relationships within the local window. This is represented as:

[0192] ;

[0193] In the formula, This represents the value of the element in the i-th row and j-th column of the preset local constraint mask matrix, which is used to adjust the attention weights; Represents the coordinate vector of the i-th pixel, which contains two-dimensional spatial coordinates. The coordinate vector refers to the two-dimensional coordinates of the pixel on the feature map. Represents the coordinate vector of the j-th pixel; This represents the L2 norm, calculated in the same way as the Euclidean distance. This represents a local neighborhood threshold, controlling the effective range of the mask, such as... This represents a local window with a radius of 7 pixels.

[0194] In the specific implementation, when the coordinate vector of the i-th pixel... The coordinate vector of the j-th pixel Distance less than the local neighborhood range threshold When the mask value is 0, attention computation is allowed; otherwise, it is not. Suppress attention weights to enhance local feature interactions.

[0195] 3) Local-Global Attention Calculation:

[0196] A pre-defined local constraint mask matrix is ​​introduced into the standard attention mechanism. The scaled dot product similarity calculation is superimposed on the mask, and after normalization using the Softmax function, the value matrix is ​​weighted and summed to output the local-global interactive attention features, represented as:

[0197] ;

[0198] In the formula, This represents local-global interactive attention features, used to enhance the features of illegal structures;

[0199] Let K be the transpose of the key matrix. This represents the product of the query matrix and the transpose of the key matrix, used to calculate the similarity matrix;

[0200] This represents the dimension of the key vector, used to scale the similarity matrix and prevent gradient vanishing; a value of 512 is preferred.

[0201] This represents a preset local constraint mask matrix, used to limit the computational scope of attention and ensure priority for local structures. The element in the i-th row and j-th column is ;

[0202] This represents the normalized exponential function, which converts the similarity matrix into a probability distribution.

[0203] Step 140: Obtain a pixel-level illegal construction probability map by utilizing multimodal fusion features and local-global interactive attention features; obtain classification probability data by utilizing the pixel-level illegal construction probability map and the data features of each modality.

[0204] It should be noted that illegal construction identification requires the simultaneous completion of localization and classification tasks. Conventional single-task heads are prone to feature coupling, causing localization and classification tasks to interfere with each other, thereby affecting the identification accuracy and generalization ability.

[0205] This step constructs a dual-branch decoupling head, which separates the feature learning paths for localization and classification to avoid interference between tasks and achieve collaborative optimization of pixel-level localization and image-level classification. The specific steps are as follows:

[0206] 1) Transformation feature generation:

[0207] The residuals of the local-global attention output features and the multimodal fusion features aligned by 1×1 convolution are added together, and the attention fusion transformation features are output through layer normalization, as follows:

[0208] ;

[0209] In the formula, This represents the attention fusion transformation feature, which fuses local-global attention with multimodal features;

[0210] The representation layer normalization operation is used to stabilize the training process and accelerate convergence.

[0211] It should be noted that, The term aligns multimodal fusion features through 1×1 convolution. Local-Global Interactive Attention Features The number of channels.

[0212] 2) Positioning head calculation:

[0213] After performing two 3×3 convolution operations on the transformed features, the image size is restored to the original size through a 4x upsampling. The resulting pixel-level illegal construction probability map is then output using the Sigmoid activation function, as shown below:

[0214] ;

[0215] In the formula, This represents a pixel-level probability map of illegal construction (probability map output by the positioning head), with dimensions of...

[0216] That is, the output is A two-dimensional matrix, where the range of each element is 1. This indicates the probability that the location is an illegal structure;

[0217] This indicates an upsampling operation, specifically a 4x upsampling operation that reduces the feature map size from... Restore to Make the output probability map The dimension is Each pixel has a value range of [0,1], representing the probability that the location belongs to an illegal building.

[0218] In the specific implementation, the two-layer calculation of the positioning head output is used. The number of channels in each convolutional layer is 256, and the output dimension is... .

[0219] In practice, the upsampling operation uses a transposed convolution with a kernel size of 4×4 and a stride of 4, or a bilinear interpolation followed by a 1×1 convolution, to eliminate the spatial information loss caused by downsampling.

[0220] 3) Classification head calculation:

[0221] Global average pooling is performed on each modality feature to extract a global vector, which is then concatenated with the satellite channel attention weight vector and the UAV multi-scale global vector to form a comprehensive feature vector. This vector is then processed by a multilayer perceptron and a softmax function to output the image-level classification probability, as shown below:

[0222] ;

[0223] In the formula, The classification head outputs a two-dimensional real vector representing the classification probability, indicating whether the image belongs to an illegal or non-illegal building. This indicates that the probability of it being a non-illegal structure is 20%, and the probability of it being an illegal structure is 80%.

[0224] The channel attention weight vector represents the satellite features, which is used to enhance satellite features. Perform global average pooling and use fully connected layers to generate vectors of size 1×1×256;

[0225] The multi-scale global vector representing the characteristics of the UAV is extracted through the global average pooling branch of the hollow spatial pyramid pooling module, and the output vector has a size of 1×1×256.

[0226] It is a fully connected neural network for the classification layer;

[0227] This indicates a vector concatenation operation that aggregates multimodal global features into a comprehensive feature vector;

[0228] This represents the global average pooling operation, which aggregates spatial features into a global feature vector.

[0229] In the actual implementation, the dimension of the concatenated feature vector is... The input layer of the fully connected neural network for classification is transformed. The input layer of the fully connected neural network for classification has 1280 dimensions, and the hidden layer is a fully connected layer with a dimension of 512 dimensions, using the ReLU activation function. A dropout layer is set after the hidden layer with a dropout rate of 0.5. The output layer of the fully connected neural network for classification is a fully connected layer with a dimension of 2 dimensions and no activation function.

[0230] Step 150: Use the pixel-level illegal construction probability map to obtain the structural loss and localization loss; use the classification probability data to obtain the classification loss; then use weighted summation to obtain the overall loss; use the overall loss to iteratively update the Transformer model until the preset iteration conditions are met to obtain the trained Transformer model.

[0231] This step can be specifically described as follows:

[0232] Calculate the loss function:

[0233] Conventional cross-entropy loss ignores the geometric characteristics of illegal structures, focusing only on pixel-level classification accuracy while neglecting the geometric consistency of boundary contours, resulting in blurred predicted boundaries that do not match the actual illegal structures.

[0234] This step employs a composite loss function, combining localization loss, classification loss, and structural loss. Through weighted summation using balance coefficients, it forces the predicted boundary to maintain curvature consistency with the actual illegal building contour, thereby improving recognition accuracy. The specific steps are as follows:

[0235] 1) Structural loss calculation:

[0236] Within the pixel set of the illegal construction area, the mean L1 norm of the difference between the localization prediction result and the second-order gradient extracted by the Laplacian operator from the true label is calculated to quantify the curvature consistency between the predicted boundary and the true contour, expressed as:

[0237] ;

[0238] In the formula, Structural loss is used to measure the geometric consistency between the predicted boundary and the true boundary.

[0239] This represents the set of pixels representing the illegal construction area, and is the actual label. The set of pixels with a median value of 1 represents manually marked areas of illegal construction.

[0240] The cardinality represents the set of pixels in the illegal construction area, that is, the total number of pixels contained in the illegal construction area;

[0241] Represents the pixel position index, corresponding to two-dimensional coordinates. The i-th row and j-th column represents;

[0242] This represents the Laplacian operator, used to calculate the second-order gradient of an image and extract boundary curvature features;

[0243] Indicates the first The localization prediction value at each pixel represents the probability map output by the localization head. In the Predicted value at each pixel;

[0244] Represents the true label, with dimensions as follows: , Indicates the first The true label value at each pixel, i.e., the true value of the manually labeled illegal construction area, with each element taking a value of... This indicates whether the location is an illegal structure;

[0245] The L1 norm is used to calculate the absolute difference to enhance robustness to outliers.

[0246] 2) Calculation of positioning loss:

[0247] By combining positive and negative sample weights, focus loss parameters, and class balancing strategies, the binary cross-entropy loss between the location prediction probability map and the true label is calculated to enhance sensitivity to illegal building boundaries and small targets, as expressed below:

[0248] ;

[0249] In the formula, This represents the binary cross-entropy loss function, used to measure the difference between the probability distribution of binary classification predictions and the true distribution.

[0250] It is a logarithmic function, with the default base being the natural constant;

[0251] The weights for positive samples are calculated as follows: ;

[0252] The negative sample weights are calculated as follows: ;

[0253] The first loss contribution parameter is used to suppress the loss contribution of easily classified samples, and is preferably set to 0.25;

[0254] The second loss contribution parameter is used to suppress the loss contribution of easily classified samples, and is preferably set to 2.

[0255] 3) Classification loss calculation:

[0256] For the smoothed true classification labels, calculate the cross-entropy loss of the fused classification predictions, and superimpose a KL divergence penalty term weighted by the confidence of each modality to ensure multimodal collaborative decision-making, expressed as:

[0257]

[0258] ;

[0259] In the formula, This represents the modal confidence cross-entropy loss function, used to measure the difference between the probability distribution of multi-class predictions and the true distribution;

[0260] This is the probability value of the c-th category in the classification prediction output;

[0261] c is the category index, and its value range is... Where 0 represents a non-illegal structure and 1 represents an illegal structure;

[0262] The value of the c-th category in the smoothed true labels is calculated as follows: ;

[0263] The label smoothing parameter is set to a small value to prevent the model from overfitting to the real labels. By adjusting the one-hot distribution of the real labels to soft labels, the generalization ability is increased. It is preferred to set it to 0.1.

[0264] The confidence weight for mode m is calculated as follows:

[0265] The higher the confidence level, the greater the weight of that mode in the penalty term;

[0266] Let be the cross-entropy classification loss using only mode m, representing the reliability of that mode;

[0267] For using only modes Cross-entropy classification loss;

[0268] This indicates a modal index that is distinct from m;

[0269] The penalty intensity factor controls the penalty intensity and promotes multimodal collaborative decision-making; it is preferably set to 0.05.

[0270] The Kullback-Leibler divergence measures the output of the classification head. Compared to classification head output using only modality m The differences between them;

[0271] The classification head output using only mode m is calculated using an auxiliary classification head with the same structure as the main classification head.

[0272] The true category label is a one-hot encoded two-dimensional vector; for example, "non-illegal construction" is... illegal construction .

[0273] It should be noted that, The term is a modal divergence penalty term, which is fused with the classification head output using the KL divergence metric. Compared to classification head output using only modality m The distribution differences, combined with the confidence weight of mode m. The weighting is such that modalities with higher confidence have larger weights. The loss function forces the fusion output to take into account the information of all modalities during training, thereby making the multimodal fusion output consistent with the output of each individual modality. This enhances the robustness and generalization ability of the model by utilizing multimodal consistency.

[0274] 4) Weighted summation of composite losses:

[0275] Preset balancing coefficients are applied to the localization loss, classification loss, and structural loss respectively, and the weighted summation yields the overall composite loss function as the model optimization objective, expressed as:

[0276] ;

[0277] In the formula, Loss represents the overall loss function, which serves as the optimization objective for model training;

[0278] This represents the balance coefficient for positioning loss, controlling the weight of positioning accuracy in the total loss, and has a value of 0.6.

[0279] This represents the balance coefficient of classification loss, controlling the weight of classification accuracy in the total loss, and has a value of 0.3.

[0280] The balance coefficient represents the structural loss and controls the weight of geometric consistency in the total loss; its value is 0.1.

[0281] Machine learning model training iterations and parameter updates:

[0282] Machine learning model training employs an iterative optimization approach, updating parameters based on the overall composite loss function;

[0283] The training process uses the Adam optimizer with an initial learning rate of 0.001, and combines a cosine annealing scheduler to dynamically adjust the learning rate to balance convergence speed and stability.

[0284] In each training cycle, batch data is randomly sampled from the training set (batch size is set to 16), multi-source data (satellite, UAV and ground point cloud) and corresponding labels are input, forward propagation is used to calculate the output probability map of the localization head and the output probability of the classification head, and then the composite loss (including localization loss, classification loss and structural loss) is calculated.

[0285] During backpropagation, the gradient of the loss function with respect to each model parameter is calculated, and the parameters are updated using the gradient descent principle. The optimization objective is to minimize the composite loss. The performance on the validation set is monitored during training.

[0286] After each cycle is completed, the loss and accuracy metrics are calculated on the validation set. If the validation loss does not decrease within 10 consecutive cycles, the early stop mechanism is triggered to terminate training and prevent overfitting.

[0287] At the same time, the model parameters at which the validation performance is optimal are saved as the final model.

[0288] The maximum number of training cycles is set to 100. If early stop is not triggered, training will stop after reaching the maximum number of cycles.

[0289] The parameter update process emphasizes gradient clipping (with a threshold set to 1.0) to avoid gradient explosion and ensure training stability.

[0290] Step 160: Input the preprocessed real-time multi-source collaborative sensing array into the trained Transformer model to obtain the illegal construction recognition result.

[0291] This step can be specifically described as follows:

[0292] After the model training is completed, the illegal construction intelligent identification process first preprocesses the newly collected air-space-ground multi-source data, including radiometric correction of satellite data, geometric correction of UAV data and denoising of ground point clouds, and unifies the spatiotemporal reference to be consistent with the training data.

[0293] The preprocessed data is input into the trained machine learning model. The model sequentially executes spatiotemporal registration, multi-scale feature collaborative extraction, cross-scale attention fusion, local-global Transformer interaction, and multi-task decoupling module, and outputs pixel-level illegal construction probability map and image-level classification probability.

[0294] Based on the location results, a threshold (such as 0.5) is applied to the probability map for binarization to generate a mask for the illegal construction area. This mask is then combined with geographic coordinates and mapped onto the actual map to identify the spatial location and extent of the illegal construction.

[0295] Based on the classification results, if the classification probability is greater than 0.5, the area is determined to have illegal construction; otherwise, it is considered non-illegal construction.

[0296] like Figure 3As shown, in one embodiment, a comparative analysis of the boundary recognition quality of different methods is conducted, focusing on the accuracy of illegal construction boundary positioning. The Intersection over Union (IoU) ratio (vertical axis) is used as the evaluation index, which measures the degree of overlap between the predicted boundary and the real boundary (range 0-1, higher is better). Experiments compare four methods: satellite data only, satellite + UAV, conventional multi-source fusion, and the method of this application. Box plots visually display the data distribution through the median line, box range (25%-75th percentile), and discrete points. Experimental results show that the satellite data only method has the lowest median boundary IoU and a large number of low-value outliers; the conventional multi-source fusion scheme improves upon the dual-source scheme but still has many outliers; the method of this application has the highest median, with the boxes concentrated in the upper interval and a significant reduction in outliers. This indicates that the structural loss function unique to this application enforces the consistency of boundary curvature, and the local-global Transformer module enhances the sensitivity of geometric features, making the predicted boundary more closely match the actual illegal construction outline.

[0297] like Figure 4 As shown, in one embodiment, the detection accuracy under different scales of illegal constructions was analyzed, focusing on the impact of changes in the scale of illegal constructions on the detection accuracy, aiming to verify the stability and adaptability of the proposed method in the identification of illegal constructions at different scales. The experiment divided illegal constructions into five scale levels according to area, from less than 50 square meters to greater than 500 square meters, and compared the changes in detection accuracy of four methods in different scale ranges. The experimental configuration simulated the diversity of illegal construction scales in real-world scenarios, with each scale level containing a sufficient number of samples to ensure statistical significance. The trend of the line graph shows that the detection accuracy of all methods increases with the increase in the scale of the illegal construction, but the improvement of the proposed method is the most stable and consistently remains at the highest level. Methods using only satellite data perform poorly in the detection of small-scale illegal constructions, which is directly related to their insufficient spatial resolution; methods using only UAV data have certain advantages in small-scale detection, but their growth is slow in large-scale scenarios due to the lack of spectral information. This application's method utilizes air-space-ground data collaboration to capture details using high resolution drones in small-scale illegal constructions, and combines multispectral information from satellites to analyze material properties in large-scale illegal constructions. It also incorporates geometric features from ground point clouds, thus achieving optimal detection across the entire scale. The curve of this method in the figure not only has the highest position but also the smallest fluctuation, demonstrating the method's robustness to scale variations.

[0298] Based on the above description, this application proposes an intelligent identification method for illegal constructions based on integrated air-space-ground perception, which is innovative in the following aspects compared with the prior art:

[0299] 1. By utilizing multi-source data (satellite, UAV, ground point cloud) from the air-space-ground integrated system, adaptive resolution registration and feature fusion were used to solve the problem of spatiotemporal differences between different data sources and improve the accuracy of multimodal data collaborative perception.

[0300] 2. A cross-scale attention mechanism is adopted to dynamically calculate the gating coefficients of each modality feature and perform weighted fusion, which fully considers the hierarchical differences and semantic importance between features and effectively avoids the information dilution problem in traditional splicing methods.

[0301] 3. The introduction of a local-global interactive encoder and Transformer structure enhances the model's sensitivity to the local geometry of illegal structures (such as walls and roofs) and improves its ability to identify subtle structural changes.

[0302] 4. A multi-task decoupling module is adopted to separate the localization and classification tasks, reducing interference between tasks and optimizing pixel-level localization accuracy and image-level classification accuracy.

[0303] In addition, this application embodiment also provides a non-volatile computer storage medium storing executable instructions, which, when executed, implement the above-described method for intelligent identification of illegal buildings based on integrated air-space-ground perception.

[0304] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent identification of illegal constructions based on integrated air-space-ground collaborative perception, characterized in that, The method includes: Collect an initial multi-source collaborative sensing array for a preset historical time period, and obtain the multi-source collaborative sensing array after preprocessing; obtain the illegal construction labeling data of the multi-source collaborative sensing array; The satellite remote sensing data, UAV data and ground point cloud data in the multi-source collaborative sensing array are fused to obtain the registration parameter matrix; the registration parameter matrix is ​​used to obtain the features of each modality in the multi-source collaborative sensing array; the features of each modality are used to quantify the weight of each modality; and the features and weights of each modality are used to obtain the multi-modal fusion features. The multimodal fusion feature transformation generates a query matrix, a key matrix, and a value matrix; The Transformer model introduces a pre-defined local constraint mask matrix to constrain the scope of attention computation within the standard attention mechanism, which includes query, key, and value matrices. It outputs local-global interactive attention features, specifically including: (via the formula:) Obtain local-global interactive attention features; in, This represents the local-to-global interactive attention features; Represents the query matrix; Represents the key matrix; Represents a value matrix; Key matrix transpose, This represents the product of the query matrix and the transpose of the key matrix; Indicates the dimension of K; This represents a preset local constraint mask matrix; This represents the normalized exponential function; By utilizing multimodal fusion features and local-global interactive attention features, a pixel-level illegal construction probability map is obtained. Then, using the pixel-level illegal construction probability map and features from each modality, classification probability data is obtained. This includes constructing a dual-branch decoupling head. The specific steps are as follows: 1) Transformation Feature Generation: The residuals of the local-global attention output features and the multimodal fusion features aligned by 1×1 convolution are added together, and the attention fusion transformation features are output through layer normalization. 2) Positioning head calculation: After performing two layers of 3×3 convolution operations on the transformed features, the original image size is restored by 4 times upsampling, and a pixel-level illegal construction probability map is output through the Sigmoid activation function; 3) Classification head calculation: Global average pooling is performed on each modality feature to extract a global vector, which is then concatenated with the satellite channel attention weight vector and the UAV multi-scale global vector to form a comprehensive feature vector. The image-level classification probability is then output through a multilayer perceptron and a Softmax function. Using pixel-level illegal construction probability maps, structural loss and localization loss are obtained; using classification probability data, classification loss is obtained; then weighted summation is used to obtain the overall loss; the overall loss is used to iteratively update the Transformer model until the preset iteration conditions are met, and a trained Transformer model is obtained. The preprocessed real-time multi-source collaborative sensing array is input into the trained Transformer model to obtain the illegal construction identification results.

2. The intelligent identification method for illegal construction based on integrated air-space-ground perception as described in claim 1, characterized in that, The multi-source collaborative sensing array includes: satellite remote sensing data, UAV data, and ground point cloud data; The initial multi-source collaborative sensing array is acquired, and after preprocessing, the resulting multi-source collaborative sensing array includes: Acquire satellite remote sensing data through satellite platforms; By using a drone equipped with an RGB camera to take aerial photos at a preset low altitude, high spatial resolution drone data can be obtained. Ground point cloud data containing three-dimensional coordinates and reflection intensity is generated by collecting data through ground-based lidar scanning or oblique photogrammetry. Radiometric calibration and atmospheric correction are performed on satellite remote sensing data; image stitching and geometric correction are performed on UAV data; and noise reduction and filtering are performed on ground point cloud data. Geographic Information System (GIS) tools are used to align processed satellite remote sensing data, UAV data, and ground point cloud data to the same coordinate system and the same timestamp.

3. The intelligent identification method for illegal construction based on integrated air-space-ground perception as described in claim 1, characterized in that, The satellite remote sensing data, UAV data, and ground point cloud data in the multi-source collaborative sensing array are fused to obtain the registration parameter matrix, specifically including: The point cloud data features are transformed into dense feature map data using a multilayer perceptron. Learnable upsampling is performed on satellite remote sensing data to match the spatial size of UAV imagery, and learnable downsampling is performed on UAV data to keep the spatial size of UAV data unchanged. The upsampled satellite remote sensing data, downsampled UAV data, and dense feature map data are then stitched together, and the number of channels is compressed by 1×1 convolution to output the registration parameter matrix.

4. The intelligent identification method for illegal construction based on integrated air-space-ground perception as described in claim 1, characterized in that, The features of each modality of data include satellite augmentation features, UAV augmentation features, and ground fusion features; Using the registration parameter matrix, the features of each modality in the multi-source collaborative sensing array are obtained, specifically including: The registration parameter matrix is ​​input into the ResNet-50 backbone network, and cross-modal shared feature maps are extracted through convolutional residual blocks and skip connections. The output size is reduced to 1 / 4 of the original UAV data shared feature map. A 3×3 convolution transformation is performed on the shared feature map, and the weight vector generated by the channel attention mechanism of the satellite remote sensing data is broadcast to the same spatial dimension. The shared feature map after convolution transformation is weighted by the Hadamard product operation to obtain satellite enhanced features. A 3×3 convolution transformation is performed on the shared feature map. The multi-scale spatial features extracted from the UAV data through dilated spatial pyramid pooling are then concatenated with the shared feature map after the convolution transformation to obtain UAV enhanced features. A 3×3 convolution transformation is performed on the shared feature map, and the rasterized point cloud features are downsampled to the same size as the shared feature map through a 4×4 convolution. Then, the ground fusion features are obtained by adding elements one by one.

5. The intelligent identification method for illegal construction based on integrated air-space-ground perception as described in claim 1, characterized in that, The weights for each modality of data include: satellite augmentation feature weights, UAV augmentation feature weights, and ground fusion feature weights; By utilizing the features of each modal data, the weights of each modal data are quantified, specifically including: The current modal data features and the shared feature map are concatenated along the channel dimension. After layer normalization and 1×1 convolution compression, the modal position encoding tensor corresponding to the current modal data features is superimposed. The dynamic gating coefficients corresponding to the current modal data features are generated by the Sigmoid function. The modal data weights corresponding to the current modal data features are quantized by the dynamic gating coefficients.

6. The intelligent identification method for illegal construction based on integrated air-space-ground perception as described in claim 1, characterized in that, Multimodal fusion features are obtained by utilizing the features and weights of each modality's data, specifically including: Group normalization and depthwise separable convolution are performed on each modal data feature to extract the preset local features of each modal data feature. The preset local features of the modal data feature are multiplied element-wise with the modal data weights corresponding to the modal data features to obtain the product results. Then, the weighted sum is obtained to obtain the multimodal fusion features.

7. The intelligent identification method for illegal construction based on integrated air-space-ground perception as described in claim 1, characterized in that, Pixel-level illegal construction probability maps are obtained by utilizing multimodal fusion features and local-global interactive attention features; classification probability data are obtained using the pixel-level illegal construction probability maps and features from each modality, specifically including: The local-global interactive attention features and the multimodal fusion features aligned by 1×1 convolution are residually added, and the addition result is normalized by layer to output the attention fusion transformation features; After performing two 3×3 convolution operations on the attention fusion transformation features, data of a preset size is obtained by upsampling by a preset multiple, and then a pixel-level illegal construction probability map is output through the Sigmoid activation function. Through the formula: Calculate the classification probability data; in, Represents categorical probability data; This indicates the satellite enhancement features in the data of each modality. Global average pooling is performed, and a channel attention weight vector of satellite features with a size of 1×1×256 is generated using a fully connected layer; This represents the UAV augmentation features in the data features of each modality. The multi-scale global vector of UAV features with a size of 1×1×256 is extracted by the global average pooling branch of the void space pyramid pooling module. It is a fully connected neural network for the classification layer; This indicates a vector concatenation operation that aggregates multimodal global features into a comprehensive feature vector; This indicates a global average pooling operation; This represents the attention fusion transformation feature; This represents the shared feature map extracted using the registration parameter matrix; This represents the ground fusion feature among the features of each modality of data; This represents the normalized exponential function.

8. The intelligent identification method for illegal construction based on integrated air-space-ground perception as described in claim 1, characterized in that, The Transformer model is iteratively updated using the overall loss until a preset iteration condition is met, resulting in a trained Transformer model. This process includes: The multi-source collaborative sensing array and labeled data for a preset historical time period are split into training set and validation set; According to the preset batch, the multi-source collaborative sensing array and labeled data in the training set are input into the model in sequence. When the overall loss of the validation set no longer decreases within a preset number of consecutive cycles or reaches the preset maximum number of iterations, the trained Transformer model is obtained.

9. A non-volatile computer storage medium, characterized in that, It stores computer instructions, which, when executed, implement a method for intelligent identification of illegal buildings based on integrated air-space-ground perception as described in any one of claims 1-8.

Images