Building extraction method, system and device based on multi-source remote sensing data fusion

Abstract

The invention provides a building extraction method, system and device based on multi-source remote sensing data fusion. The method comprises the steps that original satellite hyperspectral image data of a to-be-detected target area is acquired and preprocessed; constructing a multi-scale collaborative observation system combining a satellite hyperspectral image and an unmanned aerial vehicle high-resolution image based on the preprocessed hyperspectral satellite image, and extracting a building candidate area; obtaining unmanned aerial vehicle image data based on the coordinate data of the building candidate area; performing data fusion and sub-pixel positioning optimization based on the unmanned aerial vehicle image data to generate a hyperspectral fusion image; and performing target positioning and edge extraction on the hyperspectral fusion image based on the boundary refinement model to obtain an extracted building area. According to the method, the problems of contradiction between the coverage range and the spatial resolution, fuzzy shielding scene edge, insufficient multi-dimensional data utilization rate and the like of hyperspectral building extraction are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image data processing, and relates to a building extraction method, in particular to a building extraction method, system and device based on multi-source remote sensing data fusion. BACKGROUND

[0002] As a basic unit of urban spatial structure, the accurate extraction and dynamic monitoring of buildings have important strategic significance for urban renewal planning, disaster assessment and sustainable development. Hyperspectral remote sensing images can capture the spectral fingerprint characteristics of building materials (such as the C-H bond absorption peak of asphalt at 1730 nm) due to their nanometer-level spectral resolution (usually 5-10 nm) and hundreds of continuous bands, providing a new technical path to break through the recognition bottleneck of traditional RGB images in complex scenes. Compared with multispectral images, hyperspectral data can effectively identify buildings obscured by vegetation through mixed pixel decomposition and anomaly detection algorithms in the continuous feature space constructed in the spectral dimension, significantly improving the extraction capability in shaded scenes such as urban dense areas and forest settlements.

[0003] However, the engineering application of hyperspectral images faces several technical challenges:

[0004] First, the information redundancy and computational complexity caused by high-dimensional data, existing solutions mostly use band selection (such as SVM-RFE algorithm) or deep feature compression (such as 3D-CNN and autoencoder fusion model) to realize data dimension reduction;

[0005] Second, due to the physical limitations of the sensor, the spatial resolution of hyperspectral images is generally low (10-30 meters / pixel for spaceborne, 0.5-2 meters / pixel for airborne), which is difficult to meet the demand of fine building contour extraction.

[0006] To address this contradiction, current research mainly adopts a multi-source data collaboration strategy, combining super-resolution reconstruction algorithms and subpixel positioning technology (Subpixel ResNet) to improve the effective spatial resolution to sub-meter level while maintaining spectral discrimination. In addition, traditional hyperspectral building extraction methods mainly rely on spectral angle mapping (SAM), support vector machine (SVM) classification and other methods to classify targets with different spectral features, but their sensitivity to light changes and atmospheric interference leads to insufficient model generalization ability.

[0007] Therefore, with the development of deep learning technology, it has gradually become a more mainstream solution, introducing a 3D convolutional neural network to build an end-to-end model to directly process hyperspectral cube data, and designing a dual-stream network architecture (spectral stream + spatial stream) to fuse multi-scale features through cross-modal attention mechanisms.

[0008] However, existing methods still have two major bottlenecks:

[0009] First, it is difficult to achieve high spatial resolution and high spectral resolution simultaneously, resulting in blurred edges of the extracted results.

[0010] Second, spectral distortion in occluded scenes (such as shadow-induced reflectance reduction) leads to an increase in false alarm rate.

[0011] Therefore, it is urgent to develop a spatial-spectral joint modeling method and a hardware-algorithm co-optimization system to achieve accurate vectorization output of building contours in complex environments.

[0012] Therefore, how to solve the core problems of the contradiction between coverage range and spatial resolution, edge blur in occluded scenes, and insufficient utilization of multi-dimensional data in existing high-spectral building extraction technology is a problem that needs to be solved. SUMMARY

[0013] The purpose of the present application is to provide a building extraction method, system and device based on multi-source remote sensing data fusion, to solve the problems of contradiction between coverage range and spatial resolution, edge blur in occluded scenes, and insufficient utilization of multi-dimensional data in existing high-spectral building extraction technology.

[0014] In a first aspect, the present application provides a building extraction method based on multi-source remote sensing data fusion, comprising the following steps: obtaining original satellite hyperspectral image data of a target area to be detected; preprocessing the original satellite hyperspectral image data to obtain a first hyperspectral satellite image; constructing a multi-scale collaborative observation system of joint satellite hyperspectral image and unmanned aerial vehicle high-resolution image based on the first hyperspectral satellite image, and extracting a building candidate area and corresponding coordinate data; obtaining unmanned aerial vehicle image data based on the coordinate data of the building candidate area; performing data fusion based on the unmanned aerial vehicle image data to obtain image fusion information; performing positioning optimization based on the image fusion information, constructing a boundary refinement model, and generating a hyperspectral fusion image; performing target positioning and edge extraction on the hyperspectral fusion image based on the boundary refinement model to obtain an extracted building area.

[0015] In an implementation manner of the first aspect, constructing a multi-scale collaborative observation system of joint satellite hyperspectral image and unmanned aerial vehicle high-resolution image based on the first hyperspectral satellite image, and extracting a building candidate area and corresponding coordinate data comprises: constructing a dual-threshold detector of RX anomaly detection method based on unsupervised learning and NDBI building index; using the dual-threshold detector to screen the first hyperspectral satellite image to obtain a building candidate area and corresponding coordinate data that are significantly different from background spectral features.

[0016] In an implementation form of the first aspect, the screening the first hyperspectral satellite image by using the double-threshold detector to obtain the building candidate area and the corresponding coordinate data comprises: extracting a background sample of image pixels based on the first hyperspectral satellite image; the background sample is an entire image or a set of pixels selected by a local window; calculating a mean vector and a covariance matrix of spectral features of the background sample; calculating RX statistics of each to-be-detected pixel based on the mean vector and the covariance matrix; calculating a normalized building index of each to-be-detected pixel; screening out a building candidate area and building candidate area coordinate data which are significantly different from the background spectral features according to the RX statistics and the normalized building index of each to-be-detected pixel.

[0017] In an implementation form of the first aspect, the obtaining the unmanned aerial vehicle image data based on the coordinate data of the building candidate area comprises: obtaining a coordinate data set of the building candidate area; constructing a multi-objective function based on the coordinate data set, the multi-objective optimization function taking minimizing total flight time and total energy consumption of the unmanned aerial vehicle as an optimization objective and satisfying a constraint condition; generating an unmanned aerial vehicle dynamic scheduling scheme and a flight path based on the multi-objective function and the constraint condition; and obtaining the unmanned aerial vehicle image data of the building candidate area by collecting data by the unmanned aerial vehicle according to the unmanned aerial vehicle dynamic scheduling scheme.

[0018] In an implementation form of the first aspect, the constraint condition comprises: a working coverage radius of a single unmanned aerial vehicle is less than or equal to a radius threshold; and a battery endurance time of the single unmanned aerial vehicle is greater than or equal to an endurance time threshold.

[0019] In an implementation form of the first aspect, the data fusion based on the unmanned aerial vehicle image data to obtain image fusion information comprises: performing feature extraction based on the first hyperspectral satellite image and the unmanned aerial vehicle image data to obtain hyperspectral satellite image spectral features and high-resolution spatial edge features; performing feature fusion on the hyperspectral satellite image spectral features and the high-resolution spatial edge features to obtain spectral and spatial fusion features; performing sampling based on the spectral and spatial fusion features to generate high-spatial-resolution hyperspectral image fusion information; obtaining a joint loss function of the high-spatial-resolution hyperspectral image fusion information, and optimizing the joint loss function to obtain image fusion information.

[0020] In an implementation form of the first aspect, the joint loss function comprises: a spectral loss and a spatial loss; and a calculation formula of the joint loss function is:

[0021] L = a · L spectral + (1-a) · L spatial

[0022]

[0023] wherein, L represents a joint loss function; L spectral represents a spectral loss, which constrains the consistency of the fused image and the satellite hyperspectral image in the spectral dimension; L spatial represents a spatial loss, which constrains the consistency of the fused image and the unmanned aerial vehicle image in the spatial structure; a represents a weight coefficient; n represents the total number of sampling pixels; x i represents an input image pixel at the i-th sampling position; represents the spectral feature of the satellite hyperspectral image at the position x i ; represents the spectral feature of the fused hyperspectral image at the position x i ; represents the spatial gradient feature of the unmanned aerial vehicle image; represents the spatial gradient feature of the fused image.

[0024] In an implementation form of the first aspect, the target positioning and edge extraction on the hyperspectral fused image based on the boundary refinement model comprises: inputting the hyperspectral fused image into a pre-trained dual-flow deep interaction network to extract spatial features and spectral features; performing deep feature fusion on the spatial features and the spectral features by using a 3D spectral convolution kernel to obtain target position information and edge contours of the building; and generating building region information based on the target position information and the edge contours of the building.

[0025] In a second aspect, the application provides a building extraction system based on multi-source remote sensing data fusion, comprising: a data acquisition module configured to acquire original satellite hyperspectral image data of a target region to be detected; a preprocessing module configured to preprocess the original satellite hyperspectral image data to obtain a first hyperspectral satellite image; a candidate region extraction module configured to construct a multi-scale collaborative observation system of joint satellite hyperspectral images and unmanned aerial vehicle high-resolution images based on the first hyperspectral satellite image, and extract a building candidate region and corresponding coordinate data; an unmanned aerial vehicle scheduling module configured to acquire unmanned aerial vehicle image data based on the coordinate data of the building candidate region; an image fusion module configured to perform data fusion based on the unmanned aerial vehicle image data to obtain image fusion information; a sub-pixel positioning optimization module configured to perform positioning optimization based on the image fusion information, construct a boundary refinement model, and generate a hyperspectral fused image; and a building extraction module configured to perform target positioning and edge extraction on the hyperspectral fused image based on the boundary refinement model to obtain an extracted building region.

[0026] In a last aspect, the application provides a building extraction device based on multi-source remote sensing data fusion, comprising a processor and a memory. The memory is used to store a computer program; the processor is connected with the memory and is used to execute the computer program stored in the memory, so that the building extraction device based on multi-source remote sensing data fusion executes the building extraction method based on multi-source remote sensing data fusion.

[0027] As described above, the building extraction method, system and device based on multi-source remote sensing data fusion of the application have the following beneficial effects:

[0028] The building extraction method based on multi-source remote sensing data fusion provided by the application breaks through the technical bottleneck that the coverage range and spatial resolution of the traditional single sensor system cannot be compatible, by establishing a dynamic task planning algorithm driven by spectral anomaly and a physical constraint super-resolution fusion model through the space-air-ground collaborative multi-scale observation system architecture and the collaborative observation paradigm of "satellite global perception-unmanned aerial vehicle dynamic enhancement". Meanwhile, in the application, a sub-pixel boundary optimization framework driven by spectral unmixing is constructed, a cascade architecture of an end member automatic extraction module and a nonlinear unmixing device is designed, a pixel-level spectral unmixing-spatial optimization joint iteration strategy is created, and the boundary blur problem caused by mixed pixels in low-resolution images is solved. Furthermore, a double-flow deep interaction network architecture is designed, a spatial detail flow and a spectral feature flow double-channel deep interaction network is constructed, a feature alignment module and a cross-modal attention mechanism are designed, and the optimal fusion of multi-source remote sensing information is realized. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A hardware application scene schematic diagram in an embodiment of the building extraction method based on multi-source remote sensing data fusion described in the application is shown.

[0030] Figure 2 A flowchart in an embodiment of the building extraction method based on multi-source remote sensing data fusion described in the application is shown.

[0031] Figure 3 A flowchart of the building extraction method based on the joint of satellite hyperspectral image and unmanned aerial vehicle high-resolution image is shown.

[0032] Figure 4 A flowchart of S3 in the building extraction method based on multi-source remote sensing data fusion described in the application is shown.

[0033] Figure 5 A flowchart of S32 in the building extraction method based on multi-source remote sensing data fusion described in the application is shown.

[0034] Figure 6A flowchart showing S4 in the building extraction method based on multi-source remote sensing data fusion described in the present application.

[0035] Figure 7 A flowchart showing S5 in the building extraction method based on multi-source remote sensing data fusion described in the present application.

[0036] Figure 8 A flowchart showing S7 in the building extraction method based on multi-source remote sensing data fusion described in the present application.

[0037] Figure 9 A schematic diagram showing the principle structure of the building extraction method based on multi-source remote sensing data fusion described in the present application in an embodiment.

[0038] Figure 10 A schematic diagram showing the principle structure of the building extraction device based on multi-source remote sensing data fusion described in the present application in an embodiment.

[0039] Element number explanation

[0040] 11 base data collection module

[0041] 12 data processing module

[0042] 13 multi-scale collaborative observation system construction module

[0043] 14 sub-pixel positioning module

[0044] 15 double-flow depth interaction module

[0045] 91 data acquisition module

[0046] 92 preprocessing module

[0047] 93 candidate region extraction module

[0048] 94 unmanned aerial vehicle scheduling module

[0049] 95 image fusion module

[0050] 96 sub-pixel positioning optimization module

[0051] 97 building extraction module

[0052] 101 processor

[0053] 102 memory DETAILED DESCRIPTION

[0054] The following detailed description is presented to enable any person skilled in the art to make and use the application. Various modifications to the embodiments described herein will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.

[0055] It is noted that the drawings of the embodiments provided herein are only schematic and are non-limiting. In the drawings, the size of some of the elements can be exaggerated relative to others for illustrating conceptually the general principles described herein. The detailed description is presented only for a clear comprehension of the innovations and does not limit the scope of the application.

[0056] The embodiments of the present application provide a building extraction method based on multi-source remote sensing data fusion. A collaborative observation paradigm of "satellite global perception-unmanned aerial vehicle dynamic enhancement" is proposed. A dynamic task planning algorithm driven by spectral anomaly and a physical constraint super-resolution fusion model are established. The technical bottleneck that the coverage range and spatial resolution of a traditional single sensor system cannot be compatible is broken through. A cascade architecture of an endmember automatic extraction module and a nonlinear unmixer is designed. A joint iteration strategy of pixel-level spectral unmixing-spatial optimization is created. The boundary blur problem caused by mixed pixels in low-resolution images is solved. A spatial detail flow and a spectral feature flow dual-channel deep interaction network are constructed. A feature alignment module and a cross-modal attention mechanism are designed. The optimal fusion of multi-source remote sensing information is achieved.

[0057] As Figure 1As shown, the hardware application scene diagram of the building extraction method based on multi-source remote sensing data fusion provided in the embodiment of the present application specifically includes: a basic data collection module 11, a data processing module 12, a multi-scale collaborative observation system construction module 13, a sub-pixel positioning module 14 and a double-flow deep interaction module 15. The basic data collection module 11 is used to obtain original satellite hyperspectral image data of a target area to be detected, which includes but is not limited to: spatial data (such as: pixel / pixel, image size, etc.), spectral dimension (such as: waveband, waveband number, wavelength range, spectral resolution, etc.), and radiation information, time information and geometric information, etc.; these data can be obtained by accessing official data resources, main commercial data sources or data platforms, etc. The data processing module 12 is used to pre-process the original satellite hyperspectral image data of the target area to be detected. The multi-scale collaborative observation system construction module 13 is used to construct a space-time collaborative observation method of “satellite wide-area scanning + unmanned aerial vehicle key enhancement” based on satellite hyperspectral images to construct a basic observation layer, and to extract candidate building areas through a spectral anomaly detection algorithm and a normalized building index NDBI to generate key observation areas; then, for the key observation areas, a cluster of unmanned aerial vehicles is deployed to obtain sub-meter images through high-density revisit; and for non-key areas, probability sampling is implemented to ensure the representativeness of global data. The sub-pixel positioning module 14 is used to refine the information of a single pixel in a low spatial resolution image through a cascaded structure of an end member automatic extraction module and a nonlinear unmixing device based on a spectral unmixing sub-pixel boundary optimization framework, and to optimize the building edge positioning effect. The double-flow deep interaction module 15 is used to develop a double-flow deep interaction network, to extract spectral dimension features through a 3D spectral convolution kernel, to capture geometric details through a deformable spatial attention module, and to design a cross-modal feature alignment loss to realize deep fusion of spectral-spatial features; and then to improve the processing capability of the model for spectral and spatial dimension information.

[0058] The building extraction method based on multi-source remote sensing data fusion provided in the embodiment of the present application will be described in detail below with reference to the drawings in the embodiment of the present application.

[0059] Please refer to Figure 2 and Figure 3 , which respectively show the flowchart and the flowchart of the building extraction method based on multi-source remote sensing data fusion provided in the embodiment of the present application. Figure 2 and Figure 3 , the embodiment provides a building extraction method based on multi-source remote sensing data fusion.

[0060] To realize the information complementation between hyperspectral satellite images and high-resolution unmanned aerial vehicle images, the present example provides a space-air coordinated observation method based on "satellite wide-area scanning + unmanned aerial vehicle key enhancement", constructs a basic observation layer based on satellite hyperspectral images, and extracts candidate building areas through a spectral anomaly detection algorithm and a normalized difference built-up index (NDBI) to generate key observation areas. For the key observation areas, a cluster of unmanned aerial vehicles is deployed to obtain sub-meter images through high-density revisit; and probabilistic sampling is implemented on non-key areas to ensure the representativeness of global data.

[0061] The building extraction method based on multi-source remote sensing data fusion specifically includes the following steps:

[0062] S1, obtaining original satellite hyperspectral image data of a target area to be detected.

[0063] In the present example, the original satellite hyperspectral image data includes but is not limited to spatial data (such as pixels / pixels, image size, etc.), spectral dimension (such as waveband, waveband number, wavelength range, spectral resolution, etc.), and radiation information, time information, and geometric information.

[0064] Image data in a specified area in the target area can be extracted through multiple channels.

[0065] Specifically, the original satellite hyperspectral image data is a complex data set containing multiple dimensions and rich information. It is far more than a "picture", but a collection that can be called a "data cube". For example, the original satellite hyperspectral image data includes: pixels / pixels, image size (such as number of rows * number of columns), etc.; waveband, waveband number, wavelength range, spectral resolution, etc. Each pixel corresponds to a complete spectral curve. And radiation calibration coefficient, geometric positioning parameter, time information (image acquisition time), sensor information (such as satellite and sensor name, waveband center wavelength, waveband width, spectral response function, etc.), projection information, etc.

[0066] For example, first determine the requirements, i.e. the geographic range, area size, acquisition time, access frequency, spatial resolution, spectral resolution, spectral range, etc. of the target area to be detected; then obtain data through different data sources (such as free data sources, commercial data sources, etc.) platforms.

[0067] S2, pre-processing the original satellite hyperspectral image data to obtain a first hyperspectral satellite image.

[0068] In this embodiment, the original satellite hyperspectral image data is radiometrically corrected, the hyperspectral image is absolutely radiometrically corrected with reference to the radiometric calibration data, and FLAASH (Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes, accurate atmospheric correction model based on first principle) is used for relative radiometric correction to reduce the influence of the atmosphere on the target spectrum image. Based on the satellite orbit parameters and the RPC (Rational Polynomial Coefficients, remote procedure call model) geometric correction function, the image is coarsely corrected, and the ground control points are introduced to fine-tune the geographic spatial information of the hyperspectral image. The preprocessed hyperspectral satellite image is denoted as X corr .

[0069] Specifically, the DN value of the original satellite hyperspectral image data is obtained first, the DN value is converted into a radiometric brightness value and a radiometric brightness image is obtained through the radiometric calibration parameters of the sensor laboratory, and then the radiometric brightness image is corrected to obtain information such as surface reflectance data. Then, the hyperspectral image data after radiometric correction is preliminarily corrected through the RPC model; and the hyperspectral image after preliminary correction is geometrically corrected to obtain the first hyperspectral satellite image after radiometric correction.

[0070] S3, constructing a multi-scale collaborative observation system of joint satellite hyperspectral image and unmanned aerial vehicle high-resolution image based on the first hyperspectral satellite image, and extracting a building candidate area and corresponding coordinate data. Please refer to Figure 4 , which shows a flowchart of S3 in the building extraction method based on multi-source remote sensing data fusion described in the present application. As shown in Figure 4 , the S3 comprises the following steps:

[0071] S31, constructing a dual-threshold detector of RX anomaly detection method and NDBI building index based on unsupervised learning;

[0072] S32, using the dual-threshold detector to screen the first hyperspectral satellite image to obtain a building candidate area and corresponding coordinate data which are significantly different from the background spectral characteristics. Please refer to Figure 5 , which shows a flowchart of S32 in the building extraction method based on multi-source remote sensing data fusion described in the present application. As shown in Figure 5 , the S32 comprises the following steps:

[0073] S321, extracting a background sample of image pixels based on the first hyperspectral satellite image; the background sample is a whole image or a set of pixels selected by a local window;

[0074] S322, calculate the mean vector and covariance matrix of the spectral features of the background sample;

[0075] S323, based on the mean vector and covariance matrix, calculate the RX statistic of each pixel to be detected;

[0076] S324, calculate the normalized building index of each pixel to be detected;

[0077] S325, according to the RX statistic and the normalized building index of each pixel to be detected, screen out building candidate regions and building candidate region coordinate data that are significantly different from the background spectral features.

[0078] In this embodiment, a dual-threshold detector combining RX anomaly detection method based on unsupervised learning and NDBI building index is constructed; the pixels in the entire image or local window are selected as background samples, and the mean vector and covariance matrix of the background samples are calculated; for each pixel to be detected, the RX statistic is calculated; the NDBI building index of each pixel to be detected is calculated; and according to the RX statistic and the NDBI building index, building targets that are significantly different from the background spectral features are screened out.

[0079] Specifically, a dual-threshold detector combining RX anomaly detection method based on unsupervised learning and NDBI building index is constructed to screen out building targets that are significantly different from the background spectral features:

[0080] S = {(x, y) | RX(x, y) > r1∩NDBI(x, y) > r2}

[0081] Wherein, S represents a set of building candidate regions, containing all pixel coordinates in the image that are determined to be “possibly building”; (x, y) represents the pixel position (row coordinate, column coordinate) in the image; X(x, y) represents the RX method anomaly statistic, indicating the difference degree between the spectrum of the pixel and the background mean spectrum; NDBI(x, y) represents the normalized building index, reflecting the strength of building features; r1 and r2 represent threshold values.

[0082] The RX algorithm is a statistical method for detecting abnormal pixels, which assumes that the spectrum of the background region obeys a multivariate Gaussian distribution, and judges whether it is an abnormal value by calculating the deviation degree of each pixel from the statistical characteristics of the background. The specific steps are as follows:

[0083] (1) Select the pixels in the entire image or local window as background samples, and calculate the mean vector μ and covariance matrix σ;

[0084] (2) For each pixel to be detected x, calculate the RX statistic δ RX = (x-μ) T σ-1 (x-μ), and δ RX The greater, the more significant the difference between the pixel and the background.

[0085] NDBI is a normalized building index based on short-wave infrared and near-infrared bands, which can be used to enhance building features, and its calculation method is:

[0086]

[0087] B SWIR represents the short-wave infrared band; B NIR represents the infrared band.

[0088] It should be noted that the "dual-threshold detector" here is actually a logical intersection filter (AND strategy). RX provides anomaly discrimination (whether significantly different from the background); NDBI provides building-like feature discrimination (whether like a building); the two work together to obtain building candidate areas. In addition, in this step, the role of the RX algorithm is only to determine which pixels are significantly different from the background in terms of spectrum, and output their corresponding coordinate positions. The outliers do not need to be processed additionally, and the coordinate information of these abnormal pixels is used in the subsequent steps. In combination with the NDBI index, a double-threshold screening is performed to obtain the final building candidate area.

[0089] S4, obtaining unmanned aerial vehicle image data based on the coordinate data of the building candidate area. Please refer to Figure 6 , which shows the process diagram of S4 in the building extraction method based on multi-source remote sensing data fusion described in the present application. As Figure 6 indicated, the S4 includes the following steps:

[0090] S41, obtaining a coordinate data set S of the building candidate area;

[0091] S42, constructing a multi-objective function based on the coordinate data set, the multi-objective optimization function taking minimizing the total flight time and total energy consumption of the unmanned aerial vehicle as the optimization goal, and satisfying the constraint condition; wherein the constraint condition includes: the operation coverage radius of a single unmanned aerial vehicle is less than or equal to the radius threshold; the battery endurance time of a single unmanned aerial vehicle is greater than or equal to the endurance time threshold;

[0092] S43, generating an unmanned aerial vehicle dynamic scheduling scheme and a flight path based on the multi-objective function and the constraint condition; the multi-objective optimization function determines the optimal flight path of the unmanned aerial vehicle by balancing the flight time and the energy consumption;

[0093] S44, acquiring unmanned aerial vehicle image data of the building candidate area by data collection of the unmanned aerial vehicle according to the unmanned aerial vehicle dynamic scheduling scheme.

[0094] In this embodiment, the building candidate area coordinate data set S is input; a multi-objective optimization function is constructed, which is used to minimize the flight time and energy consumption; a constraint condition is set, which includes: the single-pass UAV coverage radius is less than or equal to the radius threshold, and the battery endurance is greater than or equal to the endurance time threshold; according to the multi-objective optimization function and the constraint condition, the UAV is dispatched to completely collect the high-resolution UAV image of the building candidate area; and a hierarchical random sampling method is used to randomly select a sub-region in a spatial grid to shoot a high-resolution UAV image.

[0095] Further, the construction of the multi-objective optimization function specifically includes: taking the total flight time of the UAV to complete all building candidate area data collection tasks as the first optimization target; taking the total energy consumption of the UAV in the entire flight and data collection process as the second optimization target; and performing weighting or constraint processing on the first optimization target and the second optimization target to form a single or multi-objective optimization function.

[0096] The hierarchical random sampling method includes: dividing the building candidate area into a plurality of spatial grids; randomly selecting a sub-region in each spatial grid; and dispatching the UAV to shoot a high-resolution image of the selected sub-region.

[0097] Specifically, this step mainly performs UAV dynamic scheduling and data collection, that is, the building candidate area coordinate data set S is input, and a multi-objective optimization function is constructed for minimizing the flight time and energy consumption:

[0098]

[0099] Wherein, t k represents the flight time of the kth UAV; E k represents the energy consumption of the kth UAV, and a represents the weight coefficient of the flight time in the optimization target; and β represents the weight coefficient of the energy consumption in the optimization target.

[0100] The constraint condition is that the operation coverage radius of a single-pass UAV is less than or equal to the radius threshold, and the battery endurance time of a single-pass UAV is greater than or equal to the endurance time threshold. In this embodiment, preferably, the single-pass UAV coverage radius is less than or equal to 5km, and the battery endurance is greater than or equal to 45 minutes, which are used as constraint conditions for illustration. The high-resolution UAV image of the building candidate area is completely collected, and a hierarchical random sampling method is used to randomly select a sub-region in a spatial grid to shoot a high-resolution UAV image.

[0101] S5, data fusion is performed based on the UAV image data to obtain image fusion information. Please refer to Figure 7 , which shows the flowchart of S5 in the building extraction method based on multi-source remote sensing data fusion described in this application. As Figure 7As shown, the S5 comprises the following steps:

[0102] S51, performing feature extraction based on the first hyperspectral satellite image and the unmanned aerial vehicle image data to obtain hyperspectral satellite image spectral features and high-resolution spatial edge features;

[0103] S52, performing feature fusion on the hyperspectral satellite image spectral features and the high-resolution spatial edge features to obtain spectral and spatial fusion features;

[0104] S53, performing sampling based on the spectral and spatial fusion features to generate high spatial resolution hyperspectral image fusion information;

[0105] S54, obtaining a joint loss function of the hyperspectral image fusion information and optimizing the same to obtain image fusion information. The joint loss function comprises spectral loss and spatial loss. The spectral loss is used to optimize the spectral information of the fused image, and the spatial loss is used to optimize the spatial information of the fused image.

[0106] In this embodiment, first, a dual-branch super-resolution network is constructed; then, multi-scale, cross-sensor hyperspectral satellite images and high-resolution unmanned aerial vehicle images are input; next, a joint loss function is defined, which is used to optimize the spectral and spatial information of the fused image; then, the dual-branch super-resolution network is used to fuse the hyperspectral satellite images and the high-resolution unmanned aerial vehicle images to obtain the fused image; finally, the fused image is optimized according to the joint loss function.

[0107] The dual-branch super-resolution network comprises a first network branch for processing hyperspectral satellite images, a second network branch for processing high-resolution unmanned aerial vehicle images, and a fusion layer for fusing the outputs of the first network branch and the second network branch.

[0108] Specifically, a dual-branch super-resolution network is constructed to fuse multi-scale, cross-sensor hyperspectral satellite images and high-resolution unmanned aerial vehicle images, and a joint loss function is defined to optimize the spectral and spatial information of the fused image, i.e.

[0109] The calculation formula of the joint loss function is:

[0110] L = a · L spectral + (1-a) · L spatial

[0111]

[0112] wherein L represents the joint loss function; L spectralrepresents spectral loss, constraining the consistency of the fusion image and the satellite hyperspectral image in the spectral dimension; L spatial represents spatial loss, constraining the consistency of the fusion image and the unmanned aerial vehicle image in the spatial structure; a represents a weight coefficient; n represents the total number of sampling pixels; x i represents the input image pixel at the i-th sampling position; represents the spectral feature of the satellite hyperspectral image at position x i ; and represents the spectral feature of the fusion hyperspectral image at position x i ; and represents the spatial gradient feature of the unmanned aerial vehicle image; and represents the spatial gradient feature of the fusion image.

[0113] SSIM (Structural Similarity Index) is an index for describing structural similarity, measuring data similarity from the angles of brightness, contrast, and structure, and improving the fusion effect of the image. The image fusion result based on the above manner is subject to mixed pixel images.

[0114] It should be noted that the multiscale, cross-sensor hyperspectral satellite image and the high-resolution unmanned aerial vehicle image have different spatial resolutions and spectral resolutions.

[0115] S6, based on the image fusion information, performing positioning optimization, constructing a boundary refinement model, and generating a hyperspectral fusion image.

[0116] To improve the edge details of the fusion result of the hyperspectral satellite image and the high-resolution unmanned aerial vehicle image, the present example provides a sub-pixel boundary optimization framework based on spectral unmixing, which refines the information of a single pixel in a low spatial resolution image through a cascade structure of an endmember automatic extraction module and a nonlinear unmixer, and optimizes the building edge positioning effect.

[0117] In the present embodiment, the building edge information of the image fused based on the above manner is not clear, and therefore a sub-pixel boundary optimization framework driven by spectral unmixing is constructed.

[0118] Specifically, the boundary refinement model based on spectral unmixing is as follows:

[0119]

[0120] wherein M ∈ R d×m represents a mixed pixel matrix; E ∈ R d×k represents an endmember spectral library; B represents an abundance matrix; and TV(B) represents a total variation regular term for constraining boundary smoothness. The output result of this step is the fused image.

[0121] S7, target positioning and edge extraction are performed on the hyperspectral fusion image based on the boundary refinement model to obtain an extracted building region. Please refer to Figure 8 , which is a flowchart of S7 in the building extraction method based on multi-source remote sensing data fusion described in the present application. As shown in Figure 8 , S7 includes the following steps:

[0122] S71, the hyperspectral fusion image is input into a pre-trained dual-stream deep interaction network to extract spatial features and spectral features;

[0123] S72, a 3D spectral convolution kernel is used to perform deep feature fusion on the spatial features and the spectral features to obtain target position information and edge contours of the building;

[0124] S73, building region information is generated based on the target position information and the edge contours of the building.

[0125] In this embodiment, the input image is processed in parallel by a dual-stream network architecture based on the hyperspectral fusion image; wherein the first branch network focuses on extracting the spatial structure features of the image; the second branch network focuses on extracting the spectral discriminant features of the image. Then, one or more interaction modules are used to perform deep fusion on the spatial structure features extracted by the first branch network and the spectral discriminant features extracted by the second branch network; the interaction module uses a 3D spectral convolution kernel for convolution operation to jointly model the features in the spatial dimension and the spectral dimension at the same time. Then, based on the fused features, a building extraction result map is generated, which includes target positioning information and edge contour information of the building.

[0126] Specifically, based on X fuse ∈R n×d An end-to-end dual-stream deep interaction network is constructed to fully utilize the spatial and spectral information of the fusion image to achieve the building extraction task based on the hyperspectral image. The network introduces a 3D spectral convolution kernel for spatial-spectral feature fusion to improve the target positioning and edge extraction function of the model for buildings. Among them, a 2D convolution network can be used to extract the spatial features of high-resolution images, and a 3D spectral convolution kernel is introduced to process hyperspectral data to directly extract spectral features in the spatial and spectral dimensions to capture the differences between different wavebands. Among them, a 3D convolution kernel is used in the spectral stream to simultaneously aggregate local spatial fields and cross-waveband spectral information to generate a fused feature map.

[0127] That is, first, the complete hyperspectral fusion image to be predicted is cut into the same size as when training, and is input into the double-flow deep interaction network that has been trained. Then forward propagation is performed, at which time the network automatically performs double-flow feature extraction, interaction fusion, and finally outputs a probability value (between 0 and 1) belonging to a building for each pixel through a classifier (such as a Softmax or Sigmoid layer). Further, post-processing is performed to convert the output probability map into a binary segmentation map (for example, pixels with a probability greater than 0.5 are judged to be buildings). Simple post-processing operations (such as morphological opening and closing operations) can be supplemented to remove small noise points and smooth the building boundaries. Finally, the output is completed, and the final building extraction result map is generated, which clearly shows the spatial position, contour and range of all buildings in the image.

[0128] Therefore, it can be known that this step fully utilizes the spatial detail information and spectral characteristics of the fused image through the double-flow deep interaction network, and the ability of the extraction model to locate and edge contour the building target.

[0129] In the present application, a double-flow deep interaction network is developed by constructing a double-flow deep interaction network architecture based on spatial-spectral feature learning. Spectral dimension features are extracted by a 3D spectral convolution kernel, and geometric details are captured by a deformable spatial attention module. A cross-modal feature alignment loss is designed to realize deep fusion of spectral-spatial features, thereby improving the processing capability of the model for spectral and spatial dimension information.

[0130] The building extraction method based on multi-source remote sensing data fusion provided in the present application is aimed at the technical bottlenecks in the existing hyperspectral observation system, such as the incompatibility of coverage range and spatial resolution, distortion of features in occlusion scenes, and high cost of manual intervention. By constructing a closed-loop optimization mechanism of “satellite global perception-unmanned aerial vehicle dynamic enhancement”, through an adaptive task planning algorithm driven by spectral anomalies and a physical constraint super-resolution fusion model, autonomous locking and sub-pixel level boundary reconstruction of large-scale building targets are realized, and the efficiency loss of single sensor observation is significantly reduced. The present application realizes information complementation between hyperspectral satellite images and high-resolution unmanned aerial vehicle images, and improves the building discrimination and edge positioning capability of the existing method.

[0131] The protection scope of the building extraction method based on multi-source remote sensing data fusion described in the embodiments of the present application is not limited to the order of steps listed in the embodiments. Any scheme realized by increasing, reducing or replacing steps of the prior art according to the principles of the present application is included in the protection scope of the present application.

[0132] The embodiments additionally provide a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the building extraction method based on multi-source remote sensing data fusion as described above. Figure 1The building extraction method based on multi-source remote sensing data fusion.

[0133] At the core of any possible technical details, the present application can be a system, a method and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, the computer readable program instructions being used to cause a processor to implement various aspects of the present application.

[0134] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a ROM, a magnetic track storage and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0135] The computer readable program here described can be downloaded to respective computing / processing devices from a computer readable storage medium or to external computers or external storage devices via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device. Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0136] The embodiment of the present application further provides a building extraction system based on multi-source remote sensing data fusion. The building extraction system based on multi-source remote sensing data fusion can implement the building extraction method based on multi-source remote sensing data fusion. However, the implementation device of the building extraction method based on multi-source remote sensing data fusion is not limited to the structure of the building extraction system based on multi-source remote sensing data fusion. Any structure modification and replacement of the prior art based on the principle of the present application are included in the protection scope of the present application.

[0137] The building extraction system based on multi-source remote sensing data fusion provided by the embodiment will be described in detail below in combination with the drawings.

[0138] The embodiment provides a building extraction system based on multi-source remote sensing data fusion, which comprises:

[0139] Referring to Figure 9 . As Figure 9 shown, the building extraction system based on multi-source remote sensing data fusion includes a data acquisition module 91, a preprocessing module 92, a candidate area extraction module 93, a UAV scheduling module 94, an image fusion module 95, a sub-pixel positioning optimization module 96, and a building extraction module 97.

[0140] The data acquisition module 91 is configured to acquire original satellite hyperspectral image data of a target area to be detected.

[0141] In this embodiment, the original satellite hyperspectral image data includes but is not limited to spatial data (such as pixels / pixels, image size, etc.), spectral dimension (such as waveband, waveband number, wavelength range, spectral resolution, etc.), and radiation information, time information, and geometric information.

[0142] The image data in the specified area in the target area can be extracted through multiple channels.

[0143] The preprocessing module 92 is configured to preprocess the original satellite hyperspectral image data to obtain a first hyperspectral satellite image.

[0144] In this embodiment, the original satellite hyperspectral image data is radiometrically corrected, the hyperspectral image is absolutely radiometrically corrected with reference to radiometric calibration data, and FLAASH is used for relative radiometric correction to reduce the influence of the atmosphere on the target spectrum. Based on satellite orbit parameters and RPC geometric correction functions, the image is coarsely corrected, and ground control points are introduced to fine-tune the geographic spatial information of the hyperspectral image.

[0145] The candidate area extraction module 93 is configured to construct a multi-scale collaborative observation system of joint satellite hyperspectral images and UAV high-resolution images based on the first hyperspectral satellite image, and extract building candidate areas and corresponding coordinate data.

[0146] In this embodiment, first, an RX anomaly detection method combined with unsupervised learning and a double-threshold detector of NDBI building index are constructed.

[0147] Secondly, the first hyperspectral satellite image is screened by using the double threshold detector to obtain building candidate areas and corresponding coordinate data which are significantly different from background spectral characteristics. The method comprises the following steps: extracting a background sample of image pixels based on the first hyperspectral satellite image; the background sample is an entire image or a set of pixels selected by a local window; calculating a mean vector and a covariance matrix of spectral characteristics of the background sample; based on the mean vector and the covariance matrix, calculating RX statistics of each to-be-detected pixel; calculating a normalized building index of each to-be-detected pixel; and screening out building candidate areas and building candidate area coordinate data which are significantly different from background spectral characteristics according to the RX statistics and the normalized building index of each to-be-detected pixel.

[0148] The unmanned aerial vehicle scheduling module 94 is configured to obtain unmanned aerial vehicle image data based on the coordinate data of the building candidate areas.

[0149] In the embodiment, building candidate area coordinate data sets are input; a multi-objective optimization function is constructed, the multi-objective optimization function being used to minimize flight time and energy consumption; constraint conditions are set, the constraint conditions comprising: a single-pass unmanned aerial vehicle coverage radius being less than or equal to a radius threshold value, and a battery endurance being greater than or equal to an endurance time threshold value; based on the multi-objective optimization function and the constraint conditions, an unmanned aerial vehicle is scheduled to completely collect high-resolution unmanned aerial vehicle images of the building candidate areas; and a hierarchical random sampling method is used to randomly select sub-regions in a spatial grid to shoot high-resolution unmanned aerial vehicle images.

[0150] Further, the construction of the multi-objective optimization function specifically comprises: taking total flight time of the unmanned aerial vehicle for completing all building candidate area data collection tasks as a first optimization target; taking total energy consumption of the unmanned aerial vehicle in the entire flight and data collection process as a second optimization target; and performing weighting or constraint processing on the first optimization target and the second optimization target to form a single or multi-objective optimization function.

[0151] The hierarchical random sampling method comprises: dividing the building candidate areas into a plurality of spatial grids; randomly selecting sub-regions in each spatial grid; and scheduling the unmanned aerial vehicle to shoot high-resolution images of the selected sub-regions.

[0152] The image fusion module 95 is configured to perform data fusion based on the unmanned aerial vehicle image data to obtain image fusion information.

[0153] In this embodiment, feature extraction is performed based on the first hyperspectral satellite image and the unmanned aerial vehicle image data to obtain hyperspectral satellite image spectral features and high-resolution spatial edge features; the hyperspectral satellite image spectral features and the high-resolution spatial edge features are fused to obtain spectral and spatial fusion features; sampling is performed based on the spectral and spatial fusion features to generate high-spatial-resolution hyperspectral image fusion information; a joint loss function of the hyperspectral image fusion information is obtained and optimized to obtain image fusion information. The joint loss function includes spectral loss and spatial loss; the spectral loss is used to optimize the spectral information of the fused image, and the spatial loss is used to optimize the spatial information of the fused image.

[0154] Specifically, first, a double-branch super-resolution network is constructed; then, multiscale, cross-sensor hyperspectral satellite images and high-resolution unmanned aerial vehicle images are input; next, a joint loss function is defined, which is used to optimize the spectral and spatial information of the fused image; then, the hyperspectral satellite images and the high-resolution unmanned aerial vehicle images are fused by the double-branch super-resolution network to obtain a fused image; finally, the fused image is optimized according to the joint loss function.

[0155] The double-branch super-resolution network includes a first network branch for processing hyperspectral satellite images, a second network branch for processing high-resolution unmanned aerial vehicle images, and a fusion layer for fusing the outputs of the first network branch and the second network branch.

[0156] The sub-pixel positioning optimization module 96 is configured to perform positioning optimization based on the image fusion information, construct a boundary refinement model, and generate a hyperspectral fusion image.

[0157] In this embodiment, the mixed pixel image is fused based on the above method, and the building edge information is not clear, so a sub-pixel boundary optimization framework driven by spectral unmixing is constructed.

[0158] The building extraction module 97 is configured to perform target positioning and edge extraction on the hyperspectral fusion image based on the boundary refinement model to obtain an extracted building region.

[0159] In this embodiment, the hyperspectral fusion image is input into a pre-trained double-flow deep interaction network to extract spatial features and spectral features; a 3D spectral convolution kernel is used to perform deep feature fusion on the spatial features and the spectral features to obtain target position information and edge contours of the building; and based on the target position information and the edge contours of the building, building region information is generated.

[0160] The building extraction model based on multi-source remote sensing data fusion is used to build a building extraction system based on multi-source remote sensing data fusion, which can solve the technical bottlenecks such as the incompatibility of coverage range and spatial resolution in the existing hyperspectral observation system, the distortion of the feature of the shielding scene, and the high cost of manual intervention. Through the closed-loop optimization mechanism of "satellite global perception-unmanned aerial vehicle dynamic enhancement", the adaptive task planning algorithm driven by spectral anomaly and the physical constraint super-resolution fusion model are used to realize the autonomous locking and sub-pixel level boundary reconstruction of large-scale building targets, and the efficiency loss of single sensor observation is significantly reduced.

[0161] It should be noted that the division of each module of the above system is only a logical functional division, and all or part of it can be integrated into a physical entity, or physically separated. And these modules can all be realized in the form of software called by the processing element; all can be realized in the form of hardware; some modules can be realized in the form of software called by the processing element, and some modules can be realized in the form of hardware. For example, the x module can be a separate processing element, or it can be integrated into a chip of the above system, in addition, it can also be stored in the form of program code in the memory of the above system, and the function of the above x module is called and executed by a processing element of the above system. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit of hardware in the processor element or the instruction in the form of software.

[0162] The above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For example, when a certain module above is realized in the form of program code called by the processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together to realize in the form of system on a chip (SOC).

[0163] Please refer to Figure 10The diagram shows a schematic representation of the building extraction device based on multi-source remote sensing data fusion described in this application, illustrating its principle structure in one embodiment. Figure 10 As shown, this embodiment provides a building extraction device based on multi-source remote sensing data fusion. The building extraction device based on multi-source remote sensing data fusion includes: a processor 101 and a memory 102; the memory 102 is used to store a computer program; the processor 101 is connected to the memory 102 and is used to execute the computer program stored in the memory 102, so that the building extraction device based on multi-source remote sensing data fusion performs the various steps of the building extraction method based on multi-source remote sensing data fusion as described above.

[0164] Preferably, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0165] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0166] In summary, the building extraction method, system, and apparatus based on multi-source remote sensing data fusion provided in this application have the following beneficial effects:

[0167] The building extraction method based on multi-source remote sensing data fusion provided in the application breaks through the technical bottleneck that the coverage range and spatial resolution of the traditional single sensor system cannot be compatible by establishing a dynamic task planning algorithm driven by spectral anomaly and a physical constraint super-resolution fusion model through a space-air-ground collaborative multi-scale observation architecture and a collaborative observation paradigm of "satellite global perception-unmanned aerial vehicle dynamic enhancement". Meanwhile, in the application, a sub-pixel boundary optimization framework driven by spectral unmixing is constructed, a cascade architecture of an end member automatic extraction module and a nonlinear unmixer is designed, a joint iteration strategy of pixel-level spectral unmixing and spatial optimization is created, and the boundary blur problem caused by mixed pixels in low-resolution images is solved. Furthermore, a double-flow deep interaction network architecture is designed, a spatial detail flow and a spectral feature flow double-channel deep interaction network is constructed, a feature alignment module and a cross-modal attention mechanism are designed, and the optimal fusion of multi-source remote sensing information is realized.

[0168] The above embodiments only exemplarily illustrate the principles and effects of the application, and are not used to limit the application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed in the application should be covered by the claims of the application.

Claims

1. A method for building extraction based on multi-source remote sensing data fusion, characterized in that, include: Acquire raw satellite hyperspectral image data of the target area to be detected; The original satellite hyperspectral image data is preprocessed to obtain the first hyperspectral satellite image; Based on the first hyperspectral satellite image, a multi-scale collaborative observation system combining satellite hyperspectral imagery and UAV high-resolution imagery was constructed, and candidate building areas and their corresponding coordinate data were extracted. UAV imagery data is obtained based on the coordinate data of the candidate building area; Data fusion is performed based on the UAV image data to obtain image fusion information; Based on the image fusion information, the localization is optimized, a boundary refinement model is constructed, and a hyperspectral fused image is generated; Based on the boundary refinement model, the hyperspectral fusion image is used for target localization and edge extraction to obtain the extracted building areas.

2. The building extraction method based on multi-source remote sensing data fusion according to claim 1, characterized in that, Based on the first hyperspectral satellite imagery, a multi-scale collaborative observation system combining satellite hyperspectral imagery and UAV high-resolution imagery was constructed, and candidate building areas and their corresponding coordinate data were extracted, including: Construct a joint unsupervised learning-based RX anomaly detection method and a dual-threshold detector for the NDBI building index; The dual-threshold detector is used to filter the first hyperspectral satellite image to obtain candidate building areas and their corresponding coordinate data that are significantly different from the background spectral features.

3. The building extraction method based on multi-source remote sensing data fusion according to claim 2, characterized in that, The dual-threshold detector is used to filter the first hyperspectral satellite imagery to obtain candidate building areas and their corresponding coordinate data, including: Background samples of image pixels are extracted based on the first hyperspectral satellite image; the background samples are the entire image or a set of pixels selected through a local window; Calculate the mean vector and covariance matrix of the spectral features of the background sample; Based on the mean vector and covariance matrix, the RX statistic for each pixel to be detected is calculated iteratively. Calculate the normalized building index for each equally detected pixel; Based on the RX statistic of each pixel to be detected and the normalized building index, candidate building areas and their coordinate data that are significantly different from the background spectral features are selected.

4. The building extraction method based on multi-source remote sensing data fusion according to claim 1, characterized in that, Obtaining UAV imagery data based on the coordinate data of the candidate building area includes: Obtain the coordinate dataset of the candidate building area; A multi-objective function is constructed based on the coordinate dataset. The multi-objective optimization function aims to minimize the total flight time and total energy consumption of the UAV, while satisfying the constraints. Based on the aforementioned multi-objective function and constraints, a dynamic scheduling scheme and flight path for unmanned aerial vehicles (UAVs) are generated. Data is collected by drones according to the aforementioned drone dynamic scheduling scheme to obtain drone image data of the candidate building area.

5. The building extraction method based on multi-source remote sensing data fusion according to claim 4, characterized in that, The constraints include: The operational coverage radius of a single drone is less than or equal to the radius threshold; the battery endurance of a single drone is greater than or equal to the endurance threshold.

6. The building extraction method based on multi-source remote sensing data fusion according to claim 1, characterized in that, Based on the aforementioned UAV imagery data, data fusion is performed to obtain image fusion information including: Feature extraction is performed based on the first hyperspectral satellite image and the UAV image data to obtain the spectral features of the hyperspectral satellite image and the high-resolution spatial edge features; The spectral features and high-resolution spatial edge features of the hyperspectral satellite image are fused to obtain spectral and spatial fused features; Based on the spectral and spatial fusion features, sampling is performed to generate high spatial resolution hyperspectral image fusion information; the joint loss function of the hyperspectral image fusion information is obtained and optimized to obtain the image fusion information.

7. The building extraction method based on multi-source remote sensing data fusion according to claim 6, characterized in that, The joint loss function includes: spectral loss and spatial loss; The formula for calculating the joint loss function is as follows: L=α·L spectral +(1-a)·L spatial Where L represents the joint loss function; L spectral Indicates spectral loss, constraining the consistency of the fused image with the satellite hyperspectral image in the spectral dimension; L spatial α represents spatial loss, constraining the consistency of the fused image and the UAV image in terms of spatial structure; α represents the weighting coefficient; n represents the total number of sampled pixels; x i This represents the input image pixel at the i-th sampling position; This indicates that the satellite hyperspectral image is located at position x. i Spectral characteristics; This indicates that the fused hyperspectral image is located at position x. i Spectral characteristics; Indicates the spatial gradient characteristics of UAV imagery; This represents the spatial gradient characteristics of the fused image.

8. The building extraction method based on multi-source remote sensing data fusion according to claim 1, characterized in that, Based on the boundary refinement model, target localization and edge extraction are performed on the hyperspectral fused image to obtain the extracted building regions, including: The hyperspectral fused image is input into a pre-trained two-stream deep interactive network to extract spatial and spectral features; The spatial and spectral features are fused using 3D spectral convolution kernels to obtain the target location information and edge contour of the building. Based on the target location information and edge contour of the building, the building area information is generated.

9. A building extraction system based on multi-source remote sensing data fusion, characterized in that, include: The data acquisition module is used to acquire raw satellite hyperspectral image data of the target area to be detected; The preprocessing module is used to preprocess the original satellite hyperspectral image data to obtain the first hyperspectral satellite image; The candidate region extraction module is used to construct a multi-scale collaborative observation system based on the first hyperspectral satellite imagery, combining satellite hyperspectral imagery and UAV high-resolution imagery, and to extract candidate building regions and their corresponding coordinate data. The drone scheduling module is used to acquire drone image data based on the coordinate data of the candidate building area; The image fusion module is used to perform data fusion based on the UAV image data to obtain image fusion information; The sub-pixel localization optimization module is used to optimize localization based on the image fusion information, construct a boundary refinement model, and generate a hyperspectral fused image. The building extraction module is used to perform target localization and edge extraction on the hyperspectral fusion image based on the boundary refinement model to obtain the extracted building regions.

10. A building extraction device based on multi-source remote sensing data fusion, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory to cause the building extraction device based on multi-source remote sensing data fusion to perform the building extraction method based on multi-source remote sensing data fusion as described in any one of claims 1 to 8.

Images