Three-dimensional geographic city building instance segmentation method based on satellite-unmanned aerial vehicle cooperation

Abstract

This invention discloses a 3D geographic urban building instance segmentation method based on satellite-UAV collaboration. It extracts multi-scale features from satellite images to obtain a satellite feature pyramid; extracts features and estimates depth from UAV images at each viewpoint, and performs depth-aware cross-view attention aggregation to generate bird's-eye view features and a height information map; uses the height information map to weight-modulate the bird's-eye view features, and fuses the features at each level of the satellite feature pyramid with the modulated bird's-eye view features through cross-view attention to obtain an enhanced feature pyramid; inputs the enhanced feature pyramid into an instance segmentation network, and outputs the instance segmentation results of buildings within the target area. This invention effectively compensates for the lack of information from a single viewpoint, significantly improving the boundary integrity of building instances, the ability to distinguish adjacent instances, and the overall segmentation accuracy in complex urban scenes, achieving a good balance between large-scale coverage and local geometric details.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and computer vision technology, specifically to a method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration. Background Technology

[0002] Accurate extraction of building information has significant practical application value in fields such as urban planning, disaster assessment, dynamic map updates, land resource management, and smart city construction. Current building extraction technologies in the remote sensing field mostly rely on single-source remote sensing image data for research and application, and their segmentation results are easily constrained by the characteristics of the data source itself.

[0003] While satellite remote sensing imagery alone offers advantages such as wide coverage, high acquisition efficiency, and suitability for large-scale surveys, it is limited by factors such as spatial resolution, imaging perspective, and occlusion. In densely populated areas, complex boundary regions, and scenes with significant differences in facade features, it is difficult to finely depict the detailed structure of buildings, thus affecting the accuracy of instance segmentation. Low-altitude UAV remote sensing imagery, on the other hand, boasts high resolution, flexible perspective, and rich local detail information, but suffers from limited coverage. Furthermore, there are significant scale differences, occlusion overlaps, and perspective shifts between multi-view UAV images, making it difficult to achieve both global understanding and spatial positioning capabilities for large-scale scenes when used alone.

[0004] To overcome the inherent limitations of single-source remote sensing imagery, fully leverage the complementary advantages of different data sources, and improve the overall effectiveness of building instance segmentation, researchers have gradually begun research on the fusion of remote sensing images from different perspectives and data sources. However, existing cross-view fusion methods still have many shortcomings in research and application, and their research focus has long been on the fusion of ground street view images and satellite images. A dedicated collaborative processing technology system for low-altitude UAV multi-view imagery and satellite imagery has not yet been established. Specific problems are reflected in the following three aspects: First, there are significant differences in geometric features between images from different perspectives. If feature stitching or simple attention mechanism fusion is performed directly, it is difficult to establish a stable cross-perspective spatial correspondence in the heterogeneous feature space. Secondly, many methods for constructing bird's-eye view representations rely heavily on the assumption of a planar scene. Such modeling methods ignore the elevation information in three-dimensional space, cannot effectively depict the height differences and three-dimensional structural features of buildings, are difficult to adapt to the complexity and diversity of building forms, and lack effective utilization of three-dimensional geospatial information. Third, existing fusion solutions are all designed around the fusion requirements of ground street view and satellite imagery, lacking feature fusion and collaborative modeling strategies for multi-view UAV images and satellite images. This makes it difficult to fully explore and utilize the advantages of UAVs' flexible perspective and fine perception, as well as the global macro coverage capabilities of satellite images. Ultimately, in building instance segmentation tasks, they exhibit limitations such as poor boundary integrity and insufficient ability to distinguish adjacent instances. Summary of the Invention

[0005] This invention provides a three-dimensional geographic city building instance segmentation method based on satellite-UAV collaboration. By constructing a depth-aware bird's-eye view representation and a height-guided cross-view fusion mechanism, it achieves collaborative processing of satellite images and UAV multi-view images, thereby effectively solving the problems of establishing geometric correspondence between different viewpoints, expressing differences in building height, and the lack of UAV-satellite collaboration solutions in existing technologies. This improves the accuracy and robustness of building instance segmentation in complex scenes, and finally outputs building instance segmentation results on two-dimensional satellite image features.

[0006] This invention provides a method for segmenting 3D geographic city building instances based on satellite-UAV collaboration, the method comprising: Acquire satellite images and drone aerial images of the target area, and construct a satellite-drone scene pairing instance segmentation image dataset, where the satellite images contain building instance-level labels, and the drone aerial images contain multi-view images and corresponding camera parameters; The satellite image is input into the satellite image processing branch, where multi-scale feature extraction is performed on the satellite image, and a satellite image feature pyramid is obtained. The drone aerial images are input into the drone image processing branch. Multi-scale feature extraction and depth estimation are performed on the drone aerial images from each viewpoint. The features from each viewpoint are combined with camera parameters to perform cross-viewpoint attention aggregation for depth perception, so as to generate bird’s-eye view features in a unified bird’s-eye view space. At the same time, the height information of each viewpoint is extracted based on the depth information of each viewpoint, and the height information of multiple viewpoints is aggregated to generate a height information map in the same bird’s-eye view space as the bird’s-eye view features. The obtained height information map is used to perform weight modulation on the bird's-eye view features, and the features at each level of the satellite image multi-scale feature pyramid are fused with the modulated bird's-eye view features through cross-view attention fusion to obtain the satellite image enhancement feature pyramid. The satellite image enhancement feature pyramid is input into the instance segmentation network, and the instance segmentation results of buildings within the target area are output.

[0007] Furthermore, satellite images and drone aerial images of the target area are acquired, and a satellite-drone scene pairing instance segmentation image dataset is constructed, specifically including: The obtained instance segmentation image dataset is divided into training, validation, and test sets for training, validation, and testing of the network model.

[0008] Furthermore, the satellite image is input into the satellite image processing branch to perform multi-scale feature extraction and obtain a satellite image feature pyramid, specifically including: A feature extraction network is used to extract multi-scale features from satellite images and output multi-scale satellite image features. Multi-scale satellite image features are mapped to a unified channel dimension and then input into a feature pyramid network to generate satellite image features at multiple levels.

[0009] Furthermore, multi-scale feature extraction and depth estimation are performed on the drone aerial images from each viewpoint, and cross-viewpoint attention aggregation with depth perception is combined with camera parameters to generate bird's-eye view features in a unified bird's-eye view space, specifically including: Multi-scale feature extraction is performed on the drone aerial images from each viewpoint using a feature extraction network to obtain multi-scale drone image features for each viewpoint. These multi-scale drone image features are then mapped to a unified channel dimension and input into a feature pyramid network to generate multiple levels of drone image features for each viewpoint. A depth estimation network is used to estimate the depth of the drone aerial images from each viewpoint to obtain a depth map for each viewpoint. Then, a downsampling operation is performed on the depth map of each viewpoint using bilinear interpolation to adjust its scale to the resolution corresponding to the features of each level, so as to obtain the depth map corresponding to each level for each viewpoint. Based on the drone image features and depth maps at each level from each viewpoint, a depth-aware cross-viewpoint attention mechanism is used to aggregate them and generate bird's-eye view features in a unified bird's-eye view space.

[0010] Furthermore, based on the UAV image features and depth maps at each level from each viewpoint, a depth-aware cross-viewpoint attention mechanism is used to aggregate them, generating bird's-eye view features in a unified bird's-eye view space, specifically including: For each level l, l=1,2,…,L, where L is the number of levels, and for each viewpoint i, i=1,2,…,I, where I is the number of viewpoints; First, calculate the depth-sensing geometric embedding at the l-th layer from the i-th viewpoint:

[0011] in, Let represent the geometric embedding enhancement feature at level l from the i-th viewpoint. This represents the 3D point coordinates at the l-th level from the i-th viewpoint, calculated in world coordinates based on camera parameters. This represents the depth map corresponding to the l-th layer from the i-th viewpoint, where Embed represents the geometric embedding function and Encode represents the depth encoding function; Then calculate the query, key, and value matrix at level l from the i-th perspective:

[0012]

[0013]

[0014] in, Represents the calculated l-th viewpoint The unified bird's-eye view feature after 1-level aggregation, when l=1, takes the value of the initial embedded prior feature obtained by adding the randomly generated learnable initial prior feature to the bird's-eye view grid position code; Let represent the drone image features at level l from the i-th viewpoint. , , For learnable query, key, and value projection matrices, , , These represent the corresponding query, key, and value matrices, respectively. Next, calculate the attention output of the l-th level from the i-th viewpoint. :

[0015] in, The key vector dimension is represented by softmax; softmax represents the normalization operation. Then, average pooling is performed on the attention outputs of the l-th level from all perspectives to obtain the unified bird's-eye view feature of the l-th level:

[0016] Where I represents the number of viewpoints. This represents the unified bird's-eye view feature after aggregation of the l-th level from all perspectives; A unified bird's-eye view feature for each level across all perspectives. Then, a layer-by-layer refinement process is performed, including skip connections and Bottleneck convolutions:

[0017] in, Indicates the lth viewpoint The unified bird's-eye view features after 1-level aggregation, when l=1, are the initial BEV embedding priors; This represents the unified bird's-eye view feature after aggregation of the l-th level from all perspectives. This represents the refined bird's-eye view features of the l-th level after skip connections and Bottleneck convolutions across all viewpoints. After progressive processing, the final output is the refined bird's-eye view feature at the last layer, i.e., the Lth layer. As a unified aerial view feature:

[0018] in, This represents the unified bird's-eye view feature of the final output. This represents the detailed bird's-eye view features of the last layer, i.e., the Lth layer.

[0019] Furthermore, height information of each viewpoint is extracted based on the depth information of each viewpoint, and the height information of multiple viewpoints is aggregated to generate a height information map of the same bird's-eye view space as the bird's-eye view feature, specifically including: Based on the depth map of each viewpoint, each pixel in the depth map is... The depth value is back-projected to the 3D space through the camera parameters to obtain the 3D point coordinates in the camera coordinate system. Then, the coordinates are transformed to the world coordinate system to obtain the 3D point coordinates (X,Y,Z) in the world coordinate system. The corresponding height value is obtained by extracting the Y coordinate component. The height values ​​of multiple viewpoints corresponding to the same bird's-eye view location are aggregated to generate a height map that unifies the bird's-eye view space with the aforementioned bird's-eye view features. The formula is as follows:

[0020] in, Indicates the first The height value corresponding to each viewpoint Indicates the grid position from a bird's-eye view perspective In the Mapping position in each viewpoint Indicates the number of viewpoints. This represents the aggregate height value at the grid position (u, v) from a bird's-eye view perspective;

[0021] in, This represents the height information map with a resolution of 128×128.

[0022] Furthermore, the obtained height information map is used to perform weighted modulation on the bird's-eye view features, specifically including: The height information map is encoded to obtain height attention information, and the calculation formula is as follows:

[0023]

[0024]

[0025]

[0026] in, This represents the input height information map. This represents the convolution operation. Indicates batch normalization, This represents the activation function. This represents the extracted basic height features. Indicates the enhanced height feature. This represents the generated attention score. express Activation function This represents highly attentional information after encoding; The bird's-eye view features are weighted and modulated based on the height attention information to enhance the feature response of building-related areas. The calculation formula is as follows:

[0027]

[0028] in, Indicates bird's-eye view features, This represents the splicing of highly attentional information with bird's-eye view features. This represents element-wise multiplication. This represents the learnable modulation intensity coefficient. Indicates the gating weight, This indicates the modulated bird's-eye view features.

[0029] Furthermore, the satellite features at each level of the multi-scale feature pyramid of the satellite image are fused with the modulated bird's-eye view features through cross-view attention fusion to obtain the satellite image enhancement feature pyramid, specifically including: Before performing attention fusion calculation, the hierarchical satellite features and the modulated bird's-eye view features are first aligned to a unified channel dimension using a 1×1 convolution. Then, the channel-aligned hierarchical satellite features are used as queries, and the channel-aligned modulated bird's-eye view features are used as keys and values ​​to generate fused features. The attention fusion calculation formula is as follows:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] in, Indicates the characteristics of satellite hierarchy. This indicates the modulated bird's-eye view features. This indicates that a 1×1 convolution is used for channel alignment. This indicates the hierarchical satellite features after channel alignment. This represents the modulation bird's-eye view features after channel alignment, where Q', K', and V' represent the query, key, and value matrices, respectively. , , The projection matrix is ​​a learnable matrix, where Flatten represents the flattening operation and softmax represents the normalization operation. This represents the fusion feature obtained after attention fusion. Indicates the dimension of the key vector; The fused features obtained from attention fusion are then adaptively weighted and fused with the corresponding original level satellite features to obtain the weighted fused features. The calculation formula is as follows:

[0036]

[0037] in, This represents the Sigmoid activation function. This represents a learnable fusion parameter scalar. Represents the adaptive fusion coefficient. This represents the fusion feature obtained after attention fusion. Indicates the characteristics of satellite hierarchy. Indicates the features after weighted fusion; The differences between the weighted fused features and the corresponding original satellite features are then refined using residuals to obtain the enhanced features for each level. The calculation formula is as follows:

[0038]

[0039] in, Representing residual characteristics, This represents a residual refinement network that includes convolutional layers, normalization layers, and activation functions. This represents the hierarchical enhancement features of the final output; The obtained enhancement features at each level are refined from top to bottom across levels to obtain the final satellite image enhancement feature pyramid.

[0040] Furthermore, the satellite image enhancement feature pyramid is input into the instance segmentation network, and the instance segmentation results of buildings within the target area are output, specifically including: The instance segmentation network includes an object detection subnetwork and a mask prediction subnetwork; The target detection subnetwork processes the satellite image enhancement feature pyramid to generate candidate target regions and their category predictions. Based on the candidate target region and the enhanced feature pyramid, the building instance mask is output through the mask prediction subnetwork; The target detection subnetwork uses an anchor box mechanism to generate candidate target regions, and the mask prediction subnetwork uses a SAM2 mask decoder to generate instance masks.

[0041] Furthermore, the method also includes: During the training phase, optimization is performed using the total loss function. The format is as follows:

[0042] in, This represents the object detection loss, which includes classification loss and bounding box regression loss; Indicates instance segmentation loss; The loss term represents the spatial alignment constraint between satellite features and bird's-eye view features; This represents the loss term corresponding to the height supervision constraint based on the height information graph; and This represents the corresponding weighting coefficient.

[0043] Furthermore, the satellite image processing branch and the UAV image processing branch extract features using a pre-trained visual encoder, and use an efficient parameter fine-tuning method to adapt the pre-trained visual encoder to the remote sensing scene.

[0044] This invention provides a three-dimensional geographic city building instance segmentation method based on satellite-UAV collaboration, which has the following beneficial effects: 1. By jointly utilizing global scene information from satellite imagery and local geometric detail information from UAV multi-view images, the two types of information can work together in a unified representation space, thereby improving the accuracy of building instance segmentation; 2. By constructing a bird's-eye view feature and height information map under a unified bird's-eye view space, and by using a height-guided cross-view fusion mechanism to enhance the feature response of building-related areas, it is beneficial to improve the ability to express building boundaries and distinguish adjacent building instances. 3. By using multi-scale enhanced features to participate in instance segmentation, the method can balance the ability to cover a wide range of scenes with the ability to model local details, making it suitable for building instance segmentation tasks in complex urban building scenes. Attached Figure Description

[0045] Figure 1 A flowchart illustrating a three-dimensional geographic city building instance segmentation method based on satellite-UAV collaboration, as provided in one embodiment of the present invention; Figure 2 This is an overall network architecture diagram of a satellite-UAV collaborative three-dimensional geographic city building instance segmentation method provided in one embodiment of the present invention. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to the present invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0047] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0048] The first embodiment of this invention provides a method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration. The following is in conjunction with... Figure 1 and Figure 2 Please provide a detailed explanation.

[0049] like Figure 1 As shown, in step S100, satellite images and drone aerial images of the target area are acquired, and a satellite-drone scene pairing instance segmentation image dataset is constructed. The satellite images contain building instance-level labels, and the drone aerial images contain multi-view images and corresponding camera parameters.

[0050] The above steps specifically include: A satellite-UAV scene pairing instance segmentation dataset is constructed. In this embodiment, satellite-view images from the University-1652 dataset are manually annotated with building instances. For UAV-view image sequences, Structure from Motion (SfM) processing is performed to estimate the camera intrinsic and extrinsic parameters to suit the instance segmentation task of this invention. This embodiment includes UAV aerial images from four different perspectives. The constructed dataset is then divided into training, validation, and test sets.

[0051] like Figure 1 As shown, in step S200, the satellite image is input to the satellite image processing branch, multi-scale feature extraction is performed on the satellite image, and a satellite image feature pyramid is obtained.

[0052] The above steps specifically include: For the satellite image processing branch: Inputting a 512×512 resolution satellite image, multi-scale feature extraction is performed through four stages of the SAM2Hiera-Large backbone network. The output feature resolutions of each stage are 128×128, 64×64, 32×32, and 16×16, with corresponding channel numbers of 1152, 576, 288, and 144. Then, the features at each level are mapped to a unified channel dimension of 512. A multi-scale feature pyramid is constructed through the neck network of feature pyramid networks (FPN) using a top-down path. The final output feature pyramid contains five levels with resolutions of 128×128, 64×64, 32×32, 16×16, and 8×8.

[0053] The feature extraction process satisfies the following relationship:

[0054] in, The embedding representation of the input image. Indicates the first Feature maps output by the stage Indicates the first The Transformer block of the stage.

[0055] The feature pyramid neck network adopts a SAM2-based FPN architecture, and its forward propagation process satisfies the following relationship:

[0056]

[0057] in, Indicates the first The feature map output by the backbone network in each stage, FPN represents the feature pyramid neck network based on SAM2. This represents the features after FPN processing. The feature pyramid representing the final output Layers. For deeper layers of features, max pooling downsampling is performed on the deepest features to obtain:

[0058] The final result is a feature pyramid with five levels, with resolutions of 128×128, 64×64, 32×32, 16×16, and 8×8 respectively.

[0059] like Figure 1 As shown, in step S300, the drone aerial images are input to the drone image processing branch. Multi-scale feature extraction and depth estimation are performed on the drone aerial images from each viewpoint. The features from each viewpoint are combined with camera parameters to perform cross-viewpoint attention aggregation for depth perception, so as to generate bird's-eye view features in a unified bird's-eye view space. At the same time, the height information of each viewpoint is extracted based on the depth information of each viewpoint, and the height information of multiple viewpoints is aggregated to generate a height information map in the same bird's-eye view space as the bird's-eye view features.

[0060] The above steps specifically include: S310 uses a feature extraction network to extract multi-scale features from drone aerial images from each viewpoint, obtaining multi-scale drone image features for each viewpoint. These multi-scale drone image features are then mapped to a unified channel dimension and input into a feature pyramid network to generate multiple levels of drone image features for each viewpoint.

[0061] Specifically, for each 512×512 resolution UAV aerial image from each viewpoint, the SAM2Hiera-Large backbone network, which is the same as the satellite image processing branch, is used to extract multi-scale features. The multi-scale feature channel dimension of each viewpoint after processing by the backbone network and FPN is 512, which is projected to 128 dimensions through 1×1 convolution, and the feature scale covers 5 levels from 128×128 to 8×8.

[0062] S320 uses a depth estimation network to estimate the depth of the drone aerial images from each viewpoint, obtaining a depth map for each viewpoint. Then, it performs a downsampling operation on the depth map of each viewpoint using bilinear interpolation to adjust its scale to the resolution corresponding to each level of features (resolutions are 128×128, 64×64, 32×32, 16×16, and 8×8 respectively), obtaining the depth map corresponding to each level for each viewpoint.

[0063] Specifically, the DepthAnythingV2 depth estimation network is used to generate depth maps for unmanned aerial images from each viewpoint. This depth network employs a ViT-Small (Vision Transformer Small) encoder and is pre-trained on a large-scale depth dataset. It should be noted that the depth estimation network is not limited to DepthAnythingV2; other depth estimation networks (such as MiDaS, DPT, etc.) can also be used to implement this invention.

[0064] The S330, based on UAV image features and depth maps at each level from each viewpoint, employs a depth-aware cross-viewpoint attention mechanism to aggregate them, generating bird's-eye view features in a unified bird's-eye view space. .

[0065] In this embodiment, the depth-perception cross-view attention mechanism satisfies the following relationship: For each level l, l=1,2,…,L, where L is the number of levels, and for each viewpoint i, i=1,2,…,I, where I is the number of viewpoints; First, calculate the depth-sensing geometric embedding at the l-th layer from the i-th viewpoint:

[0066] in, Let represent the geometric embedding enhancement feature at level l from the i-th viewpoint. This represents the 3D point coordinates at the l-th level from the i-th viewpoint, calculated in world coordinates based on camera parameters. This represents the depth map corresponding to the l-th layer from the i-th viewpoint, where Embed represents the geometric embedding function and Encode represents the depth encoding function; It should be noted that, in this embodiment, based on the depth map at each level of each viewpoint, the depth value of each pixel in the depth map is back-projected to three-dimensional space through camera parameters to obtain the three-dimensional point coordinates in the camera coordinate system, and then transformed to the world coordinate system to obtain the three-dimensional point coordinates in the world coordinate system. ; Then calculate the query, key, and value matrix at level l from the i-th perspective:

[0067]

[0068]

[0069] in, Represents the calculated l-th viewpoint The unified bird's-eye view feature after 1-level aggregation (i.e., the aggregation result of the previous layer is used as the query feature of the current layer) is the value obtained by adding the learnable initial prior feature (randomly generated) and the position code (calculated by encoding according to the grid coordinates of the bird's-eye view) when l=1. Let represent the drone image features at level l from the i-th viewpoint. , , For learnable query, key, and value projection matrices, , , These represent the corresponding query, key, and value matrices, respectively. Next, calculate the attention output of the l-th level from the i-th viewpoint. :

[0070] in, The key vector dimension is represented by softmax; softmax represents the normalization operation. Then, average pooling is performed on the attention outputs of the l-th level from all perspectives to obtain the unified bird's-eye view feature of the l-th level:

[0071] Where I represents the number of viewpoints. ∑ represents the unified bird's-eye view feature after aggregation at level l across all viewpoints; ∑ represents the summation of the attention outputs across all viewpoints. Indicates the weight coefficients for average pooling; A unified bird's-eye view feature for each level across all perspectives. Then, a layer-by-layer refinement process is performed, including skip connections and Bottleneck convolutions:

[0072] in, Indicates the lth viewpoint The unified bird's-eye view features after 1-level aggregation, when l=1, are the initial BEV embedding priors; This represents the unified bird's-eye view feature after aggregation of the l-th level from all perspectives. This represents the refined bird's-eye view features of the l-th level after skip connections and Bottleneck convolutions across all viewpoints; Bottleneck represents a feature extraction module composed of multiple convolutional residual blocks.

[0073] After progressive processing, the final output is the refined bird's-eye view feature at the last layer, i.e., the Lth layer. As a unified aerial view feature:

[0074] in, This represents the unified bird's-eye view feature of the final output (with a resolution of 128×128 and a channel dimension of 128). This represents the detailed bird's-eye view features of the last layer, i.e., the Lth layer.

[0075] S340, then based on the depth map of each viewpoint, each pixel in the depth map... The depth value is back-projected to the 3D space through camera parameters to obtain the 3D point coordinates in the camera coordinate system. Then, the coordinates are transformed to the world coordinate system to obtain the 3D point coordinates (X,Y,Z) in the world coordinate system. The corresponding height value is obtained by extracting the Y coordinate component. The height values ​​of multiple viewpoints corresponding to the same bird's-eye view position are aggregated to finally generate a height information map with a resolution of 128×128 that is consistent with the bird's-eye view feature in the bird's-eye view space.

[0076] Specifically, the process of backprojecting depth values ​​into three-dimensional space satisfies the following relationship:

[0077] in, Represents pixel coordinates Depth value at that location, Indicates the coordinates of the principal point of the camera's intrinsic parameters. This represents the focal length parameter. This represents the coordinates of a three-dimensional point in the camera coordinate system.

[0078] The process of transforming 3D point coordinates from the camera coordinate system to the world coordinate system satisfies the following relationship:

[0079] in, Represents the homogeneous coordinates in the camera coordinate system. Represents the camera extrinsic parameter matrix. Let be a rotation matrix. It is a translation vector. Represents the coordinates of a three-dimensional point in the world coordinate system; the height value is determined by... of The coordinate components were extracted.

[0080] The height value aggregation process satisfies the following relationship:

[0081] in, Indicates the first The height value corresponding to each viewpoint Indicates the grid position from a bird's-eye view perspective In the Mapping position in each viewpoint Represents the number of viewpoints and is an integer greater than or equal to 2. This represents the aggregate height value at the grid position (u, v) from a bird's-eye view perspective.

[0082] Further information on altitude was obtained:

[0083] Where H(u,v) represents the aggregate height value at the grid position (u, v) from the bird's-eye view. This represents the obtained height information map.

[0084] like Figure 1 As shown, in step S400, the obtained height information map is used to perform weight modulation on the bird's-eye view features, and the features of each level of the satellite image multi-scale feature pyramid are respectively fused with the modulated bird's-eye view features to obtain the satellite image enhancement feature pyramid.

[0085] The above steps specifically include: This embodiment first encodes the height map to generate a height attention map, which makes the attention map more responsive in building areas with higher height values. Then, a gating mechanism is used to combine the height attention map with the bird's-eye view features to obtain a gating weight map, thereby modulating the bird's-eye view features, enhancing the feature response of building-related areas and suppressing interference from ground areas.

[0086] S410, the height information map is encoded to obtain height attention information.

[0087] The highly attentive encoding process satisfies the following relationship:

[0088]

[0089]

[0090]

[0091] in, This represents the input height information map. This represents the convolution operation. Indicates batch normalization, This represents the activation function. This represents the extracted basic height features. Indicates the enhanced height feature. This represents the generated attention score. express Activation function This represents highly attentional information after encoding.

[0092] S420, the bird's-eye view features are weighted based on the height attention information to enhance the feature response of the building-related areas.

[0093] The bird's-eye view feature-gated modulation process satisfies the following relationship:

[0094]

[0095] in, Indicates bird's-eye view features, This represents the splicing of highly attentional information with bird's-eye view features. This represents element-wise multiplication. This represents the learnable modulation intensity coefficient. Indicates the gating weight, This indicates the modulated bird's-eye view features.

[0096] Before performing attention fusion calculation, S430 first uses 1×1 convolution to align the hierarchical satellite features and the modulated bird's-eye view features to a unified channel dimension. Then, it uses the channel-aligned hierarchical satellite features as the query and the channel-aligned modulated bird's-eye view features as the key and value to generate fused features.

[0097] Specifically, the cross-perspective attention fusion process satisfies the following relationship:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] in, Indicates the characteristics of satellite hierarchy. This indicates the modulated bird's-eye view features. This indicates that a 1×1 convolution is used for channel alignment. This indicates the hierarchical satellite features after channel alignment. This represents the modulation bird's-eye view features after channel alignment, where Q', K', and V' represent the query, key, and value matrices, respectively. , , The projection matrix is ​​a learnable matrix, where Flatten represents the flattening operation and softmax represents the normalization operation. This represents the fusion feature obtained after attention fusion. This represents the dimension of the key vector.

[0104] In this embodiment, the attention mechanism is implemented using multi-head attention, satisfying the following relationship:

[0105]

[0106] in, , , These represent the query, key, and value matrices, respectively. , , Indicates the first The projection matrix of each attention head. Indicates the first The output of each attention head, Indicates the number of attention heads. This indicates a splicing operation. This represents the output projection matrix, and MultiHead(Q',K',V') represents the multi-head attention mechanism.

[0107] S440, then the residuals of the differences between the weighted fused features and the corresponding original level satellite features are refined to obtain the enhanced features of each level.

[0108] Specifically, the adaptive weighting and residual refinement process satisfy the following relationship:

[0109]

[0110]

[0111]

[0112] in, This represents the Sigmoid activation function. This represents a learnable fusion parameter scalar. Represents the adaptive fusion coefficient. This represents the fusion feature obtained by attention fusion. Indicates the characteristics of satellite hierarchy. This represents the features after weighted fusion. Representing residual characteristics, This represents a residual refinement network that includes convolutional layers, normalization layers, and activation functions. This represents the hierarchical enhancement features of the final output.

[0113] S450, the obtained enhancement features at each level are refined from top to bottom across levels to obtain the final satellite image enhancement feature pyramid. In this embodiment, the top-down cross-level feature refinement is based on a built-in component of the SAM2Hiera-Large architecture.

[0114] like Figure 1 As shown, in step S500, the satellite image enhancement feature pyramid is input into the instance segmentation network, and the instance segmentation results of buildings in the target area are output.

[0115] The above steps specifically include: An enhanced feature pyramid is input into the instance segmentation network to segment building instances based on 2D satellite image features. The instance segmentation network includes a Region Proposal Network (RPN), a Region of Interest (RoI) alignment layer, an RoI head, and a mask prediction head. The RPN generates candidate target regions, the RoI alignment layer maps features at different scales to a uniform size, the RoI head performs classification and bounding box regression, and the mask prediction head outputs a binary mask for the building instances.

[0116] In this embodiment, during the training phase, a multi-task loss function is used for end-to-end optimization, and the total loss function is expressed as:

[0117] in, This represents the object detection loss (including classification loss). and bounding box regression loss ), This represents the instance segmentation loss (mask prediction loss). For spatial alignment loss, This represents the high variance loss. and These are the weighting coefficients for spatial alignment loss and height variance loss, respectively. In this embodiment, they are set to 0.1 and 0.01, respectively.

[0118] Specifically, classification loss in object detection loss. Using cross-entropy loss, the following relationship is satisfied:

[0119] in, This represents the true label (1 for foreground, 0 for background). This represents the class probability predicted by the model.

[0120] Bounding box regression loss Using smoothed L1 loss, the following relationship is satisfied:

[0121] in, This represents the predicted bounding box parameters (including center coordinates and width and height). Represents the actual bounding box parameters. This represents the smoothed L1 loss function.

[0122] Instance segmentation loss Using binary cross-entropy loss, the following relationship is satisfied:

[0123] in, This represents the actual mask label (1 represents the target area, and 0 represents the background area). This represents the mask probability predicted by the model.

[0124] In this embodiment, and The weights of each are 1, and they are directly added to the total loss.

[0125] Highly supervised constraints in this embodiment For variance loss, the following relationship is satisfied:

[0126] in, This represents variance operations used to calculate high attention maps. The variance. The negative sign indicates maximizing the variance. Its purpose is to encourage high-attention maps to create a clear distinction between building areas and ground areas.

[0127] In this embodiment, the spatial alignment constraint is achieved by maximizing the similarity between satellite features and bird's-eye view features, satisfying the following relationship:

[0128] The similarity is obtained by calculating the cosine similarity of the normalized feature vectors:

[0129] in, Indicates satellite characteristics, Indicates bird's-eye view features, Let B represent the L2 norm of the vector, B represent the batch size, H represent the feature height, W represent the feature width, and C represent the number of feature channels. This represents the similarity value of the b-th sample, the h-th row, the w-th column, and the c-th channel.

[0130] This embodiment employs the following training strategy and hyperparameter configuration: The AdamW optimizer is used with a base learning rate of 1e-4 and a weight decay coefficient of 0.01. Cosine annealing is used for learning rate scheduling, with a total of 50 training epochs. The first 100 iterations are set as a linear warm-up phase. Differential learning rates are used: a 1.0x learning rate is applied to the SAM2Hiera-Large backbone network and the UAV branch, while a 2.0x learning rate is used for scene alignment-related modules to promote cross-view alignment. LoRA (Low-Rank Adaptation) is applied to the SAM2Hiera-Large backbone network for efficient parameter fine-tuning. LoRA adaptation is applied only to the qkv layer (query + key + value projection layer) and the proj layer (output projection layer), while full parameter fine-tuning is performed on the neck network. In one specific embodiment, the rank of the LoRA adapter is 16, and the scaling factor is 32.

[0131] To verify the effectiveness of the satellite-UAV cross-view collaborative building instance segmentation model designed in this embodiment, the performance of the model in this embodiment was compared with that of various models on the constructed satellite-UAV scene-paired instance segmentation image dataset. As can be seen from the results in Table 1, the model in this embodiment shows a significant improvement in performance metrics compared to existing methods. A description of each model is shown in Table 2.

[0132] Table 1 Performance of different models

[0133] Table 2 Introduction to Different Models

[0134] Those skilled in the art will understand that the above training strategies and hyperparameter configurations can be adjusted according to actual application scenarios and hardware conditions. For example, when memory is limited, the batch size can be reduced or a stronger FSDP sharding strategy can be enabled; in scenarios with large amounts of data, the number of training epochs can be increased or the learning rate scheduling strategy can be adjusted.

[0135] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A method for segmenting 3D geographic city building instances based on satellite-UAV collaboration, characterized in that, The method includes: Acquire satellite images and drone aerial images of the target area, and construct a satellite-drone scene pairing instance segmentation image dataset, where the satellite images contain building instance-level labels, and the drone aerial images contain multi-view images and corresponding camera parameters; The satellite image is input into the satellite image processing branch, where multi-scale feature extraction is performed on the satellite image, and a satellite image feature pyramid is obtained. The drone aerial images are input into the drone image processing branch. Multi-scale feature extraction and depth estimation are performed on the drone aerial images from each viewpoint. The features from each viewpoint are combined with camera parameters to perform cross-viewpoint attention aggregation for depth perception, so as to generate bird’s-eye view features in a unified bird’s-eye view space. At the same time, the height information of each viewpoint is extracted based on the depth information of each viewpoint, and the height information of multiple viewpoints is aggregated to generate a height information map in the same bird’s-eye view space as the bird’s-eye view features. The obtained height information map is used to perform weight modulation on the bird's-eye view features, and the features at each level of the satellite image multi-scale feature pyramid are fused with the modulated bird's-eye view features through cross-view attention fusion to obtain the satellite image enhancement feature pyramid. The satellite image enhancement feature pyramid is input into the instance segmentation network, and the instance segmentation results of buildings within the target area are output.

2. The method for segmenting 3D geographic city building instances based on satellite-UAV collaboration as described in claim 1, characterized in that, Acquire satellite and drone aerial images of the target area, and construct a satellite-drone scene pairing instance segmentation image dataset, specifically including: The obtained instance segmentation image dataset is divided into training, validation, and test sets for training, validation, and testing of the network model.

3. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 1, characterized in that, Satellite images are input into the satellite image processing branch, where multi-scale feature extraction is performed to obtain a satellite image feature pyramid, specifically including: A feature extraction network is used to extract multi-scale features from satellite images and output multi-scale satellite image features. Multi-scale satellite image features are mapped to a unified channel dimension and then input into a feature pyramid network to generate satellite image features at multiple levels.

4. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 1, characterized in that, Multi-scale feature extraction and depth estimation are performed on drone aerial images from each viewpoint. Combined with camera parameters, cross-viewpoint attention aggregation with depth perception is applied to the features from each viewpoint to generate bird's-eye view features in a unified bird's-eye view space. Specifically, this includes: Multi-scale feature extraction is performed on the drone aerial images from each viewpoint using a feature extraction network to obtain multi-scale drone image features for each viewpoint. These multi-scale drone image features are then mapped to a unified channel dimension and input into a feature pyramid network to generate multiple levels of drone image features for each viewpoint. A depth estimation network is used to estimate the depth of the drone aerial images from each viewpoint to obtain a depth map for each viewpoint. Then, a downsampling operation is performed on the depth map of each viewpoint using bilinear interpolation to adjust its scale to the resolution corresponding to the features of each level, so as to obtain the depth map corresponding to each level for each viewpoint. Based on the drone image features and depth maps at each level from each viewpoint, a depth-aware cross-viewpoint attention mechanism is used to aggregate them and generate bird's-eye view features in a unified bird's-eye view space.

5. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 4, characterized in that, Based on the UAV image features and depth maps at each level from each viewpoint, a depth-aware cross-viewpoint attention mechanism is used for aggregation to generate bird's-eye view features in a unified bird's-eye view space, specifically including: For each level l, l=1,2,…,L, where L is the number of levels, and for each viewpoint i, i=1,2,…,I, where I is the number of viewpoints; First, calculate the depth-sensing geometric embedding at the l-th layer from the i-th viewpoint: in, Let represent the geometric embedding enhancement feature at level l from the i-th viewpoint. This represents the 3D point coordinates at the l-th level from the i-th viewpoint, calculated in world coordinates based on camera parameters. This represents the depth map corresponding to the l-th layer from the i-th viewpoint, where Embed represents the geometric embedding function and Encode represents the depth encoding function; Then calculate the query, key, and value matrix at level l from the i-th perspective: in, Represents the calculated l-th viewpoint The unified bird's-eye view feature after 1-level aggregation, when l=1, takes the value of the initial embedded prior feature obtained by adding the randomly generated learnable initial prior feature to the bird's-eye view grid position code; Let represent the drone image features at level l from the i-th viewpoint. , , For learnable query, key, and value projection matrices, , , These represent the corresponding query, key, and value matrices, respectively. Next, calculate the attention output of the l-th level from the i-th viewpoint. : in, The key vector dimension is represented by softmax; softmax represents the normalization operation. Then, average pooling is performed on the attention outputs of the l-th level from all perspectives to obtain the unified bird's-eye view feature of the l-th level: Where I represents the number of viewpoints. This represents the unified bird's-eye view feature after aggregation of the l-th level from all perspectives; A unified bird's-eye view feature for each level across all perspectives. Then, a layer-by-layer refinement process is performed, including skip connections and Bottleneck convolutions: in, Indicates the lth viewpoint The unified bird's-eye view features after 1-level aggregation, when l=1, are the initial BEV embedding priors; This represents the unified bird's-eye view feature after aggregation of the l-th level from all perspectives. This represents the refined bird's-eye view features of the l-th level after skip connections and Bottleneck convolutions across all viewpoints. After progressive processing, the final output is the refined bird's-eye view feature at the last layer, i.e., the Lth layer. As a unified aerial view feature: in, This represents the unified bird's-eye view feature of the final output. This represents the detailed bird's-eye view features of the last layer, i.e., the Lth layer.

6. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 4, characterized in that, Based on the depth information of each viewpoint, the height information of each viewpoint is extracted, and the height information of multiple viewpoints is aggregated to generate a height information map of the same bird's-eye view space as the bird's-eye view feature. Specifically, this includes: Based on the depth map of each viewpoint, each pixel in the depth map is... The depth value is back-projected to the 3D space through the camera parameters to obtain the 3D point coordinates in the camera coordinate system. Then, the coordinates are transformed to the world coordinate system to obtain the 3D point coordinates (X,Y,Z) in the world coordinate system. The corresponding height value is obtained by extracting the Y coordinate component. The height values ​​of multiple viewpoints corresponding to the same bird's-eye view location are aggregated to generate a height map that unifies the bird's-eye view space with the aforementioned bird's-eye view features. The formula is as follows: in, Indicates the first The height value corresponding to each viewpoint Indicates the grid position from a bird's-eye view perspective In the Mapping position in each viewpoint Indicates the number of viewpoints. This represents the aggregate height value at the grid position (u, v) from a bird's-eye view perspective; in, This represents the height information map with a resolution of 128×128.

7. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 1, characterized in that, The bird's-eye view features are weighted and modulated using the obtained height information map, specifically including: The height information map is encoded to obtain height attention information, and the calculation formula is as follows: in, This represents the input height information map. This represents the convolution operation. Indicates batch normalization, This represents the activation function. This represents the extracted basic height features. Indicates the enhanced height feature. This represents the generated attention score. express Activation function This represents highly attentional information after encoding; The bird's-eye view features are weighted and modulated based on the height attention information to enhance the feature response of building-related areas. The calculation formula is as follows: in, Indicates bird's-eye view features, This represents the splicing of highly attentional information with bird's-eye view features. This represents element-wise multiplication. This represents the learnable modulation intensity coefficient. Indicates the gating weight, This indicates the modulated bird's-eye view features.

8. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 1, characterized in that, The satellite features at each level of the multi-scale feature pyramid of the satellite image are fused with the modulated bird's-eye view features through cross-view attention fusion to obtain the satellite image enhancement feature pyramid, which specifically includes: Before performing attention fusion calculation, the hierarchical satellite features and the modulated bird's-eye view features are first aligned to a unified channel dimension using a 1×1 convolution. Then, the channel-aligned hierarchical satellite features are used as queries, and the channel-aligned modulated bird's-eye view features are used as keys and values ​​to generate fused features. The attention fusion calculation formula is as follows: in, Indicates the characteristics of satellite hierarchy. This indicates the modulated bird's-eye view features. This indicates that a 1×1 convolution is used for channel alignment. This indicates the hierarchical satellite features after channel alignment. This represents the modulation bird's-eye view features after channel alignment, where Q', K', and V' represent the query, key, and value matrices, respectively. , , The projection matrix is ​​a learnable matrix, where Flatten represents the flattening operation and softmax represents the normalization operation. This represents the fusion feature obtained after attention fusion. Indicates the dimension of the key vector; The fused features obtained from attention fusion are then adaptively weighted and fused with the corresponding original level satellite features to obtain the weighted fused features. The calculation formula is as follows: in, This represents the Sigmoid activation function. This represents a learnable fusion parameter scalar. Represents the adaptive fusion coefficient. This represents the fusion feature obtained after attention fusion. Indicates the characteristics of satellite hierarchy. Indicates the features after weighted fusion; The differences between the weighted fused features and the corresponding original satellite features are then refined using residuals to obtain the enhanced features for each level. The calculation formula is as follows: in, Representing residual characteristics, This represents a residual refinement network that includes convolutional layers, normalization layers, and activation functions. This represents the hierarchical enhancement features of the final output; The obtained enhancement features at each level are refined from top to bottom across levels to obtain the final satellite image enhancement feature pyramid.

9. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 1, characterized in that, The satellite image enhancement feature pyramid is input into the instance segmentation network, which outputs instance segmentation results of buildings within the target area, specifically including: The instance segmentation network includes an object detection subnetwork and a mask prediction subnetwork; The target detection subnetwork processes the satellite image enhancement feature pyramid to generate candidate target regions and their category predictions. Based on the candidate target region and the enhanced feature pyramid, the building instance mask is output through the mask prediction subnetwork; The target detection subnetwork uses an anchor box mechanism to generate candidate target regions, and the mask prediction subnetwork uses a SAM2 mask decoder to generate instance masks.

10. The method for segmenting three-dimensional geographic city building instances based on satellite-UAV collaboration as described in claim 1, characterized in that, The method further includes: During the training phase, optimization is performed using the total loss function. The format is as follows: in, This represents the object detection loss, which includes classification loss and bounding box regression loss; Indicates instance segmentation loss; The loss term represents the spatial alignment constraint between satellite features and bird's-eye view features; This represents the loss term corresponding to the height supervision constraint based on the height information graph; and This represents the corresponding weighting coefficient.

Images