A Building Extraction Method from Airborne Oblique Remote Sensing Images Based on Boxes and Image-Level Annotations

By combining bounding box annotation and image-level annotation into an airborne oblique remote sensing image building extraction model, and utilizing pseudo-instance masking refinement and peak attention modules, the problems of large data annotation volume and insufficient information in building information extraction from remote sensing images are solved, achieving efficient and accurate building extraction.

CN117593642BActive Publication Date: 2026-11-17HARBIN ENG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311589951.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-24
Publication Date
2026-11-17
Estimated Expiration
2043-11-24

AI Technical Summary

Technical Problem

In existing technologies, methods for extracting building information from remote sensing images rely on a large number of pixel-level annotations, resulting in a large workload for data annotation. Furthermore, insufficient image-level annotations and bounding box annotations lead to problems such as mask overlap and foreground/background classification errors.

Method used

A building extraction model based on bounding box annotation and image-level annotation for airborne oblique remote sensing images is adopted. Redundant boundary information is filtered out by a pseudo-instance mask refinement module. Combined with a peak attention module and a mask classification refinement module, the mask boundary is optimized and background pixels are reduced to achieve accurate building extraction.

Benefits of technology

It effectively reduces the need for data annotation, improves the accuracy and efficiency of building extraction, solves the problems of mask overlap and unclear boundaries, and achieves efficient building information extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117593642B_ABST
    Figure CN117593642B_ABST
Patent Text Reader

Abstract

This invention relates to a method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations, belonging to the field of remote sensing image information extraction technology. To address the issues of relying on large amounts of annotation information for pixel-level instance segmentation and mask overlap caused by insufficient image-level annotation information, this invention employs a strategy combining a weakly supervised instance segmentation network with a pseudo-mask refinement module to generate pseudo-instance masks. A pseudo-instance mask boundary refinement module filters out redundant boundary information in the pseudo-instance masks, and supplements and optimizes the mask boundary information by incorporating information from the remote sensing image. Simultaneously, a peak attention module is used to obtain the peak response map of instances after enhanced peak attention, extracting precise location clues for the instances. This, combined with the boundary-refined pseudo-instance mask, confirms the center point position and offset map of the target building. Finally, a mask classification refinement module obtains more accurate instance information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image information extraction technology, and specifically relates to a method for extracting information about buildings in remote sensing images. Background Technology

[0002] In the field of remote sensing, the shooting perspective can be divided into frontal and oblique perspectives. Frontal remote sensing images are limited by the shooting angle, only capturing the texture of the building's top and a portion of its facade, failing to comprehensively capture information about obscured parts of the building. Oblique remote sensing images, on the other hand, observe ground features from multiple angles, providing a more comprehensive view of the building's top and facade, and also revealing information about any obstructions. Therefore, in order to extract information from the building's top and facade, obtaining information such as building size, outline, shape, and surface texture, in-depth research on oblique remote sensing images is necessary.

[0003] With the continuous development of deep learning, various methods such as object detection, semantic segmentation, instance segmentation, and panoramic segmentation can be used to extract information about buildings in remote sensing images. Instance segmentation, in particular, can accurately locate and classify multiple targets and extract pixel-level information, meeting the requirements for building extraction in airborne oblique remote sensing images. However, traditional instance segmentation heavily relies on a large amount of pixel-level annotation information, resulting in significant time and manpower waste. Furthermore, the ever-expanding remote sensing datasets further increase the difficulty of data annotation, hindering the further development of instance segmentation. In 2018, scholars introduced weakly supervised learning methods into instance segmentation tasks, greatly reducing the reliance on pixel-level annotations and lowering the workload. Currently, common weakly supervised instance segmentation methods use point annotations, image-level annotations, and bounding box annotations, requiring only a small number of point annotations, one image category annotation, and one tight rectangular box annotation, respectively. Point annotations significantly reduce annotation time but cannot provide detailed descriptions of the instance's shape, size, and other features, resulting in poor segmentation performance. Image-level annotations only require annotating the category of the target in the image, further reducing the reliance on labels. Bounding box annotations provide more detailed and accurate annotation information, and the annotation process takes significantly less time compared to pixel-level annotations. They are suitable for large-scale image data and can even achieve segmentation performance comparable to pixel-level annotations.

[0004] However, image-level annotations and bounding box annotations still provide relatively weak annotation information. When applying image-level annotations to instance segmentation tasks, they may fail to distinguish between different instances of the same category, leading to mask overlap. When applying bounding box annotations to instance segmentation tasks, since bounding box annotations do not provide building outline information, they are prone to foreground and background classification errors in the segmentation results, causing problems such as unclear mask boundaries and the inclusion of too many unnecessary background pixels.

[0005] Secondly, when performing bounding-supervised instance segmentation on airborne tilted remote sensing images with complex scenes, mask overlap may occur due to occlusion relationships between buildings. Summary of the Invention

[0006] This invention addresses the problem of relying on a large amount of annotation information for pixel-level instance segmentation, as well as the problem of mask overlap caused by insufficient image-level annotation information.

[0007] A method for extracting buildings from airborne oblique remote sensing images based on bounding box and image-level annotation is proposed. This method uses an airborne oblique remote sensing image building extraction model based on bounding box and image-level annotation to extract individual building regions from the remote sensing images to be processed.

[0008] The aforementioned building extraction model for airborne oblique remote sensing images based on bounding box annotation and image-level annotation is obtained through the following steps:

[0009] Step 1: Crop the original large remote sensing image and the semantic map obtained from the annotation to obtain a small remote sensing image;

[0010] Only remote sensing thumbnails containing buildings are labeled as "buildings" as image-level annotation information. The smallest bounding rectangle of the building in the cropped image is taken as the ground truth information of its corresponding bounding box. The remote sensing thumbnails are directly recorded as remote sensing images. The original dataset is constructed based on the cropped remote sensing images, image-level annotation information and bounding box ground truth information. The training set is determined based on the original dataset.

[0011] Step 2: Transfer the remote sensing images X from the training set i With bounding box truth information G box The pseudo-instance mask x is generated in the weakly supervised instance segmentation network. p ;

[0012] Step 3: Mask the pseudo-instance x p In the input pseudo-instance mask refinement module, redundant boundary information is filtered out, and missing boundary information is supplemented and optimized to achieve pseudo-instance mask x. p Boundary refinement; the processing flow of the pseudo-instance mask refinement module is as follows:

[0013] S31, using a pseudo-instance mask xp Centered on the boundary pixels, a sliding window strategy is used to densely apply the pseudo-instance mask x. p With remote sensing image X i Several square mask blocks and image blocks of the same size with side length 'a' are extracted from the image. The set of mask blocks is then represented as {P}. m The set of image patches is represented as {P}. i};

[0014] S32. Using nonmaximum suppression, for the mask block set {P} m The mask blocks containing redundant boundary information in {P} are suppressed, and the set of mask blocks after filtering out redundant boundary information is represented as {P'}. m The set of image blocks corresponding to the preserved mask blocks is represented as {P'}. i};

[0015] S33, the set of mask blocks {P' after filtering out redundant boundary information m The set of image patches {P'} corresponding to the preserved mask patches i The input is refined into the network, where boundary information of the mask block is optimized.

[0016] S331. Convert the number of input channels to 4, where the number of channels for the mask block is 1 and the number of channels for the image block is 3, with each of the RGB values ​​of the image block occupying one channel; the mask block provides positional and semantic information.

[0017] S332. Perform convolutional downsampling on the feature map, and then send it to the first stage of processing;

[0018] S333. In the first stage, four parallel multi-scale group convolutions are performed on the input feature map. The results of the four parallel multi-scale group convolutions are fused together in a concatenated manner, and the result is input into the first transition stage. Then, the first transition stage is processed as follows: the input feature map is transformed by convolution to output a downsampled feature map with a scale of 4 times; at the same time, the input feature map is transformed by convolution to output a downsampled feature map with a scale of 8 times; finally, downsampled feature maps with scales of 4 times and 8 times are output respectively; and then sent to the first stage for processing.

[0019] S334. In the second stage, four parallel multi-scale group convolutions are first performed on the 4x and 8x downsampled feature maps respectively; the results of the four parallel multi-scale group convolutions are fused together by adding them element by element to obtain the fused 4x and 8x downsampled feature maps respectively.

[0020] Then, multi-scale convolutional fusion is performed on the 4x and 8x downsampled feature maps after parallel multi-scale group convolutions respectively; then, ReLU activation is performed on the two fused feature maps respectively, and they enter the second transition stage respectively; in the second transition stage, convolution transformation is used on the 8x downsampled feature map to output a downsampled feature map with a scale of 16x.

[0021] Finally, downsampled feature maps with scales of 4x, 8x, and 16x are output respectively; then they are sent to the third stage of processing.

[0022] S335. In the third stage, firstly, four parallel multi-scale group convolutions are performed on the 4x, 8x, and 16x downsampled feature maps respectively. The results of the four parallel multi-scale group convolutions are then fused together by adding them element by element to obtain the fused 4x, 8x, and 16x downsampled feature maps respectively.

[0023] Then, multi-scale convolutional fusion is performed on the 4x, 8x, and 16x downsampled feature maps after parallel multi-scale group convolutions respectively; then, ReLU activation is performed on the three fused feature maps respectively, and they enter the third transition stage respectively; in the third transition stage, convolution transformation is used on the 16x downsampled feature map to output a downsampled feature map with a scale of 32x.

[0024] Finally, downsampled feature maps with scales of 4x, 8x, 16x, and 32x are output respectively; then they are sent to the fourth stage of processing.

[0025] S336. In the fourth stage, firstly, parallel multi-scale group convolutions are performed on the downsampled feature maps of 4x, 8x, 16x, and 32x respectively. Each multi-scale group convolution has the same structure as the multi-scale group convolution in S334. The results of the four parallel multi-scale group convolutions are fused together by adding them element by element to obtain the fused downsampled feature maps of 4x, 8x, 16x, and 32x respectively.

[0026] Then, multi-scale convolutional fusion is performed on the downsampled feature maps that have undergone parallel multi-scale group convolutions at 4x, 8x, 16x, and 32x respectively; then, ReLU activation is performed on the four fused feature maps respectively; finally, downsampled feature maps with scales of 4x, 8x, 16x, and 32x are output after feature fusion.

[0027] S337. The three low-resolution feature maps of 8x, 16x and 32x are scaled up by bilinear upsampling and uniformly scaled to a high-resolution feature map of 4x and then fused.

[0028] S338. The mixed feature map is convolved with a 1×1 convolution, and its size is changed to a mask block with side length a. The final output is a binarized fine segmentation result, which is the optimized mask block set {P”.m};

[0029] S34. Mask the pseudo-instance x p In the middle, the original mask block is replaced with the optimized mask block {P”. m After re-merging the optimized mask blocks, a compact and precise mask is formed, completing the pseudo-instance mask x. p The boundary refinement represents the new pseudo-instance mask as x. n ;

[0030] Step 4: Extract the intermediate feature map X∈R from the feature extraction process in the weakly supervised instance segmentation network of Step 2. H×W×K , where H, W, and K are the height, width, and number of channels of the feature map, respectively; global max pooling is performed on the feature map X to obtain the peak response map P corresponding to the feature map;

[0031] Step 5: Input the peak response map P into the peak attention module for processing, and generate the peak response map P. s ;

[0032] Step 6: Combine the pseudo-instance mask x generated in the pseudo-instance mask refinement module. n Peak response map P generated by the peak attention module s Confirm the pseudo-center image C p With pseudo offset map O p To represent instance-level information within the foreground region, the pseudo-center graph C is ignored. p With pseudo offset map O p The area considered as background in the middle;

[0033] Step 7: Then, the code is sent to the code classification and refinement module for processing. In the code classification and refinement module, the offset map O is first processed. m Central clustering; then the results of central clustering are combined with the pseudo-instance mask x. n Central Figure C m With offset map O m Jointly supervise the completion of the pseudo-instance mask x n The instances are grouped; based on the results of the instance grouping, the refined center map C can be extracted. r Compared with the refined offset map O r ;

[0034] Based on the above steps, iterative training is performed using the training set to obtain a trained building extraction model for airborne oblique remote sensing images based on bounding box annotation and image-level annotation.

[0035] Furthermore, the remote sensing images X in the training set i With bounding box truth information G boxThe pseudo-instance mask x is generated in the weakly supervised instance segmentation network. p The process includes the following steps:

[0036] S21. Utilize Feature Pyramid Network (FPN) to process the input remote sensing image X. i Feature extraction of buildings is performed, and the extracted features are rated as F1.

[0037] S22. For the extracted feature F1, use the bounding box ground truth information G. box Constraints are applied by removing feature information outside the truth box in F1 and retaining only the feature information within the truth box, thus obtaining the input remote sensing image X. i Feature map F2 within the bounding box;

[0038] S23, From the bounding box truth information G box In the corresponding bounding box feature map F2, the feature information of each pixel i is obtained, and the color difference C between each pixel i and its eight neighboring pixels j is calculated. (i,j) ;

[0039] S24. From different bounding box truth information G box Internally, pixel information is obtained to obtain the spatial correspondence S between pixels. (i,k) i and k are two different truth information r of bounding boxes. n r s pixels within;

[0040] S25, adjust the color difference C of neighboring pixels (i,j) and spatial correspondence S (i,k) As weighting information, it is embedded into the remote sensing image X. i On the feature map F2 within the bounding box, the pseudo-instance mask x is obtained. p =C (i,j) ·S (i,k) ·F2.

[0041] Furthermore, the aforementioned color difference Among them, f i with f j These are the ROI features of pixel i and its eight neighboring pixels j in the in-boundary feature map F2; |f i |、|f j | is the magnitude of the ROI feature of pixel i and its eight neighboring pixels j in the feature map F2 within the bounding box.

[0042] Furthermore, the spatial correspondence between the pixels Where j represents the truth value information of the bounding box r n The pixels within the eight neighborhoods of i, where l is the ground truth information of the bounding box r. s pixels within the eight neighborhoods of inner k; offi,k For r n Medium pixel i and r s The relative spatial offset between pixels k, off j,l For r n Medium pixel j and r s The relative spatial offset between pixels l, where T(j,l) is the bounding box ground truth information r. n r s The soft assignment matrix is ​​used to determine the matching score between the truth information of the two bounding boxes.

[0043] Furthermore, in the first stage of S333, during the process of performing four parallel multi-scale group convolutions, each multi-scale group convolution first uses 1×1 convolution to reduce the dimensionality of the feature map, that is, to reduce the number of channels of the input feature map, then uses 3×3 convolution to complete feature extraction, and finally uses 1×1 convolution to increase the dimensionality of the feature map to restore the number of channels of the input feature map.

[0044] In S334, during the second stage of performing four parallel multi-scale group convolutions on the 4x and 8x downsampled feature maps, each multi-scale group convolution first uses two 3×3 convolutions to extract the input features; then the features that have not undergone multi-scale group convolutions are directly added to the outputs of the two 3×3 convolutions to obtain the final output feature map in each multi-scale group convolution.

[0045] Each of the four multi-scale group convolutions in the third stage of S335 has the same multi-scale group convolution structure as S334.

[0046] Each of the four multi-scale group convolutions in the fourth stage of S336 has the same multi-scale group convolution structure as that in S334.

[0047] Furthermore, the process of multi-scale convolutional fusion in S334 for the 4x and 8x downsampled feature maps after parallel multi-scale group convolutions is as follows:

[0048] The 8x downsampled feature map is upsampled by 2x and then fused with the 4x downsampled feature map. At the same time, the 4x downsampled feature map is downsampled by 2x and then fused with the 8x downsampled feature map.

[0049] Furthermore, the multi-scale convolutional fusion process in S335 of the downsampled feature maps after parallel multi-scale group convolutions at 4x, 8x, and 16x times includes the following steps:

[0050] The 8x downsampled feature map is upsampled by 2x, and the 16x downsampled feature map is upsampled by 4x. Then, the feature maps obtained by fusing the 4x downsampled feature map, the 8x downsampled feature map after 2x upsampling, and the 16x downsampled feature map after 4x upsampling are fused together.

[0051] Furthermore, the process of multi-scale convolutional fusion in S336 of downsampled feature maps that have undergone parallel multi-scale group convolutions at 4x, 8x, 16x, and 32x times includes the following steps:

[0052] The 8x downsampled feature map is upsampled by 2x, the 16x downsampled feature map is upsampled by 4x, and the 32x downsampled feature map is upsampled by 8x. Then, the feature maps obtained from the 4x downsampled feature map, the 8x downsampled feature map (upsampled by 2x), the 16x downsampled feature map (upsampled by 4x), and the 32x downsampled feature map (upsampled by 8x) are fused together. Simultaneously, the 4x downsampled feature map is downsampled by 2x, the 16x downsampled feature map is upsampled by 2x, and the 32x downsampled feature map is upsampled by 4x. Finally, the feature maps obtained from the 8x downsampled feature map, the 4x downsampled feature map (upsampled by 2x), the 16x downsampled feature map (upsampled by 2x), and the 32x downsampled feature map (upsampled by 4x) are fused together. Simultaneously, the 4x downsampled feature map is downsampled by 4x, the 8x downsampled feature map is downsampled by 2x, and the 32x downsampled feature map is upsampled by 2x. Then, the feature maps obtained by downsampling the 16x downsampled feature map, the 4x downsampled feature map, the 8x downsampled feature map, and the 32x downsampled feature map are fused together.

[0053] Furthermore, step five, which involves inputting the peak response map P into the peak attention module for processing, includes the following steps:

[0054] S51. For the input peak response map P, a selector is used to select the unique peak region with the largest peak value among the multiple peak regions corresponding to each building in the peak response map P. All points within this peak region are then extracted. Let there be K points, which are denoted as K standard points. The result after selection is represented as S. p ∈R 1×1×K ;

[0055] S52. For the input peak response map P, a controller is used to enhance attention. The controller extracts K standard points within the peak region from the selector, and applies a control value to each standard point. This value represents the degree to which the peak attention module enhances attention to the peak region. The control value is represented as G. p ∈[0,1] 1×1×K ;

[0056] S53. Then, peak stimulator processing is used, in which element-wise multiplication of τ is employed. p =S p ·G p Set peak region boundaries for the peak region; if the value of the standard point extracted by the selector is higher than τ. p If the value is zero, the point is considered to be in the peak region and will receive special attention; otherwise, it is considered to belong to the noise region, set to zero, and disabled.

[0057] S54. Finally, generate the peak response map P after enhancing peak attention. s .

[0058] Furthermore, the specific process of step six includes the following steps:

[0059] S61, in the peak response diagram P s In the middle, each peak region corresponds to a pseudo-instance mask x. n In an example building, max pooling is performed on the retained peak regions. This operation helps remove overlapping masks. If the values ​​of the peak regions are the same before and after max pooling, then the center information obtained after max pooling is considered a pseudo-center image C. p A portion; if the values ​​in the peak regions are different, then the center information obtained after max pooling of the peak regions is considered not to belong to the pseudo-center graph C. p Part of it, ultimately generating the pseudo-center graph C p ;

[0060] S62, Based on the pseudo-center diagram C pThe peak regions that are preserved in the original intermediate feature map X are used to extract the feature information of the buildings corresponding to these peak regions. The two-dimensional offset vector information of these buildings relative to the peak regions is then calculated using these features; this offset information is the pseudo-offset map O. p .

[0061] Furthermore, the instance grouping process in step seven includes the following steps:

[0062] First, remove the pseudo-instance mask x n Background information is used, retaining only the foreground mask information belonging to buildings; secondly, the central image C is used. m Identify the center position for each instance; then, use the offset map O. m The result of central clustering is used for the pseudo-instance mask x. n The boundary information is corrected; finally, the pseudo-instance mask x is used as the basis for the correction. n The extracted foreground mask information of the buildings and the corrected boundary information are used to generate an optimized mask, which is then used based on the center image C. m The central location corresponding to each instance is identified to distinguish each instance and achieve instance grouping.

[0063] Beneficial effects:

[0064] This invention addresses the problem that instance segmentation methods based on pixel-level annotation rely on large amounts of data annotation. It proposes a method for extracting buildings from airborne oblique remote sensing images based on bounding box and image-level annotations. By combining bounding box annotation with image-level annotation, building extraction from airborne oblique remote sensing images can be achieved simply by creating tight bounding box annotations for buildings and adding image-level annotations indicating that the instance category is "building" in the image. To address the issues of unclear mask boundaries and excessive unnecessary background pixels caused by insufficient bounding box annotation information, a pseudo-instance mask refinement module is proposed. To address the mask overlap problem caused by insufficient image-level annotation information, a peak attention module and a mask classification refinement module are proposed to optimize the prediction mask. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the process for extracting buildings from airborne oblique remote sensing images.

[0066] Figure 2 This is an overall block diagram for building extraction from airborne oblique remote sensing images based on bounding box annotation and image-level annotation.

[0067] Figure 3 This is a schematic diagram of the pseudo-instance mask refinement module.

[0068] Figure 4 This is a schematic diagram of the peak attention module.

[0069] Figure 5 This is a schematic diagram of the mask classification and refinement module. Detailed Implementation

[0070] To address the issue of relying on extensive annotation information for pixel-level instance segmentation, this invention proposes a method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations. This invention also solves the problems of difficulty in distinguishing foreground and background within the bounding box mask and unclear boundaries caused by insufficient bounding box annotation information, as well as the mask overlap problem caused by insufficient image-level annotation information.

[0071] This invention employs a strategy combining a weakly supervised instance segmentation network with a pseudo-mask refinement module to generate pseudo-instance masks. The pseudo-instance mask boundary refinement module filters out redundant boundary information and eliminates the influence of background pixels. Furthermore, this module incorporates information from remote sensing images to supplement and optimize the mask's boundary information. This pseudo-instance mask boundary refinement method only requires image information to provide richer detail information for instance boundaries, meeting the requirements of weakly supervised learning. Next, a peak attention module is used to obtain the peak response map of instances after enhanced peak attention, extracting precise location clues for the instances. This, combined with the refined pseudo-instance mask, confirms the center point position and offset map of the target building. A mask classification refinement module is then used to obtain more accurate instance information, eliminating mask overlap problems caused by insufficient image-level annotation information. The following detailed description of the specific implementation method further illustrates this.

[0072] Specific implementation method one: Combining Figure 1 and Figure 2 This implementation method is described below.

[0073] This implementation method is a building extraction method for airborne oblique remote sensing images based on bounding boxes and image-level annotations, including the following steps:

[0074] Step 1: Crop the original large remote sensing image and the semantic image obtained from the annotation to obtain a small remote sensing image of 640 pixels × 640 pixels.

[0075] Only remote sensing thumbnails containing buildings are labeled as "buildings" as image-level annotation information. The minimum bounding rectangle of the building in the cropped semantic image is taken as the ground truth information of its corresponding bounding box. The remote sensing thumbnails are directly recorded as remote sensing images. The original dataset is constructed based on the cropped remote sensing images, image-level annotation information and bounding box ground truth information, and the original dataset is divided into training set and test set.

[0076] Step 2: Transfer the remote sensing images X from the training set i With bounding box truth information G box The pseudo-instance mask is generated in the weakly supervised instance segmentation network. The specific workflow is as follows:

[0077] S21. Utilize Feature Pyramid Network (FPN) to process the input remote sensing image X. i The building features are extracted, and the extracted features are called F1.

[0078] S22. For the extracted feature F1, use the bounding box ground truth information G. box Constraints are applied by removing feature information outside the truth box in F1 and retaining only the feature information within the truth box, thus obtaining the input remote sensing image X. i The feature map F2 within the bounding box.

[0079] S23, From the bounding box truth information G box In the corresponding bounding box feature map F2, the feature information of each pixel i is obtained, and the color difference C between each pixel i and its eight neighboring pixels j is calculated. (i,j) The truth information G of the same bounding box was obtained. box The relationship between pixels within the domain.

[0080]

[0081] Among them, f i with f j These are the ROI features of pixel i and its eight neighboring pixels j in the in-boundary feature map F2; |f i |、|f j | is the magnitude of the ROI feature of pixel i and its eight neighboring pixels j in the feature map F2 within the bounding box.

[0082] S24. From different bounding box truth information G box Retrieve pixel information, where i and k are the ground truth information of two different bounding boxes r. n r s Calculate the spatial correspondence S between pixels within the range. (i,k) This allows us to obtain the dense correspondence between pixels within the ground truth information of different bounding boxes.

[0083]

[0084] Where j represents the truth value information of the bounding box r n The pixels within the eight neighborhoods of i, where l is the ground truth information of the bounding box r. s pixels within the eight neighborhoods of inner k; off i,k For r n Medium pixel i and r s The relative spatial offset between pixels k, off j,l For r n Medium pixel j and r s The relative spatial offset between pixels l, where T(j,l) is the bounding box ground truth information r.n r s The soft allocation matrix between them.

[0085] Relative offset represents the offset between a pixel within one bounding box and a corresponding pixel within another bounding box, including offsets in both the horizontal and vertical directions. The so-called corresponding position does not require that the pixel positions in the two bounding boxes correspond exactly. Instead, it is based on the position ratio of a certain pixel in the first bounding box relative to its position within the first bounding box. Then, according to the position ratio, the corresponding pixel position in the second bounding box is determined, and the relative offset between the two is calculated.

[0086] The soft assignment matrix records the degree of matching between two boxes, where each element represents the matching score between the ground truth information of the two bounding boxes. The matching score is determined based on the overlap metric IOU to indicate the degree of matching between them.

[0087] S25, adjust the color difference C of neighboring pixels (i,j) and spatial correspondence S (i,k) As weighting information, it is embedded into the remote sensing image X. i On the feature map F2 within the bounding box, the pseudo-instance mask x is obtained. p The specific implementation process is shown in the following formula:

[0088] x p =C (i,j) ·S (i,k) ·F2

[0089] Step 3: Mask the pseudo-instance x p In the input pseudo-instance mask refinement module, redundant boundary information is filtered out, and missing boundary information is supplemented and optimized to achieve pseudo-instance mask x. p Boundary refinement. For example... Figure 3 As shown, the processing flow of the pseudo-instance mask refinement module is as follows:

[0090] S31, using a pseudo-instance mask x p Centered on the boundary pixels, a sliding window strategy is used to densely apply the pseudo-instance mask x. p With remote sensing image X i Several square mask blocks and image blocks of the same size with side length 'a' are extracted from the image. The set of mask blocks is then represented as {P}. m The set of image patches is represented as {P}. i}

[0091] S32. Using nonmaximum suppression (NMS), apply it to the mask block set {P}. mBy suppressing mask blocks containing redundant boundary information, mask blocks with overlapping information can be effectively eliminated, retaining only those mask blocks that require further boundary information supplementation, thereby reducing information redundancy. The set of mask blocks after filtering out redundant boundary information is represented as {P'}. m The set of image blocks corresponding to the preserved mask blocks is represented as {P'}. i}

[0092] S33, the set of mask blocks {P' after filtering out redundant boundary information m The set of image patches {P'} corresponding to the preserved mask patches i The input is refined into the network, where boundary information of the mask block is optimized.

[0093] S331. Convert the number of input channels to 4, where the mask block has 1 channel and the image block has 3 channels, with each of the RGB values ​​in the image block occupying one channel. The mask block provides both positional and semantic information, so the refined network does not need instance-level semantics to complete mask optimization; it only needs to learn the pixels at the boundaries and perform binary classification of foreground and background.

[0094] S332. First, a 3×3 convolution kernel is used on the input feature map to perform a convolution transformation with a stride of 2 and padding of 1 to complete downsampling, and the number of channels is converted to 64. Then, it goes through a batch normalization (BN) layer and ReLU activation.

[0095] Then, a 3×3 convolution kernel is used again to perform a convolution transformation on the image with a stride of 2 and padding of 1 to complete the downsampling, converting the number of channels to 64, and then passing it through a batch normalization (BN) layer and ReLU activation.

[0096] The feature map after two downsampling processes can effectively reduce the resolution and computational load, and then be input into the first stage.

[0097] S333. In the first stage, four parallel multi-scale group convolutions are performed on the input feature map. In this process, each multi-scale group convolution first uses a 1×1 convolution to reduce the dimensionality of the feature map, that is, to reduce the number of channels of the input feature map, then uses a 3×3 convolution to complete feature extraction, and finally uses a 1×1 convolution to increase the dimensionality of the feature map to restore the number of channels of the input feature map. The results of the four parallel multi-scale group convolutions are fused together in a concatenated manner, and the result is input into the first transition stage.

[0098] Then, the first transition stage is initiated: a 3×3 convolution kernel is used to perform a convolution transformation on the input feature map with a stride of 1 and padding of 1, outputting a downsampled feature map with a scale of 4, and activated using Batch Normalization (BN) and ReLU. Simultaneously, a 3×3 convolution kernel is used to perform a convolution transformation on the input feature map with a stride of 2 and padding of 1, outputting a downsampled feature map with a scale of 8, and activated using BN and ReLU. Finally, downsampled feature maps with scales of 4 and 8 are output respectively.

[0099] S334. In the second stage, four parallel multi-scale group convolutions are first performed on the 4x and 8x downsampled feature maps respectively. The multi-scale group convolution processes of these two feature maps do not affect each other. In this process, each multi-scale group convolution first uses two 3×3 convolutions to extract the representation of the input features and learn high-level feature information. Then, the features that have not undergone multi-scale group convolution are directly added to the outputs of the two 3×3 convolutions (i.e., Basic block processing) to obtain the final output feature map in each multi-scale group convolution. This multi-scale group convolution allows gradients to propagate directly to shallower layers, which helps to alleviate the gradient vanishing problem and is helpful for training very deep networks. The results of the four parallel multi-scale group convolutions are fused together in an element-wise addition manner to obtain the fused 4x and 8x downsampled feature maps respectively.

[0100] Then, multi-scale convolutional fusion is performed on the 4x and 8x downsampled feature maps after parallel multi-scale group convolution: the 8x downsampled feature map is upsampled by 2x and then fused with the 4x downsampled feature map, and the 4x downsampled feature map is downsampled by 2x and then fused with the 8x downsampled feature map.

[0101] Next, ReLU activation is applied to both fused feature maps, and each enters the second transition stage. In the second transition stage, a 3×3 convolution kernel is used to perform a convolution transformation with a stride of 2 and padding of 1 on the 8x downsampled feature map, outputting a 16x downsampled feature map, which is then activated using BN and ReLU. Finally, downsampled feature maps with scales of 4x, 8x, and 16x are output, respectively.

[0102] S335. In the third stage, four parallel multi-scale group convolutions are first performed on the downsampled feature maps at 4x, 8x, and 16x magnification, respectively. Each multi-scale group convolution has the same structure as the multi-scale group convolution in S334. The results of the four parallel multi-scale group convolutions are then fused together element-wise to obtain the fused downsampled feature maps at 4x, 8x, and 16x magnification, respectively. The multi-scale group convolution processes for these three feature maps do not affect each other.

[0103] Then, multi-scale convolutional fusion is performed on the 4x, 8x, and 16x downsampled feature maps after parallel multi-scale group convolutions: the 8x downsampled feature map is upsampled by 2x, and the 16x downsampled feature map is upsampled by 4x. Then, the feature maps after the 4x downsampled feature map, the 8x downsampled feature map after the 2x upsampled feature map, and the 16x downsampled feature map after the 4x upsampled feature map are fused. Simultaneously, the 4x downsampled feature map is downsampled by 2x, and the 16x downsampled feature map is upsampled by 2x. Then, the feature maps after the 8x downsampled feature map, the 4x downsampled feature map after the 2x downsampled feature map, and the 16x downsampled feature map after the 2x upsampled feature map are fused. Simultaneously, the 4x downsampled feature map is downsampled by 4x, and the 8x downsampled feature map is downsampled by 2x. Then, the feature maps after the 16x downsampled feature map, the 4x downsampled feature map after the 4x downsampled feature map, and the 8x downsampled feature map after the 2x downsampled feature map are fused.

[0104] Next, ReLU activation is applied to the three fused feature maps, and each map proceeds to the third transition stage. In the third transition stage, a 3×3 convolution kernel is used to perform a convolution transformation with a stride of 2 and padding of 1 on the 16-fold downsampled feature map, outputting a downsampled feature map with a scale of 32-fold, and then activated using BN and ReLU.

[0105] Finally, downsampled feature maps with scales of 4x, 8x, 16x, and 32x are output respectively.

[0106] S336. In the fourth stage, parallel multi-scale group convolutions are first performed on the downsampled feature maps at 4x, 8x, 16x, and 32x magnification. Each multi-scale group convolution has the same structure as the multi-scale group convolution in S334. The results of the four parallel multi-scale group convolutions are then fused together element-wise to obtain the fused downsampled feature maps at 4x, 8x, 16x, and 32x magnification, respectively. The multi-scale group convolution processes of these four feature maps do not affect each other.

[0107] Then, multi-scale convolutional fusion is performed on the downsampled feature maps of 4x, 8x, 16x, and 32x after parallel multi-scale group convolutions: the 8x downsampled feature map is upsampled by 2x, the 16x downsampled feature map is upsampled by 4x, and the 32x downsampled feature map is upsampled by 8x. Then, the feature maps obtained by fusing the 4x downsampled feature map, the 8x downsampled feature map after 2x upsampling, the 16x downsampled feature map after 4x upsampling, and the 32x downsampled feature map after 8x upsampling are fused. Simultaneously, the 4x downsampled feature map is downsampled by 2x, the 16x downsampled feature map is upsampled by 2x, the 32x downsampled feature map is upsampled by 4x, and then the feature maps obtained by fusing the 8x downsampled feature map, the 4x downsampled feature map after 2x downsampling, and the 16x downsampled feature map after 2x upsampling are fused. The following steps are performed: 1. Fusing a 32x downsampled feature map with a 4x upsampled feature map; 2. Simultaneously, downsampling the 4x downsampled feature map by 4x, downsampling the 8x downsampled feature map by 2x, and upsampling the 32x downsampled feature map by 2x. Then, fusing the 16x downsampled feature map, the 4x downsampled feature map with a 4x downsampled feature map, the 8x downsampled feature map with a 2x downsampled feature map, and the 32x downsampled feature map with a 2x upsampled feature map; 3. Simultaneously, downsampling the 4x downsampled feature map by 8x, downsampling the 8x downsampled feature map by 4x, and downsampling the 16x downsampled feature map by 2x. Then, fusing the 32x downsampled feature map, the 4x downsampled feature map with a 8x downsampled feature map, the 8x downsampled feature map with a 4x downsampled feature map, and the 16x downsampled feature map with a 2x downsampled feature map.

[0108] Next, ReLU activation is applied to the four fused feature maps respectively. Finally, downsampled feature maps after feature fusion are output at scales of 4x, 8x, 16x, and 32x.

[0109] S337. Three low-resolution feature maps (8x, 16x, and 32x) are scaled using bilinear upsampling to achieve a 4x high-resolution feature map. Specifically, the 8x downsampled feature map is upsampled by 2x, the 16x downsampled feature map by 4x, and the 32x downsampled feature map by 8x. These three upsampled feature maps are then concatenated with the 4x downsampled feature map to blend the feature maps from the four paths. This fusion method achieves a high-resolution representation, allowing for higher resolution processing of mask block boundary information.

[0110] S338. The mixed feature map is convolved with a 1×1 convolution, and its size is changed to the size of the meta-mask block, i.e., a mask block with a side length of a. Finally, the binarized (foreground and background) fine segmentation result is output, which is the optimized mask block set {P”. m}

[0111] S34. Mask the pseudo-instance x p In the middle, the original mask block is replaced with the optimized mask block {P”. m After re-merging the optimized mask blocks, a compact and precise mask is formed, completing the pseudo-instance mask x. p The boundary refinement represents the new pseudo-instance mask as x. n .

[0112] Step 4: For the intermediate feature map X∈R extracted by Feature Pyramid Network in Step 2... H×W×K Here, H, W, and K represent the height, width, and number of channels of the feature map, respectively. Global max pooling is performed on the feature map X to obtain the corresponding peak response map P. This map provides clues to the location of instances but contains considerable noise.

[0113] Step 5: Input the peak response map P into the peak attention module to enhance attention to the peak region and deactivate the remaining noise regions. The peak attention module consists of three parts: a selector, a controller, and a peak stimulator. Figure 4 As shown, its specific workflow is as follows:

[0114] S51. For the input peak response map P, a selector is used to select only one peak region with the maximum peak value for each building in the peak response map P, eliminating the influence of noise and ensuring that each building corresponds to only one peak region. This selector filters out the unique peak region with the maximum peak value from the multiple peak regions corresponding to each building in the peak response map P, and extracts all points within that peak region. Let there be K points, which are denoted as K standard points. The result after selection is represented as S. p ∈R 1×1×K .

[0115] S52. For the input peak response map P, a controller is used to enhance attention, making the peak response map P focus more on the peak region with the largest peak value. This controller extracts K standard points within the peak region from the selector, and applies a control value to each standard point. This value represents the degree to which the peak attention module enhances attention to the peak region, and is denoted as G. p ∈[0,1] 1×1×K In some embodiments, it is set to G p =0.7.

[0116] S53. Among the K standard points in the peak region with the largest peak value selected by the selector, there are still some noise regions. The standard points in these noise regions have low values ​​and do not need to be given excessive attention. Therefore, this invention proposes a peak stimulator to suppress the noise regions. In the peak stimulator, τ is used by element-wise multiplication. p =S p ·G p Set peak region boundaries for the peak region; if the value of the standard point extracted by the selector is higher than τ. p If the value is zero, the point is considered to be in the peak region and will receive special attention; otherwise, it is considered to belong to the noise region, is set to zero, and is disabled.

[0117] After passing through the peak stimulator, in the input peak response map P, each building will retain only one peak region, and within that peak region, only peak values ​​greater than τ will be retained. p The area is clearly defined. Using a peak stimulator effectively eliminates the influence of noisy areas and provides precise location clues for each building instance.

[0118] S54. Finally, generate the peak response map P after enhancing peak attention. s The graph can locate the sparse representative area of ​​each target, and each local maximum point in the graph can provide a precise instance clue for its corresponding building.

[0119] Step 6: Combine the pseudo-instance mask x generated in the pseudo-instance mask refinement module. n Peak response map P generated by the peak attention module s Confirm the pseudo-center image C p With pseudo offset map O p To represent instance-level information within the foreground region, the pseudo-center graph C is ignored. p With pseudo offset map O p The area is considered the background. Its specific workflow is as follows:

[0120] S61, in the peak response diagram P s In the middle, each peak region corresponds to a pseudo-instance mask x. n In an example building, max pooling is performed on the retained peak regions. This operation helps remove overlapping masks. If the values ​​of the peak regions are the same before and after max pooling, then the center information obtained after max pooling is considered a pseudo-center image C. p A portion; if the values ​​in the peak regions are different, then the center information obtained after max pooling of the peak regions is considered not to belong to the pseudo-center graph C. p Part of it, ultimately generating the pseudo-center graph C p .

[0121] S62, Based on the pseudo-center diagram C p The peak regions that are preserved in the original intermediate feature map X are used to extract the feature information of the buildings corresponding to these peak regions. The two-dimensional offset vector information of these buildings relative to the peak regions is then calculated using these features; this offset information is the pseudo-offset map O. p .

[0122] In practice, each peak region corresponds to a building, and each building has only one corresponding peak region. This peak region is extracted from the peak response map P using the peak attention module. The peak response map P is obtained by max pooling the intermediate feature map X. The intermediate feature map X describes the feature information of each building. Each peak region has a corresponding, individual building information in the intermediate feature map X. The "feature information of the building corresponding to the peak region" refers to the feature information of the building corresponding to that peak region in the intermediate feature map X.

[0123] Step 7: Perform overall training within the network architecture involving the mask classification and refinement module. As the number of training rounds increases, based on the training results from Steps 2 to 6 in each round, the network's generalization ability gradually stabilizes, and the pseudo-instance mask x... n It can gradually capture instance-level information that was lost due to overlap being identified as background information. (Pseudo-center image C) p With pseudo offset map O p It was also updated accordingly, and the central diagram C was guided accordingly. m With offset map O m The generation of .

[0124] The input to the mask classification refinement module is the improved pseudo-instance mask x, which is generated as the training process progresses. n Central Figure C m With offset map O m During training, the pseudo-instance mask x n Pseudo-center image C p Pseudo-offset map O p Central Figure C m With offset map O m It's not fixed. These five images will only stop changing once training is complete. As the training process progresses through steps two through six, the pseudo-instance mask x... n Gradually, more accurate instance information can be captured, leading to optimization and improvement, and consequently, the pseudo-center graph C... p With pseudo offset map O p It will also be continuously updated. Based on the update results, the central image C... m With offset map O m It will also be updated. In the first round of training, the central image C m With offset map Om That is, the pseudo-center image C p With pseudo offset map O p However, it cannot make "Central Image C" m With offset map O m "Completely understood as "pseudo-center image C" p With pseudo offset map O p "Because from the start of the second round of training until the end of the training process, "Central Figure C" m With offset map O m It will be updated to the refined center map C generated in the previous training process. r Compared with the refined offset map O r "and "pseudo-center map C" p With pseudo offset map O p "The result of finding the union. That is, as training progresses, the pseudo-instance mask x..." n Pseudo-center image C p Pseudo-offset map O p First it is updated, then the central image C. m With offset map O m Updated, after passing through the mask classification refinement module, the refined center image C r Compared with the refined offset map O r Updated. After these steps, the refined center plot C r Compared with the refined offset map O r Conversely, the pseudo-center graph C p Pseudo-offset map O p Help Center Image C m With offset map O m Update. The pseudo-instance mask x remains unchanged until the training process is complete. n Pseudo-center image C p Pseudo-offset map O p Central Figure C m With offset map O m Updates stopped then.

[0125] like Figure 5 As shown, in the mask classification refinement module, the offset map O is first processed. m Centroid clustering is used to eliminate the influence of noise masks and provide more accurate grouping boundaries for instance groups. The results of centroid clustering are then combined with the pseudo-instance mask x. n Central Figure C m With offset map O m Jointly supervise the completion of the pseudo-instance mask x n Instance grouping in the context of instance grouping. The specific process of instance grouping is as follows: First, remove the pseudo-instance mask x. n The background information is used, retaining only the foreground mask information belonging to buildings. Secondly, the center image C is used...m Identify the center position for each instance. Next, use the offset map O... m The result of central clustering is used for the pseudo-instance mask x. n The boundary information is corrected (the result of the center clustering is compared with the pseudo-instance mask x). n Cosine similarity calculation is performed to avoid mask overlap. Finally, based on the pseudo-instance mask x... n The extracted foreground mask information of the buildings and the corrected boundary information are used to generate an optimized mask, which is then used based on the center image C. m The central location corresponding to each instance is identified to distinguish each instance and achieve instance grouping.

[0126] Based on the results of grouping the instances, the refined center map C can be extracted. r Compared with the refined offset map O r During training, the refined center map C r Compared with the refined offset map O r Further guidance center diagram C m With offset map O m This process generates more accurate instance grouping results. At the end of training, the instance grouping results become the final output mask X. o This mask combines the instance mask x, which is continuously refined during training. m Instance-level information and central diagram C m Instance location information and offset map O m The classification information effectively eliminates the mask overlap problem caused by insufficient image-level annotation information. Furthermore, through the training process, a building extraction model for airborne oblique remote sensing images based on bounding box annotations and image-level annotations was obtained.

[0127] Step 8: Use the trained airborne tilted remote sensing image building extraction model based on bounding box annotation and image-level annotation to test the images in the test set, and obtain the individual building regions extracted from the test set.

[0128] This invention addresses the problem that instance segmentation methods based on pixel-level annotation rely on large amounts of data annotation. It proposes a method for extracting buildings from airborne oblique remote sensing images based on bounding box and image-level annotations. By combining bounding box annotation with image-level annotation, building extraction from airborne oblique remote sensing images can be achieved simply by creating tight bounding box annotations for buildings and adding image-level annotations indicating that the instance category is "building" in the image. To address the issues of unclear mask boundaries and excessive unnecessary background pixels caused by insufficient bounding box annotation information, a pseudo-instance mask refinement module is proposed. To address the mask overlap problem caused by insufficient image-level annotation information, a peak attention module and a mask classification refinement module are proposed to optimize the prediction mask.

[0129] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotation, characterized in that, An airborne oblique remote sensing image building extraction model based on bounding box annotation and image-level annotation was used to extract individual building regions from the remote sensing image to be processed. The aforementioned building extraction model for airborne oblique remote sensing images based on bounding box annotation and image-level annotation is obtained through the following steps: Step 1: Crop the original large remote sensing image and the semantic map obtained from the annotation to obtain a small remote sensing image; Only remote sensing thumbnails containing buildings are labeled as "buildings" as image-level annotation information. The smallest bounding rectangle of the building in the cropped image is taken as the ground truth information of its corresponding bounding box. The remote sensing thumbnails are directly recorded as remote sensing images. The original dataset is constructed based on the cropped remote sensing images, image-level annotation information and bounding box ground truth information. The training set is determined based on the original dataset. Step 2: Transfer the remote sensing images X from the training set i With bounding box truth information G box The pseudo-instance mask x is generated in the weakly supervised instance segmentation network. p ; Step 3: Mask the pseudo-instance x p In the input pseudo-instance mask refinement module, redundant boundary information is filtered out, and missing boundary information is supplemented and optimized to achieve pseudo-instance mask x. p Boundary refinement; the processing flow of the pseudo-instance mask refinement module is as follows: S31, using a pseudo-instance mask x p Centered on the boundary pixels, a sliding window strategy is used to densely apply the pseudo-instance mask x. p With remote sensing image X i Several square mask blocks and image blocks of the same size with side length 'a' are extracted from the image. The set of mask blocks is then represented as {P}. m The set of image patches is represented as {P}. i }; S32. Using nonmaximum suppression, for the mask block set {P} m The mask blocks containing redundant boundary information in {P} are suppressed, and the set of mask blocks after filtering out redundant boundary information is represented as {P'}. m The set of image blocks corresponding to the preserved mask blocks is represented as {P'}. i }; S33, the set of mask blocks {P' after filtering out redundant boundary information m The set of image patches {P'} corresponding to the preserved mask patches i The input is refined into the network, where boundary information of the mask block is optimized. S331. Convert the number of input channels to 4, where the number of channels for the mask block is 1 and the number of channels for the image block is 3, with each of the RGB values ​​of the image block occupying one channel; the mask block provides positional and semantic information. S332. Perform convolutional downsampling on the feature map, and then send it to the first stage of processing; S333. In the first stage, four parallel multi-scale group convolutions are performed on the input feature map. The results of the four parallel multi-scale group convolutions are fused together in a concatenated manner, and the result is input into the first transition stage. Then, the first transition stage is processed as follows: the input feature map is transformed by convolution to output a downsampled feature map with a scale of 4 times; at the same time, the input feature map is transformed by convolution to output a downsampled feature map with a scale of 8 times; finally, downsampled feature maps with scales of 4 times and 8 times are output respectively; and then sent to the second stage for processing. S334. In the second stage, firstly, four parallel multi-scale group convolutions are performed on the 4x and 8x downsampled feature maps respectively; then, the results of the four parallel multi-scale group convolutions are fused together by adding them element by element to obtain the fused 4x and 8x downsampled feature maps respectively. Then, multi-scale convolutional fusion is performed on the 4x and 8x downsampled feature maps after parallel multi-scale group convolutions respectively; then, ReLU activation is performed on the two fused feature maps respectively, and they enter the second transition stage respectively; in the second transition stage, convolution transformation is used on the 8x downsampled feature map to output a downsampled feature map with a scale of 16x. Finally, downsampled feature maps with scales of 4x, 8x, and 16x are output respectively; then they are sent to the third stage of processing. S335. In the third stage, firstly, four parallel multi-scale group convolutions are performed on the 4x, 8x, and 16x downsampled feature maps respectively. The results of the four parallel multi-scale group convolutions are then fused together by adding them element by element to obtain the fused 4x, 8x, and 16x downsampled feature maps respectively. Then, multi-scale convolutional fusion is performed on the 4x, 8x, and 16x downsampled feature maps after parallel multi-scale group convolutions respectively; then, ReLU activation is performed on the three fused feature maps respectively, and they enter the third transition stage respectively; in the third transition stage, convolution transformation is used on the 16x downsampled feature map to output a downsampled feature map with a scale of 32x. Finally, downsampled feature maps with scales of 4x, 8x, 16x, and 32x are output respectively; then they are sent to the fourth stage of processing. S336. In the fourth stage, firstly, parallel multi-scale group convolutions are performed on the downsampled feature maps of 4x, 8x, 16x, and 32x respectively. Each multi-scale group convolution has the same structure as the multi-scale group convolution in S334. The results of the four parallel multi-scale group convolutions are fused together by adding them element by element to obtain the fused downsampled feature maps of 4x, 8x, 16x, and 32x respectively. Then, multi-scale convolutional fusion is performed on the downsampled feature maps that have undergone parallel multi-scale group convolutions at 4x, 8x, 16x, and 32x respectively; then, ReLU activation is performed on the four fused feature maps respectively; finally, downsampled feature maps with scales of 4x, 8x, 16x, and 32x are output after feature fusion. S337. The three low-resolution feature maps of 8x, 16x and 32x are scaled up by bilinear upsampling and uniformly scaled to a high-resolution feature map of 4x and then fused. S338. The mixed feature map is convolved with a 1×1 convolution, and its size is changed to a mask block with side length a. Finally, the binarized fine segmentation result is output, which is the optimized mask block set {P''}. m }; S34. Mask the pseudo-instance x p In the original mask block, the optimized mask block {P'' is replaced. m After re-merging the optimized mask blocks, a compact and precise mask is formed, completing the pseudo-instance mask x. p The boundary refinement represents the new pseudo-instance mask as x. n ; Step 4: Extract intermediate feature maps from the feature extraction process in the weakly supervised instance segmentation network of Step 2. , where H, W, and K are the height, width, and number of channels of the feature map, respectively; global max pooling is performed on the feature map X to obtain the peak response map P corresponding to the feature map; Step 5: Input the peak response map P into the peak attention module for processing, and generate the peak response map P. s ; Step 6: Combine the pseudo-instance mask x generated in the pseudo-instance mask refinement module. n Peak response map P generated by the peak attention module s Confirm the pseudo-center image C p With pseudo offset map O p To represent instance-level information within the foreground region, the pseudo-center graph C is ignored. p With pseudo offset map O p The area considered as background in the middle; Step 7: Then, the image is sent to the mask classification and refinement module for processing. In the mask classification and refinement module, the offset image O is first processed. m Perform center clustering; then combine the results of center clustering with the pseudo-instance mask x. n Central Figure C m With offset map O m Jointly supervise the completion of the pseudo-instance mask x n The instances are grouped; based on the results of the instance grouping, the refined center map C can be extracted. r Compared with the refined offset map O r ; Based on the above steps, iterative training is performed using the training set to obtain a trained building extraction model for airborne oblique remote sensing images based on bounding box annotation and image-level annotation.

2. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to claim 1, characterized in that, X the remote sensing images in the training set i With bounding box truth information G box The pseudo-instance mask x is generated in the weakly supervised instance segmentation network. p The process includes the following steps: S21. Utilize Feature Pyramid Network (FPN) to process the input remote sensing image X. i Feature extraction of buildings is performed, and the extracted features are rated as F1. S22. For the extracted feature F1, use the bounding box ground truth information G. box Constraints are applied by removing feature information outside the truth box in F1 and retaining only the feature information within the truth box, thus obtaining the input remote sensing image X. i Feature map F2 within the bounding box; S23, From the bounding box truth information G box In the corresponding bounding box feature map F2, the feature information of each pixel i is obtained, and the color difference between each pixel i and its eight neighboring pixels j is calculated. ; S24. From different bounding box truth information G box Internally, pixel information is obtained to determine the spatial correspondence between pixels. i and k are two different truth information r of bounding boxes. n r s pixels within; S25, Difference in color between neighboring pixels and spatial correspondence As weighting information, it is embedded into the remote sensing image X. i On the feature map F2 within the bounding box, the pseudo-instance mask is obtained. .

3. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to claim 2, characterized in that, The color difference ; where f i with f j These are the ROI features of pixel i and its eight neighboring pixels j in the bounding box feature map F2, respectively. , It is the magnitude of the ROI feature of pixel i and its eight neighboring pixels j in the feature map F2 within the bounding box.

4. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to claim 3, characterized in that, The spatial correspondence between pixels Where j represents the truth information of the bounding box r n The pixels within the eight neighborhoods of i, where l is the ground truth information of the bounding box r. s pixels within the eight neighborhoods of inner k; off i,k For r n Medium pixel i and r s The relative spatial offset between pixels k, off j,l For r n Medium pixel j and r s The relative spatial offset between pixels l, where T(j, l) is the bounding box ground truth information r. n r s The soft assignment matrix is ​​used to determine the matching score between the truth information of the two bounding boxes.

5. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to any one of claims 1 to 4, characterized in that, In the first stage of S333, during the process of performing four parallel multi-scale group convolutions, each multi-scale group convolution first uses 1×1 convolution to reduce the dimensionality of the feature map, that is, to reduce the number of channels of the input feature map, then uses 3×3 convolution to complete feature extraction, and finally uses 1×1 convolution to increase the dimensionality of the feature map to restore the number of channels of the input feature map. In S334, during the second stage of performing four parallel multi-scale group convolutions on the 4x and 8x downsampled feature maps respectively, each multi-scale group convolution first uses two 3×3 convolutions to extract input features. Then, the features that have not undergone multiple scale group convolutions are directly added to the outputs of two 3×3 convolutions to obtain the final output feature map in each multi-scale group convolution. Each of the four multi-scale group convolutions in the third stage of S335 has the same multi-scale group convolution structure as S334. Each of the four multi-scale group convolutions in the fourth stage of S336 has the same multi-scale group convolution structure as that in S334.

6. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to claim 5, characterized in that, In S334, the process of multi-scale convolutional fusion of the 4x and 8x downsampled feature maps after parallel multi-scale group convolution is as follows: The 8x downsampled feature map is upsampled by 2x and then fused with the 4x downsampled feature map. At the same time, the 4x downsampled feature map is downsampled by 2x and then fused with the 8x downsampled feature map.

7. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to claim 6, characterized in that, The process of multi-scale convolutional fusion of downsampled feature maps (4x, 8x, and 16x) after parallel multi-scale group convolutions in S335 includes the following steps: The 8x downsampled feature map is upsampled by 2x, and the 16x downsampled feature map is upsampled by 4x. Then, the feature maps obtained by fusing the 4x downsampled feature map, the 8x downsampled feature map after 2x upsampling, and the 16x downsampled feature map after 4x upsampling are fused together.

8. The method for extracting buildings from airborne oblique remote sensing images based on box and image-level annotation according to claim 7, characterized in that, The process of multi-scale convolutional fusion of downsampled feature maps (4x, 8x, 16x, and 32x) after parallel multi-scale group convolutions in S336 includes the following steps: The following steps are repeated: 1) Upsample the 8x downsampled feature map by 2x, 1) Upsample the 16x downsampled feature map by 4x, and 2) Upsample the 32x downsampled feature map by 8x. Then, these feature maps are fused together. Simultaneously, the following steps are repeated: 1) Downsample the 4x downsampled feature map by 2x, 1) Upsample the 16x downsampled feature map by 4x, and 2) Upsample the 32x downsampled feature map by 8x. The 32x downsampled feature map is upsampled by 2x, and then the following feature maps are fused: the 16x downsampled feature map, the 4x downsampled feature map (after being downsampled by 4x), the 8x downsampled feature map (after being downsampled by 2x), and the 32x downsampled feature map (after being upsampled by 2x). Simultaneously, the 4x downsampled feature map is downsampled by 8x, the 8x downsampled feature map is downsampled by 4x, and the 16x downsampled feature map is downsampled by 2x. Finally, the following feature maps are fused: the 32x downsampled feature map, the 4x downsampled feature map (after being downsampled by 8x), the 8x downsampled feature map (after being downsampled by 4x), and the 16x downsampled feature map (after being downsampled by 2x).

9. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to claim 8, characterized in that, Step five, which involves inputting the peak response map P into the peak attention module for processing, includes the following steps: S51. For the input peak response map P, a selector is used to select the unique peak region with the largest peak value among the multiple peak regions corresponding to each building in the peak response map P. All points within this peak region are then extracted. Let there be K points, which are denoted as K standard points. The result after selection is represented as... ; S52. For the input peak response map P, a controller is used to enhance attention. The controller extracts K standard points within the peak region from the selector, and applies a control value to each standard point. This value represents the degree to which the peak attention module enhances attention to the peak region. The control value is expressed as... ; S53. Then, peak stimulator processing is used, in which element-wise multiplication is employed. Set peak region boundaries for the peak region; if the value of the standard point extracted by the selector is higher than... If the value is zero, the point is considered to be in the peak region and will receive special attention; otherwise, it is considered to belong to the noise region, is set to zero, and is disabled. S54. Finally, generate the peak response map P after enhancing peak attention. s .

10. The method for extracting buildings from airborne oblique remote sensing images based on bounding boxes and image-level annotations according to claim 9, characterized in that, The specific process of step six includes the following steps: S61, in the peak response diagram P s In the middle, each peak region corresponds to a pseudo-instance mask x. n In an example building, max pooling is performed on the retained peak regions. This operation helps remove overlapping masks. If the values ​​of the peak regions are the same before and after max pooling, then the center information obtained after max pooling is considered a pseudo-center image C. p A portion; if the values ​​in the peak regions are different, then the center information obtained after max pooling of the peak regions is considered not to belong to the pseudo-center graph C. p Part of it, ultimately generating the pseudo-center graph C p ; S62, Based on the pseudo-center diagram C p The peak regions that are preserved in the original intermediate feature map X are used to extract the feature information of the buildings corresponding to these peak regions. The two-dimensional offset vector information of these buildings relative to the peak regions is then calculated using these features; this offset information is the pseudo-offset map O. p .

Citation Information

Patent Citations

  • High-resolution image weak supervision building extraction method combining pixel semantic association and boundary attention

    CN112052783A

  • Building and sheltered area monomer extraction method based on virtual knowledge migration

    CN113487738A