A building contour extraction method based on attention mechanism and dynamic vertex

Abstract

The present application relates to a kind of building contour extraction method based on attention mechanism and dynamic vertex, belong to image processing technical field, solve the non-closed building contour obtained in prior art, building contour prediction precision is low, further lead to the problem of low building extraction precision index.The present application proposes building vertex sequence prediction branch, and adds the branch in existing building detection network, improves the precision of building vertex extraction, the building contour precision of using the model prediction in building vertex extraction task is high, and no extra building vertex and edge are predicted, and building extraction precision AP index is high.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method for extracting building contours based on attention mechanisms and dynamic vertices. Background Technology

[0002] Existing methods for building vertex extraction typically fall into two categories: post-processing-based methods and end-to-end building footprint extraction methods. Post-processing-based methods require segmenting the building using a network to obtain a building segmentation mask, and then using post-processing to fit the building's contour to obtain vectorized information about the building's outline.

[0003] End-to-end building vertex extraction methods are divided into two-stage and single-stage methods. The two-stage building vertex extraction method uses a detection or segmentation network to extract building regions, and then uses a vertex extraction network to extract vertex sequences, outputting vectorized information of building outlines. The single-stage building vertex extraction method directly extracts building vertices from remote sensing images, then predicts the connection matrix between vertices, and finally obtains the vectorized information of buildings based on the predicted connection matrix.

[0004] Post-processing-based building vertex extraction methods divide the task into two sub-tasks: building extraction and contour fitting. The output of the building extraction sub-task affects the fitting effect of the subsequent contour fitting sub-task. Moreover, the contour fitting sub-task usually relies on cumbersome polygon constraints to fit polygon contours, resulting in a complex process and long extraction time. Therefore, the building contour accuracy obtained by this type of method is not high. Although single-stage building vertex extraction methods can obtain vectorized information of building contours end-to-end, due to the special acquisition methods of remote sensing images and the complexity of the scenes, vertex omissions are prone to occur, leading to connection matrix prediction failures. For complex building contours, self-intersecting or non-closed connection matrices are easily predicted. Although two-stage building vertex extraction methods have higher accuracy than the previous two types of methods, they are limited by sequence models, resulting in low accuracy in predicting the contours of complex buildings, further leading to low accuracy of the building vertex extraction model. Summary of the Invention

[0005] In view of the above problems, the present invention provides a building contour extraction method based on attention mechanism and dynamic vertices, which solves the problems of non-closed building contours, low building contour prediction accuracy, and low building extraction accuracy in the prior art.

[0006] This invention provides a method for extracting building contours based on an attention mechanism and dynamic vertices, comprising:

[0007] S1. Input the image to be detected into a deep convolutional neural network, extract the multi-scale features of the image, and fuse the multi-scale features of the image through a feature pyramid network in a top-down and lateral connection manner to obtain a feature pyramid structure.

[0008] S2. Using the feature pyramid structure as input to the region proposal network, generate regions of interest for buildings;

[0009] S3. Using the feature pyramid structure and the region of interest of the building as input to the building outline prediction network, the building is classified, the building bounding box regression is predicted, and the building vertex sequence is predicted to obtain the building category label, bounding box, and vertex sequence.

[0010] S4. The building outlines in the image are constructed using the building category labels, bounding boxes, and vertex sequences.

[0011] Preferably, in step S1, the image to be detected is input into a deep convolutional neural network to extract multi-scale features of the image. The multi-scale features of the image are then fused through a pyramid network in a top-down and lateral connection manner to obtain a feature pyramid structure. The specific steps are as follows:

[0012] The ResNet-50 network structure is used as the backbone network of the deep convolutional neural network; the backbone network is a multi-scale 5-layer convolutional network.

[0013] After the image to be detected passes through the backbone network, based on the different feature map sizes of different layers in the backbone network, five levels of feature maps from low to high are output according to different dimensions, defined as follows:

[0014] C = {C1, C2, C3, C4, C5}

[0015] in, C a Let H represent the feature map of layer a, where a = 1, 2, 3…A, representing A layers, R is a real number, and b is the number of feature map channels, where b = 64, 256, 512, 1024, 2048, H. a H represents the height of the feature map at layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32; W a W is the width of the feature map of layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32;

[0016] The feature map C = {C2, C3, C4, C5} output by the backbone network is used as the input to the feature pyramid network. This input is then convolved through a 1×1 convolutional layer of the feature pyramid network to generate a convolutional feature map L = {L2, L3, L4, L5}, where...

[0017] H a H represents the height of the feature map at layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32; W a W is the width of the feature map of layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32;

[0018] The convolutional feature maps L = {L2, L3, L4, L5} are fused together by top-down and lateral connections. This includes: convolving feature map L5 through a 3×3 convolutional layer to generate a pyramid network feature map P5; then, using max pooling, generating a pyramid network feature map P6; upsampling feature map L5 by a factor of 2 using the nearest neighbor method, adding the upsampled result to feature map L4, and convolving it through a 3×3 convolutional layer to generate a pyramid network feature map P4; upsampling feature map L4 by a factor of 2 using the nearest neighbor method, adding the upsampled result to feature map L3, and convolving it through a 3×3 convolutional layer to obtain a pyramid network feature map P3; upsampling feature map L3 by a factor of 2 using the nearest neighbor method, adding the upsampled result to feature map L2, and convolving it through a 3×3 convolutional layer to obtain a pyramid network feature map P2; finally, a multi-scale feature pyramid structure is obtained, defined as follows.

[0019] P = {P2, P3, P4, P5, P6}

[0020] in, a = 2, 3, 4, 5, 6, H a H represents the height of the feature map at layer a. a =1 / 2, 1 / 4,

[0021] 1 / 8, 1 / 16, 1 / 32; W a W is the width of the feature map of layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32;

[0022] Furthermore, the top-down connection is used to transfer high-level semantic information from the top-level feature map to the high-resolution bottom-level feature map.

[0023] Preferably, in step S2, the feature pyramid structure is used as input to the region proposal network to generate regions of interest for buildings. The specific steps are as follows:

[0024] Multiple original anchor boxes are set on each pixel of each feature map in the feature pyramid structure; the feature maps of the feature pyramid structure are {P2,P3,P4,P5,P6}, where P2 is the second layer feature map, P3 is the third layer feature map, P4 is the fourth layer feature map, P5 is the fifth layer feature map, and P6 is the sixth layer feature map.

[0025] The category confidence and bounding box prediction value of each original anchor box are obtained using the region proposal network;

[0026] Each original anchor frame with a category confidence score is corrected based on its corresponding boundary prediction value to obtain a corrected anchor frame. This corrected anchor frame is then represented as a building region of interest with a category confidence score. Furthermore, the boundary prediction value is the offset between the actual position and the current position of each anchor frame.

[0027] The region of interest for a building is a quadrilateral with class confidence, obtained by adding the feature map anchor box and the boundary prediction value.

[0028] The set multiple anchor frames have one size and multiple aspect ratios; the number of anchor points is equal to the number of aspect ratio types;

[0029] Furthermore, the region proposal network is an algorithm that automatically extracts key regions in an image. It takes an image (of any size) as input and outputs a set of rectangular target proposal boxes, each box having a score.

[0030] The region of interest for a building is a selected area in the image to be analyzed. This area is the focus of image analysis. The region proposal network is used to delineate this area to facilitate further processing. The region of interest for a building is the area in the image where buildings are located. Extracting these areas will greatly improve the efficiency and accuracy of image processing and analysis.

[0031] Furthermore, the region proposal network takes the feature map {P2,P3,P4,P5,P6} of the feature pyramid structure as input;

[0032] Three anchor boxes are set at each pixel location of the feature map at each scale. The feature map is first passed through a 3×3 convolution to generate a feature map with 256 channels. The confidence scores for the d-th class are then obtained by performing two different 1×1 convolutions. and the predicted value of the e-th bounding box

[0033]

[0034] Where 3 represents the class confidence of the three anchor boxes for each pixel, 12 represents the four coordinates of the bounding box corresponding to the three anchor boxes for each pixel, the four coordinates represent the offset of the anchor point coordinates in the coordinates of the image to be detected, d = 1, 2, 3...D, representing D class confidences; e = 1, 2, 3...E, representing E bounding box prediction values;

[0035] Traverse the feature maps of the five scales in the feature pyramid structure to obtain the class confidence and bounding box prediction values ​​corresponding to multiple original anchor boxes in the feature pyramid structure.

[0036] Each original anchor frame is corrected based on its corresponding bounding box prediction value to obtain a corrected anchor frame, which is then represented as a building region of interest; the building region of interest is a building region of interest with a category confidence prediction value.

[0037] Preferably, in step S3, the feature pyramid structure and the building region of interest are used as input to the building outline prediction network to classify the building, predict building bounding box regression, and predict building vertex sequence, thereby obtaining the building's category label, bounding box, and vertex sequence; the specific steps include:

[0038] Based on the threshold of class confidence, class confidence is determined for each region of interest of a building with class confidence.

[0039] Filter out the regions of interest for buildings with low category confidence scores, and what remains are the regions of interest for buildings with high category confidence scores;

[0040] Collect building regions of interest with high confidence scores across multiple categories to construct a set of building regions of interest;

[0041] The set of regions of interest and the feature pyramid structure of buildings are input into the building classification branch, the bounding box regression branch and the building vertex sequence prediction branch of the building outline prediction network, respectively, and the output includes the category labels, bounding boxes and vertex sequences of multiple buildings.

[0042] Furthermore, the threshold value for the category confidence score is between 0.5 and 1. When the category confidence score is less than 0.5, it is considered a low category confidence score.

[0043] Furthermore, the specific steps for outputting the category labels of buildings are as follows: input the feature maps {P2,P3,P4,P5} in the feature pyramid structure and the set of building regions of interest into the building classification branch of the building outline prediction network;

[0044] Based on the size of each building region of interest in the set of building regions of interest, each building region of interest is mapped to the features of {P2, P3, P4, P5} in the feature pyramid structure to obtain multiple building region of interest feature maps.

[0045] The ROI-Align algorithm is used to perform fully connected and softmax activation function calculations on the feature map of each building's region of interest to obtain the category label of the region feature map. The category label of the region feature map is used to represent the category label of the building. The category label of the region feature map includes the building target and the background, with 1 representing the building target and 0 representing the background.

[0046] Furthermore, the ROI-Align algorithm is used to extract 7×7 pixels of the region feature map from the feature map of each building's region of interest.

[0047] Preferably, step S3, outputting the bounding boxes of multiple buildings, specifically includes the following steps:

[0048] The feature maps {P2, P3, P4, P5} in the feature pyramid structure and the set of building regions of interest are input into the bounding box regression branch of the building outline prediction network.

[0049] Based on the size of each building region of interest in the set of building regions of interest, each building region of interest is mapped to the features of {P2, P3, P4, P5} in the feature pyramid structure to obtain multiple building region of interest feature maps.

[0050] The ROI-Align algorithm is used to compute fully connected layers in the feature map of each building's region of interest, outputting the bounding box (x) of each region's feature map. j ,y j ,w j ,h j ), where x j The x-axis coordinates and y-axis coordinates of the center point of the bounding box of the j-th region feature map are represented by the following values. j w represents the y-axis coordinate of the center point of the bounding box of the j-th region feature map; j h represents the width of the bounding box of the feature map of the j-th region. j Let J represent the height of the bounding box of the j-th region feature map, where j = 1, 2, 3...J, and represents the J-th region feature map.

[0051] Furthermore, the ROI-Align algorithm is used to extract 7×7 pixels of the region feature map from the feature map of each building's region of interest.

[0052] Preferably, step S3, which involves outputting a vertex sequence of multiple buildings, specifically includes the following steps:

[0053] The feature map and the region of interest of the building in the feature pyramid structure are input into the building vertex sequence prediction branch of the building outline prediction network;

[0054] The feature maps of the feature pyramid structure are subjected to feature fusion processing. The intermediate feature map of the feature pyramid structure is used as the base feature map. Feature maps larger than the base feature map are adjusted to the same size as the base feature map through pooling operation. Feature maps smaller than the base feature map are adjusted to the same size as the base feature map through linear interpolation. The processed feature maps of the same size are added together and averaged to obtain the fused feature map.

[0055] By mapping the regions of interest of buildings in the set of building regions of interest to the fused feature map, multiple feature maps of building regions of interest are obtained.

[0056] The feature maps of multiple building regions of interest are flattened into two-dimensional sequence feature maps in the spatial dimension, and then the two-dimensional positional encoding of the two-dimensional sequence feature maps is obtained by using sine functions of different frequencies.

[0057] The feature vectors of multiple region feature maps are obtained by adding the value of each two-dimensional sequence feature and the corresponding two-dimensional position encoding.

[0058] The feature vectors of multiple region feature maps are input into the encoding block, and after point-by-point, row-by-row and column-by-column self-attention processing, a forward fully connected layer is used to obtain multiple encoding layer building feature maps with rich semantics and geometric structure.

[0059] Define a learnable query set, input the learnable query set into the decoding block and process it through a self-attention mechanism to obtain an updated learnable query set;

[0060] Multiple building feature maps from the coding layers are input into the decoding block, and cross-attention processing is performed between the updated learnable query set and each building feature map from the coding layer. Then, the result is passed through a forward fully connected layer to obtain the dynamic vertices of multiple buildings.

[0061] Based on the updated learnable query set, the dynamic vertices of multiple buildings are updated iteratively. After passing through the four layers of decoders in the decoding block, the iteration stops and the final dynamic vertices of multiple buildings are output.

[0062] The final dynamic vertices of the building are processed by a fully connected layer and an activation function to obtain multiple predicted building vertex sequences;

[0063] The final dynamic vertices of multiple buildings are processed through a fully connected layer and a softmax activation function to obtain multiple predicted building vertex sequences.

[0064] Furthermore, the feature maps in the feature pyramid structure are {P2, P3, P4, P5}. The feature maps {P2, P3, P4, P5} of the feature pyramid structure are subjected to feature fusion processing. Based on feature map P3, feature map P2 is adjusted to the same size as P3 through pooling operations, and feature maps P4 and P5 are adjusted to the same size as P3 through linear interpolation. The fused feature map is obtained by adding and averaging the feature maps of all processed sizes. Furthermore, the ROI-Align algorithm is used to extract the region feature map of the building region of interest for each building region of interest with a pixel size of 20×20.

[0065] The learnable query set Z∈R 20×256 This is a 20×256 vector, used to query vertex information from the feature map output by the encoder. The technical solution of this invention obtains the dynamic vertices of the building, updating points based on other points in each iteration, fully utilizing the information from previous and subsequent points to improve accuracy.

[0066] Preferably, the building outline prediction network in step S3 is obtained by training the building extraction dataset;

[0067] The building outline prediction network described in step S3 is trained using a joint loss function; the building extraction dataset is the Crowdai dataset.

[0068] The expression for the joint loss function is:

[0069] L=λ cls L cls +λ reg L reg +λ verseq L verseq

[0070] Where L is the joint loss function, L cls L represents the classification loss function. reg L represents the bounding box regression loss function. verseq Let λ represent the vertex sequence loss function. cls λ represents the weights of the classification loss function. reg λ represents the weights of the bounding box regression loss function. verseq This represents the weights of the vertex sequence loss function.

[0071] Furthermore, the classification loss function is expressed as follows:

[0072]

[0073] Among them, L cls For the classification loss value, y iLet y be the predicted category label for the i-th building, i = 1, 2, 3…N, where N represents the total number of buildings. i = 0 or 1, 0 represents the background, 1 represents the building; p(y i The classification branch module outputs the probability of the predicted category label for the i-th building.

[0074] Furthermore, the bounding box regression loss function L reg The formula is:

[0075]

[0076] Among them, L reg The bounding box regression loss value, For the bounding box prediction of the i-th building, t i Let be the truth value of the bounding box of the i-th building, where i = 1, 2, 3...N, and N represents the total number of buildings; Let these be the coordinates of the center point of the true bounding box of the i-th building. Let be the width and height of the true bounding box of the i-th building; Let the coordinates be the center point coordinates of the i-th building boundary prediction box. Let be the width and height of the predicted bounding box for the i-th building.

[0077] Furthermore, the vertex sequence loss function L verseq It is a bidirectional loop loss, specifically including:

[0078] Obtain the ground truth value O of the vertex sequence of manually labeled buildings and the first vertex o0 of the ground truth value O of the vertex sequence of buildings, and the predicted value of the vertex sequence of buildings. Find the vertex corresponding to the first vertex o0. Q represents the total number of predicted vertex sequence values ​​for the building;

[0079] With the corresponding vertex Based on the moving vertex sequence prediction value Other vertices in the equation make the corresponding vertex Starting from this point, we obtain new vertex sequence predictions.

[0080] Fixed new vertex sequence prediction values Starting from the given point, reset the other vertices in the new vertex sequence prediction values ​​in a counter-clockwise order to obtain the reverse order of the new vertex sequence prediction values.

[0081] Calculate the true value O of the building vertex sequence and the predicted value of the new vertex sequence, respectively. The reverse order of the new vertex sequence prediction values The loss between them is used to obtain the loss L of the new vertex clockwise. ce And the loss of the new vertex counterclockwise

[0082] Loss L of taking the new vertex clockwise ce And the loss of the new vertex counterclockwise The minimum value is used as the loss L for vertex sequence prediction. verseq ;

[0083] The loss L of the new vertex clockwise ce And the loss of the new vertex counterclockwise The expressions are as follows:

[0084]

[0085]

[0086] Among them, L ce The loss is calculated clockwise from the new vertices, where m = 1, 2, 3…M, and M is the number of samples for the new vertices. For a new vertex function in a clockwise direction, if the m-th new vertex sample corresponds to the v-th class label in a clockwise direction, then... Otherwise, it equals 0, v = 1, 2, 3...V, V clockwise building category labels. Let m be the probability that the m-th new vertex sample belongs to the v-th class label in the clockwise direction;

[0087] The loss is due to the counter-clockwise rotation of the new vertex. Let m be the new vertex function in the counter-clockwise direction. If the m-th new vertex sample corresponds to the invv-th category label in the counter-clockwise direction, then take... Otherwise, it equals 0, and invv = 1, 2, 3, ... INNV building category labels in a counter-clockwise direction. This represents the probability that the m-th new vertex sample belongs to the invv-th category label in the counterclockwise direction;

[0088] Loss L of taking the new vertex clockwise ce And the loss of the new vertex counterclockwise The minimum value is used as the loss L for vertex sequence prediction. verseq .

[0089] The loss L for predicting the vertex sequence verseq The expression is:

[0090]

[0091] Among them, L verseq L is the loss for predicting the vertex sequence.ce The loss is clockwise for the new vertex. The loss is counterclockwise for the new vertex.

[0092] Preferably, step S4, which involves obtaining the building outline in the image using the building's category label, bounding box, and vertex sequence, specifically includes the following steps:

[0093] By locating buildings in an image using bounding boxes, and then using the bounding boxes to supervise the training of a building contour prediction network, the building contour prediction network can accurately locate the building regions in the image.

[0094] The building category label is used to determine whether the building in the bounding box-located image is a building area;

[0095] If it is a building area, obtain the vertex sequence prediction value of the building area, and represent the vertex sequence prediction value as the building outline.

[0096] Compared with the prior art, the present invention has at least the following beneficial effects:

[0097] This invention proposes a building vertex sequence prediction branch and adds it to the existing building detection network, which improves the accuracy of building vertex extraction. In the building vertex extraction task, the building outline predicted by this model has high accuracy and will not predict redundant building vertices and edges, resulting in a high building extraction accuracy (AP) index. Attached Figure Description

[0098] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0099] Figure 1 This is a schematic diagram of the framework of the remote sensing image building extraction model based on attention and dynamic vertices of the present invention;

[0100] Figure 2 This is a schematic diagram of the building vertex sequence prediction branch structure of the present invention;

[0101] Figure 3 This is a schematic diagram of the self-attention-based feature encoding structure of the present invention;

[0102] Figure 4 This is a flowchart of the bidirectional loop loss calculation of the present invention;

[0103] Figure 5 This is a schematic diagram of the building vertex prediction results of the present invention. Detailed Implementation

[0104] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0105] A specific embodiment of the present invention, such as Figure 1-5 This invention discloses a method for extracting building contours based on an attention mechanism and dynamic vertices. To illustrate the effectiveness of the proposed method, a specific embodiment is provided below for detailed explanation of the above technical solution. The specific implementation steps are as follows:

[0106] S1. Input the image to be detected into a deep convolutional neural network, extract the multi-scale features of the image, and fuse the multi-scale features of the image through a feature pyramid network in a top-down and lateral connection manner to obtain a feature pyramid structure.

[0107] S2. Using the feature pyramid structure as input to the region proposal network, generate regions of interest for buildings;

[0108] S3. Using the feature pyramid structure and the region of interest of the building as input to the building outline prediction network, the building is classified, the building bounding box regression is predicted, and the building vertex sequence is predicted to obtain the building category label, bounding box, and vertex sequence.

[0109] S4. The building outlines in the image are constructed using the building category labels, bounding boxes, and vertex sequences.

[0110] Preferably, in step S1, the image to be detected is input into a deep convolutional neural network to extract multi-scale features of the image. The multi-scale features of the image are then fused through a pyramid network in a top-down and lateral connection manner to obtain a feature pyramid structure. The specific steps are as follows:

[0111] The ResNet-50 network structure is used as the backbone network of the deep convolutional neural network; the backbone network is a multi-scale 5-layer convolutional network.

[0112] After the image to be detected passes through the backbone network, based on the different feature map sizes of different layers in the backbone network, five levels of feature maps from low to high are output according to different dimensions, defined as follows:

[0113] C = {C1, C2, C3, C4, C5}

[0114] in, Ca Let H represent the feature map of layer a, where a = 1, 2, 3…A, representing A layers, R is a real number, and b is the number of feature map channels, where b = 64, 256, 512, 1024, 2048, H. a H represents the height of the feature map at layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32; W a W is the width of the feature map of layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32;

[0115] The feature map C = {C2, C3, C4, C5} output by the backbone network is used as the input to the feature pyramid network. This input is then convolved through a 1×1 convolutional layer of the feature pyramid network to generate a convolutional feature map L = {L2, L3, L4, L5}, where...

[0116] H a H represents the height of the feature map at layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32; W a W is the width of the feature map of layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32;

[0117] The convolutional feature maps L = {L2, L3, L4, L5} are fused together by top-down and lateral connections. This includes: convolving feature map L5 through a 3×3 convolutional layer to generate a pyramid network feature map P5; then, using max pooling, generating a pyramid network feature map P6; upsampling feature map L5 by a factor of 2 using the nearest neighbor method, adding the upsampled result to feature map L4, and convolving it through a 3×3 convolutional layer to generate a pyramid network feature map P4; upsampling feature map L4 by a factor of 2 using the nearest neighbor method, adding the upsampled result to feature map L3, and convolving it through a 3×3 convolutional layer to obtain a pyramid network feature map P3; upsampling feature map L3 by a factor of 2 using the nearest neighbor method, adding the upsampled result to feature map L2, and convolving it through a 3×3 convolutional layer to obtain a pyramid network feature map P2; finally, a multi-scale feature pyramid structure is obtained, defined as follows.

[0118] P = {P2, P3, P4, P5, P6}

[0119] in, a = 2, 3, 4, 5, 6, H a H represents the height of the feature map at layer a. a =1 / 2, 1 / 4,

[0120] 1 / 8, 1 / 16, 1 / 32; Wa W is the width of the feature map of layer a. a =1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32;

[0121] Furthermore, the top-down connection is used to transfer high-level semantic information from the top-level feature map to the high-resolution bottom-level feature map.

[0122] Preferably, in step S2, the feature pyramid structure is used as input to the region proposal network to generate regions of interest for buildings. The specific steps are as follows:

[0123] Multiple original anchor boxes are set on each pixel of each feature map in the feature pyramid structure; the feature maps of the feature pyramid structure are {P2,P3,P4,P5,P6}, where P2 is the second layer feature map, P3 is the third layer feature map, P4 is the fourth layer feature map, P5 is the fifth layer feature map, and P6 is the sixth layer feature map.

[0124] The category confidence and bounding box prediction value of each original anchor box are obtained using the region proposal network;

[0125] Each original anchor frame with a category confidence level is corrected based on the corresponding boundary prediction value to obtain a corrected anchor frame, which is then represented as a building region of interest with a category confidence level.

[0126] Furthermore, the bounding box prediction value is the offset between the true position and the current position of each anchor box;

[0127] The region of interest for a building is a quadrilateral with class confidence, obtained by adding the feature map anchor box and the boundary prediction value.

[0128] The set multiple anchor frames have one size and multiple aspect ratios; the number of anchor points is equal to the number of aspect ratio types;

[0129] Furthermore, the region proposal network is an algorithm that automatically extracts key regions in an image. It takes an image (of any size) as input and outputs a set of rectangular target proposal boxes, each box having a score.

[0130] The region of interest for a building is a selected area in the image to be analyzed. This area is the focus of image analysis. The region proposal network is used to delineate this area to facilitate further processing. The region of interest for a building is the area in the image where buildings are located. Extracting these areas will greatly improve the efficiency and accuracy of image processing and analysis.

[0131] Furthermore, the region proposal network takes the feature map {P2,P3,P4,P5,P6} of the feature pyramid structure as input;

[0132] Three anchor boxes are set at each pixel location of the feature map at each scale. The feature map is first passed through a 3×3 convolution to generate a feature map with 256 channels. The confidence scores for the d-th class are then obtained by performing two different 1×1 convolutions. and the predicted value of the e-th bounding box

[0133]

[0134] Where 3 represents the class confidence of the three anchor boxes for each pixel, 12 represents the four coordinates of the bounding box corresponding to the three anchor boxes for each pixel, the four coordinates represent the offset of the anchor point coordinates in the coordinates of the image to be detected, d = 1, 2, 3...D, representing D class confidences; e = 1, 2, 3...E, representing E bounding box prediction values;

[0135] Traverse the feature maps of the five scales in the feature pyramid structure to obtain the class confidence and bounding box prediction values ​​corresponding to multiple original anchor boxes in the feature pyramid structure.

[0136] Each original anchor frame is corrected based on its corresponding bounding box prediction value to obtain a corrected anchor frame, which is then represented as a building region of interest; the building region of interest is a building region of interest with a category confidence prediction value.

[0137] Preferably, in step S3, the feature pyramid structure and the building region of interest are used as input to the building outline prediction network to classify the building, predict building bounding box regression, and predict building vertex sequence, thereby obtaining the building's category label, bounding box, and vertex sequence; the specific steps include:

[0138] Based on the threshold of class confidence, class confidence is determined for each region of interest of a building with class confidence.

[0139] Filter out the regions of interest for buildings with low category confidence scores, and what remains are the regions of interest for buildings with high category confidence scores;

[0140] Collect building regions of interest with high confidence scores across multiple categories to construct a set of building regions of interest;

[0141] The set of regions of interest and the feature pyramid structure of buildings are input into the building classification branch, the bounding box regression branch and the building vertex sequence prediction branch of the building outline prediction network, respectively, and the output includes the category labels, bounding boxes and vertex sequences of multiple buildings.

[0142] Furthermore, the threshold value for the category confidence score is between 0.5 and 1. When the category confidence score is less than 0.5, it is considered a low category confidence score.

[0143] Furthermore, the specific steps for outputting the category labels of buildings are as follows:

[0144] The feature maps {P2, P3, P4, P5} in the feature pyramid structure and the set of building regions of interest are input into the building classification branch of the building outline prediction network.

[0145] Based on the size of each building region of interest in the set of building regions of interest, each building region of interest is mapped to the features of {P2, P3, P4, P5} in the feature pyramid structure to obtain multiple building region of interest feature maps.

[0146] The ROI-Align algorithm is used to perform fully connected and softmax activation function calculations on the extracted region feature maps of each building's region of interest feature map to obtain the category label of the region feature map. The category label of the region feature map is used to represent the category label of the building. The category label of the region feature map includes the building target and the background, with 1 representing the building target and 0 representing the background.

[0147] Furthermore, the ROI-Align algorithm is used to extract 7×7 pixels of the region feature map from the feature map of each building's region of interest.

[0148] Preferably, step S3, outputting the bounding boxes of multiple buildings, specifically includes the following steps:

[0149] The feature maps {P2, P3, P4, P5} in the feature pyramid structure and the set of building regions of interest are input into the bounding box regression branch of the building outline prediction network.

[0150] Based on the size of each building region of interest in the set of building regions of interest, each building region of interest is mapped to the features of {P2, P3, P4, P5} in the feature pyramid structure to obtain multiple building region of interest feature maps.

[0151] The ROI-Align algorithm is used to compute a fully connected layer on the feature map of each building's region of interest, outputting the bounding box (x) of each region's feature map. j ,y j ,w j ,h j ), where x j The x-axis coordinates and y-axis coordinates of the center point of the bounding box of the j-th region feature map are represented by the following values. j w represents the y-axis coordinate of the center point of the bounding box of the j-th region feature map; j h represents the width of the bounding box of the feature map of the j-th region. jLet J represent the height of the bounding box of the j-th region feature map, where j = 1, 2, 3...J, and represents the J-th region feature map.

[0152] Furthermore, the ROI-Align algorithm is used to extract 7×7 pixels of the region feature map from the feature map of each building's region of interest.

[0153] Preferably, step S3, which involves outputting a vertex sequence of multiple buildings, specifically includes the following steps:

[0154] The feature map and the region of interest of the building in the feature pyramid structure are input into the building vertex sequence prediction branch of the building outline prediction network;

[0155] The feature maps of the feature pyramid structure are subjected to feature fusion processing. The intermediate feature map of the feature pyramid structure is used as the base feature map. Feature maps larger than the base feature map are adjusted to the same size as the base feature map through pooling operation. Feature maps smaller than the base feature map are adjusted to the same size as the base feature map through linear interpolation. The processed feature maps of the same size are added together and averaged to obtain the fused feature map.

[0156] By mapping the regions of interest of buildings in the set of building regions of interest to the fused feature map, multiple feature maps of building regions of interest are obtained.

[0157] The feature maps of multiple building regions of interest are flattened into two-dimensional sequence feature maps in the spatial dimension, and then the two-dimensional positional encoding of the two-dimensional sequence feature maps is obtained by using sine functions of different frequencies.

[0158] The feature vectors of multiple region feature maps are obtained by adding the value of each two-dimensional sequence feature and the corresponding two-dimensional position encoding.

[0159] The feature vectors of multiple region feature maps are input into the encoding block, and after point-by-point, row-by-row and column-by-column self-attention processing, a forward fully connected layer is used to obtain multiple encoding layer building feature maps with rich semantics and geometric structure.

[0160] Define a learnable query set, input the learnable query set into the decoding block and process it through a self-attention mechanism to obtain an updated learnable query set;

[0161] Multiple building feature maps from the coding layers are input into the decoding block, and cross-attention processing is performed between the updated learnable query set and each building feature map from the coding layer. Then, the result is passed through a forward fully connected layer to obtain the dynamic vertices of multiple buildings.

[0162] Based on the updated learnable query set, the dynamic vertices of multiple buildings are updated iteratively. After passing through the four layers of decoders in the decoding block, the iteration stops and the final dynamic vertices of multiple buildings are output.

[0163] The final dynamic vertices of the building are processed by a fully connected layer and an activation function to obtain multiple predicted building vertex sequences;

[0164] The final dynamic vertices of multiple buildings are processed through a fully connected layer and a softmax activation function to obtain multiple predicted building vertex sequences.

[0165] Furthermore, the feature maps in the feature pyramid structure are {P2, P3, P4, P5}. The feature maps {P2, P3, P4, P5} of the feature pyramid structure are subjected to feature fusion processing. Based on feature map P3, feature map P2 is adjusted to the same size as P3 through pooling operations, and feature maps P4 and P5 are adjusted to the same size as P3 through linear interpolation. The fused feature map is obtained by adding and averaging the feature maps of all processed sizes. Furthermore, the ROI-Align algorithm is used to extract the region feature map of the building region of interest for each building region of interest with a pixel size of 20×20.

[0166] The learnable query set Z∈R 20×256 This is a 20×256 vector, used to query vertex information from the feature map output by the encoder. The technical solution of this invention obtains the dynamic vertices of the building, updating points based on other points in each iteration, fully utilizing the information from previous and subsequent points to improve accuracy.

[0167] Preferably, the building outline prediction network in step S3 is obtained by training the building extraction dataset;

[0168] The building outline prediction network described in step S3 is trained using a joint loss function; the building extraction dataset is the Crowdai dataset.

[0169] The expression for the joint loss function is:

[0170] L=λ cls L cls +λ reg L reg +λ verseq L verseq

[0171] Where L is the joint loss function, L cls L represents the classification loss function. reg L represents the bounding box regression loss function. verseq Let λ represent the vertex sequence loss function. cls λ represents the weights of the classification loss function. reg λ represents the weights of the bounding box regression loss function.verseq This represents the weights of the vertex sequence loss function.

[0172] Furthermore, the classification loss function is expressed as follows:

[0173]

[0174] Among them, L cls For the classification loss value, y i Let y be the predicted category label for the i-th building, i = 1, 2, 3…N, where N represents the total number of buildings. i = 0 or 1, 0 represents the background, 1 represents the building; p(y i The classification branch module outputs the probability of the predicted category label for the i-th building.

[0175] Furthermore, the bounding box regression loss function L reg The formula is:

[0176]

[0177] Among them, L reg The bounding box regression loss value, For the bounding box prediction of the i-th building, t i Let be the truth value of the bounding box of the i-th building, where i = 1, 2, 3...N, and N represents the total number of buildings; Let these be the coordinates of the center point of the true bounding box of the i-th building. Let be the width and height of the true bounding box of the i-th building; Let the coordinates be the center point coordinates of the i-th building boundary prediction box. Let be the width and height of the predicted bounding box for the i-th building.

[0178] Furthermore, the vertex sequence loss function L verseq It is a bidirectional loop loss, specifically including:

[0179] Obtain the ground truth value O of the vertex sequence of manually labeled buildings and the first vertex o0 of the ground truth value O of the vertex sequence of buildings, and the predicted value of the vertex sequence of buildings. Find the vertex corresponding to the first vertex o0. Q represents the total number of predicted vertex sequence values ​​for the building;

[0180] With the corresponding vertex Based on the moving vertex sequence prediction value Other vertices in the equation make the corresponding vertex Starting from this point, we obtain new vertex sequence predictions.

[0181] Fixed new vertex sequence prediction values Starting from the given point, reset the other vertices in the new vertex sequence prediction values ​​in a counter-clockwise order to obtain the reverse order of the new vertex sequence prediction values.

[0182] Calculate the true value O of the building vertex sequence and the predicted value of the new vertex sequence, respectively. The reverse order of the new vertex sequence prediction values The loss between them is used to obtain the loss L of the new vertex clockwise. ce And the loss of the new vertex counterclockwise

[0183] Loss L of taking the new vertex clockwise ce And the loss of the new vertex counterclockwise The minimum value is used as the loss L for vertex sequence prediction. verseq ;

[0184] The loss L of the new vertex clockwise ce And the loss of the new vertex counterclockwise The expressions are as follows:

[0185]

[0186]

[0187] Among them, L ce The loss is calculated clockwise from the new vertices, where m = 1, 2, 3…M, and M is the number of samples for the new vertices. For a new vertex function in a clockwise direction, if the m-th new vertex sample corresponds to the v-th class label in a clockwise direction, then... Otherwise, it equals 0, v = 1, 2, 3...V, V clockwise building category labels. Let m be the probability that the m-th new vertex sample belongs to the v-th class label in the clockwise direction;

[0188] The loss is due to the counter-clockwise rotation of the new vertex. Let m be the new vertex function in the counter-clockwise direction. If the m-th new vertex sample corresponds to the invv-th category label in the counter-clockwise direction, then take... Otherwise, it equals 0, and invv = 1, 2, 3, ... INNV building category labels in a counter-clockwise direction. This represents the probability that the m-th new vertex sample belongs to the invv-th category label in the counterclockwise direction;

[0189] Loss L of taking the new vertex clockwise ce And the loss of the new vertex counterclockwise The minimum value is used as the loss L for vertex sequence prediction. verseq.

[0190] The loss L for predicting the vertex sequence verseq The expression is:

[0191]

[0192] Among them, L verseq L is the loss for predicting the vertex sequence. ce The loss is clockwise for the new vertex. The loss is counterclockwise for the new vertex.

[0193] Preferably, step S4, which involves obtaining the building outline in the image using the building's category label, bounding box, and vertex sequence, specifically includes the following steps:

[0194] By locating buildings in an image using bounding boxes, and then using the bounding boxes to supervise the training of a building contour prediction network, the building contour prediction network can accurately locate the building regions in the image.

[0195] The building category label is used to determine whether the building in the bounding box-located image is a building area;

[0196] If it is a building area, obtain the vertex sequence prediction value of the building area, and represent the vertex sequence prediction value as the building outline.

[0197] To verify the impact of feature encoding of different dimensions, learning rate weights of different modules, and model training strategies on model performance, relevant experiments were conducted.

[0198] (1) Feature encoding in different dimensions

[0199] In the building vertex sequence prediction branch, a self-attention mechanism is first used to encode building features to obtain semantic and geometric features. During feature encoding, the feature matrix can be encoded along three dimensions: points, rows, and columns. When encoding features point-by-point, the feature matrix is ​​stretched into a sequence point by point, and then encoded using the self-attention mechanism. When encoding features row-by-row or column-by-column, the feature matrix is ​​stretched into a sequence row by row, and then encoded using the self-attention mechanism. As shown in Table 1, the model performs better when using point-by-point, row-by-row, and column-by-column feature encoding alone than when using all three. Therefore, the model chooses a combination of all three methods for feature encoding.

[0200] Table 1 Comparison of Feature Encoding Performance in Different Dimensions

[0201]

[0202]

[0203] (2) Learning rate weights for different modules

[0204] In the process of extracting building vertices, there are three sub-tasks: building classification, bounding box regression, and vertex sequence prediction. Therefore, the model uses classification loss, regression loss, and vertex sequence prediction loss to supervise model training, respectively. The learning rate weights of different modules have a significant impact on model training. This embodiment uses different λ... cls , λ reg and λ verseq To verify the learning rate weights, the experimental results are shown in Table 3 below. The three modules have a consistent impact on the model training, so the model chooses the same weights to supervise the model training.

[0205] Table 2 Performance impact of learning rate weights for different modules

[0206]

[0207] (3) Comparison with other methods

[0208] This invention compares its technical solution with those of Mask R-CNN, PANet, PolyMapper, PolyWorld, and FrameField on publicly available datasets. Since Mask R-CNN and PANet output building masks, the experimental process post-processes the obtained building masks, using a contour fitting algorithm to fit the mask results into building polygons for comparison. Other comparison methods and this method can directly obtain the building contour polygons without post-processing. As shown in Table 3, in F1... 50 The accuracy of this method is 18.4% higher than that of the fitting algorithm and 1.6% higher than that of other building vertex extraction methods; in F1... 75 The accuracy of the proposed method is 16.8% higher than that of the fitting algorithm and 4.4% higher than that of other building vertex extraction methods, thus proving the effectiveness of the proposed method.

[0209] Table 3 compares the proposed method with other methods.

[0210]

[0211] This invention utilizes the attention-based and dynamic vertex-based remote sensing image building extraction method proposed by MS COCO. 50 AP 75 AR 50 AR 75 Evaluation metrics are provided for reference. AP and AR represent average precision and average recall, respectively, with subscripts 50 and 75 indicating that the intersection ratio between predicted and true buildings is 0.5 and 0.75, respectively. 50 and AR50 The overall precision and recall of the buildings extracted by the evaluation model, AP 75 and AR 75 The model's building extraction precision and recall were evaluated at a higher matching accuracy. Finally, the F1 composite score was used to comprehensively evaluate the building extraction performance, considering both AP and AR metrics.

[0212] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting building contours based on attention mechanisms and dynamic vertices, characterized in that, include: S1. Input the image to be detected into a deep convolutional neural network, extract the multi-scale features of the image, and fuse the multi-scale features of the image to obtain a feature pyramid structure. S2. Using the feature pyramid structure as input to the region proposal network, generate regions of interest for buildings. The specific steps are as follows: The image to be detected is input into a deep convolutional neural network, and feature maps of a feature pyramid structure are output according to the feature map sizes of different layers in the backbone of the deep convolutional neural network. Multiple original anchor boxes are set on each pixel of each feature map in the feature pyramid structure; The category confidence and bounding box prediction value for each original anchor box are obtained using a region proposal network; Each original anchor frame with a category confidence score is corrected according to the corresponding boundary prediction value to obtain a corrected anchor frame, which is then represented as a building region of interest with a category confidence score. S3. Using the feature pyramid structure and the building region of interest as input to the building outline prediction network, obtain the building's category label, bounding box, and vertex sequence. This specifically includes the following steps: The feature map and the region of interest of the building in the feature pyramid structure are input into the building vertex sequence prediction branch of the building outline prediction network; The feature maps of the feature pyramid structure are subjected to feature fusion processing. The intermediate feature map of the feature pyramid structure is used as the base feature map. Feature maps larger than the base feature map are adjusted to the same size as the base feature map through pooling operation. Feature maps smaller than the base feature map are adjusted to the same size as the base feature map through linear interpolation. The processed feature maps of the same size are added together and averaged to obtain the fused feature map. By mapping the regions of interest of buildings in the set of regions of interest of buildings to the fused feature map, feature maps of multiple regions of interest of buildings are obtained. The feature maps of multiple building regions of interest are flattened into two-dimensional sequence feature maps in the spatial dimension, and the two-dimensional sequence feature maps are then encoded using sine functions of different frequencies to obtain two-dimensional positional codes for the two-dimensional sequence feature maps. The feature vectors of multiple region feature maps are obtained by adding the value of each two-dimensional sequence feature and the corresponding two-dimensional position code. The feature vectors of multiple region feature maps are input into the encoding block, and after point-by-point, row-by-row and column-by-column self-attention processing, a forward fully connected layer is used to obtain building feature maps of multiple encoding layers. Define a learnable query set, input the learnable query set into the decoding block and process it through a self-attention mechanism to obtain an updated learnable query set; Multiple building feature maps from the coding layers are input into the decoding block, and cross-attention processing is performed between the updated learnable query set and each building feature map from the coding layer. Then, the result is passed through a forward fully connected layer to obtain the dynamic vertices of multiple buildings. Based on the updated learnable query set, the dynamic vertices of multiple buildings are updated iteratively. After passing through multiple decoders of the decoding block, the iteration stops and the final dynamic vertices of multiple buildings are output. The final dynamic vertices of multiple buildings are processed through fully connected layers and activation functions to obtain the predicted building vertex sequence; The predicted sequence of building vertices is represented as the outline of the buildings in the image; S4. Obtain the building outlines in the image using the building category labels, bounding boxes, and vertex sequences.

2. The building outline extraction method according to claim 1, characterized in that, Step S3 involves using the feature pyramid structure and the building's region of interest as input to a building contour prediction network to obtain the building's category label, bounding box, and vertex sequence; the specific steps include: Based on the threshold of class confidence, class confidence is determined for each region of interest of a building with class confidence. Identify regions of interest for buildings with high category confidence scores; The regions of interest of buildings with high confidence scores across multiple categories constitute a set of regions of interest for buildings. The set of regions of interest and the feature pyramid structure of buildings are input into the building classification branch, the bounding box regression branch and the building vertex sequence prediction branch of the building outline prediction network, respectively, and the output includes the category labels, bounding boxes and vertex sequences of multiple buildings.

3. The building outline extraction method according to claim 1, characterized in that, Step S4 obtains the building outline in the image using the building's category label, bounding box, and vertex sequence. Specific steps include: Locate buildings in an image using bounding boxes; The category label of the building is used to determine whether the building in the bounding box positioning image is a building area; If it is a building area, obtain the vertex sequence prediction value of the building area, and represent the vertex sequence prediction value as the building outline.

4. The building outline extraction method according to claim 1, characterized in that, The specific steps for obtaining the building category label in step S3 are as follows: The feature map and the set of building regions of interest in the feature pyramid structure are input into the building classification branch of the building outline prediction network; Based on the size of each region of interest in the set of building regions of interest, each building region of interest is mapped to a feature map in the feature pyramid structure to obtain the feature map corresponding to each region of interest; Fully connected components and activation functions are calculated in the feature map corresponding to each region of interest to obtain the category label of the region feature map. The category label of the region feature map is then used to represent the category label of the building.

5. The building outline extraction method according to claim 1, characterized in that, The extracted building dataset is input into the joint loss function for training to obtain the building outline prediction network described in step S3; The expression for the joint loss function is: Where L is the joint loss function, Represents the classification loss function. This represents the bounding box regression loss function. Represents the vertex sequence loss function. Represents the weights of the classification loss function. This represents the weights of the bounding box regression loss function. This represents the weights of the vertex sequence loss function.

6. The building outline extraction method according to claim 5, characterized in that, The classification loss function is expressed as follows: in, For classification loss value, Let be the predicted category label for the i-th building, where i = 1, 2, 3…N, and N represents the total number of buildings. =0 or 1, where 0 represents the background and 1 represents the building; Output the probability of the predicted category label for the i-th building to the classification branch module.

7. The building outline extraction method according to claim 6, characterized in that, Bounding box regression loss function The formula is: in, The bounding box regression loss value, Let be the truth value of the bounding box of the i-th building, i=1,2,3…N, where N represents the total number of buildings.

8. The building outline extraction method according to claim 6, characterized in that, The vertex sequence loss function It is a bidirectional loop loss, specifically including: Obtain the ground truth value O of the vertex sequence of manually labeled buildings and the first vertex o0 of the ground truth value O of the vertex sequence of buildings, and the predicted value of the vertex sequence of buildings. Find the vertex corresponding to the first vertex o0. ,q=1,2,3…Q, where Q represents the total number of predicted vertex sequence values ​​for the building; With the corresponding vertex Based on the moving vertex sequence prediction value Other vertices in the equation make the corresponding vertex Starting from this point, we obtain new vertex sequence predictions. ; Fixed new vertex sequence prediction values Starting from the given point, reset the other vertices in the new vertex sequence prediction values ​​in a counter-clockwise order to obtain the reverse order of the new vertex sequence prediction values. ; Calculate the true value O of the building vertex sequence and the predicted value of the new vertex sequence, respectively. The reverse order of the new vertex sequence prediction values The loss between them is used to obtain the loss of the new vertex clockwise. And the loss of the new vertex counterclockwise ; Loss of taking a new vertex clockwise And the loss of the new vertex counterclockwise The minimum value is used as Loss of vertex sequence prediction .

Images