Building fine-grained edge extraction and classification method, device, equipment and medium

Abstract

The application provides a building fine-grained edge extraction and classification method, device, equipment and medium, relates to the technical field of remote sensing detection, and includes size cropping on the obtained building remote sensing image, and constructing a training sample set, inputting the training set into a plurality of instance segmentation networks selected in advance for training, obtaining a plurality of first building identification models, and obtaining corresponding first identification results output after inputting a test set; the first identification result is used as a label of the test set to expand and optimize the training of the training set, obtain a second building identification model corresponding to each first building identification model, and obtain corresponding second identification results output after inputting a test set; the plurality of second identification results are subjected to fine-grained mask fusion processing to obtain a building category, a confidence score, a building detection frame and a building fine-grained mask. The application can effectively improve the building boundary extraction precision and the classification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing detection, in particular to a building fine-grained edge extraction and classification method, device, equipment and medium. BACKGROUND

[0002] At present, when building recognition extraction is performed, related technologies adopt a building recognition algorithm based on deep learning. However, such an algorithm usually labels buildings according to one category, which cannot completely meet the application level requirements of building category detection in city planning; meanwhile, for instance segmentation extraction networks, the outline extraction of the mask is not fine enough, which cannot meet the fine-grained extraction of buildings. Moreover, in actual application, the building categories have a long-tail distribution, and the extraction accuracy of tail samples is even worse. Therefore, the current building recognition and extraction method has the problems of not fine enough extraction outline and poor extraction accuracy. SUMMARY

[0003] The purpose of the present application is to provide a building fine-grained edge extraction and classification method, device, equipment and medium, which can effectively improve the building boundary extraction accuracy and classification result.

[0004] In a first aspect, the present application provides a building fine-grained edge extraction and classification method, which comprises: acquiring a building remote sensing image, performing size cropping on the building remote sensing image, and constructing a training sample set based on the cropped building remote sensing image; the training sample set comprises a training set and a test set, and the building roof outline in the training set picture is instance labeled according to the building type; the training set is input into a plurality of pre-selected instance segmentation networks, the training set is subjected to data enhancement by using a class dynamic balance sampling enhancement method, the training set after data enhancement is used to train the instance segmentation networks respectively, a first building recognition model corresponding to each instance segmentation network is obtained, the test set is input into the corresponding first building recognition model, and a corresponding first recognition result output by each first building recognition model is obtained; the first recognition result is used as a label of the test set to expand the training set, each first building recognition model is subjected to optimization training through the expanded training set, a second building recognition model corresponding to each first building recognition model is obtained, the test set is input into each second building recognition model respectively, and a corresponding second recognition result output by each second building recognition model is obtained; the corresponding second recognition result output by each second building recognition model is subjected to fine-grained mask fusion processing, and a target recognition result is obtained; wherein the target recognition result comprises a building category, a confidence score, a building detection frame and a building fine-grained mask.

[0005] In an optional implementation, the training set is augmented using a category dynamic equalization sampling method, including: calculating the instance proportion of each building category in the training set; determining the augmentation probability of each category based on the instance proportion and a preset equalization rate; determining the image sampling probability based on the building categories included in the training set; selecting source and target images sequentially in the training set based on the image sampling probability, cropping the target objects from the source images, and performing data augmentation processing, which includes at least random scaling, random rotation, and random blurring; and randomly selecting a preset number of target objects based on the augmentation probability and pasting them into the target images to obtain the data-augmented training set.

[0006] In an optional implementation, the percentage of instances corresponding to each building category in the training set is calculated using the following formula:

[0007]

[0008] Among them, f c c represents the percentage of instances; n represents the building category; m represents the number of categories. c This refers to the number of instances under the corresponding building category.

[0009] In an optional implementation, determining the image sampling probability based on the building categories included in the training set includes: when the number of target objects in the images included in the training set is 0, image I is not sampled; when the number of target object categories in the images included in the training set is 1, the category augmentation probability is determined as the image sampling probability corresponding to that image; when the number of target object categories in the images included in the training set is greater than 1, the value with the largest category augmentation probability among the target categories included in that image is determined as the image sampling probability corresponding to that image.

[0010] In an optional implementation, each first building recognition model is optimized and trained using the expanded training set to obtain a second building recognition model corresponding to each first building recognition model. This includes: when optimizing and training each first building recognition model using the expanded training set, the learning rate is reduced by a preset factor and the feature extraction network parameters are fixed for optimization training.

[0011] In an optional implementation, the corresponding second recognition result output by each second building recognition model is subjected to fine-grained mask fusion processing to obtain the target recognition result. This includes: dividing the corresponding second recognition result output by each second building recognition model according to building category; clustering the results in each building category based on the building fine-grained mask to obtain the target cluster corresponding to each building category; fusing the masks in each target cluster and recalculating their confidence to obtain a new building fine-grained mask and corresponding confidence score; determining the maximum bounding rectangle of the building fine-grained mask as the building detection box; and obtaining the target recognition result after traversing all target clusters and categories.

[0012] In an optional implementation, the second identification result includes building category, confidence score, building detection box, and building fine-grained mask. The second identification result output by each second building identification model is divided according to building category. Based on the building fine-grained mask, the results in each building category are clustered to obtain target clusters corresponding to each building category. This includes: dividing the second identification result output by each second building identification model according to building category; for each building category, sorting by confidence score in descending order, and determining the result with the highest confidence score as the first target cluster; calculating the intersection-union ratio (IUGR) of the building fine-grained mask with the second-highest confidence score and the corresponding building fine-grained mask in the first target cluster; if the IUGR is greater than a threshold, it is classified as the first target cluster; if the IUGR is less than a threshold, it is the second target cluster; and so on, selecting results with descending confidence scores for IUGR calculation until all results are traversed to determine all target clusters.

[0013] Secondly, the present invention provides a fine-grained edge extraction and classification device for buildings. The device includes: a sample construction module for acquiring remote sensing images of buildings, cropping the remote sensing images, and constructing a training sample set based on the cropped remote sensing images; the training sample set includes a training set and a test set, and the roof outlines of buildings in the training set images are instantiated and labeled according to the building type; a first recognition module for inputting the training set into multiple pre-selected instance segmentation networks, using a class dynamic equalization sampling enhancement method to perform data augmentation on the training set, training the instance segmentation networks with the data-enhanced training set to obtain a first building recognition model corresponding to each instance segmentation network, and inputting the test set into the corresponding first building recognition model. The system consists of a first building recognition model and a second recognition module. The first recognition module uses the first recognition results as labels for the test set to expand the training set. The expanded training set is then used to optimize and train each first building recognition model, resulting in a second building recognition model corresponding to each first building recognition model. The test set is then input into each second building recognition model to obtain the corresponding second recognition result. A fusion module performs fine-grained mask fusion processing on the corresponding second recognition results output by each second building recognition model to obtain the target recognition result. The target recognition result includes building category, confidence score, building detection box, and fine-grained building mask.

[0014] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the building fine-grained edge extraction and classification method of any of the foregoing embodiments.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the building fine-grained edge extraction and classification method of any of the foregoing embodiments.

[0016] The method, apparatus, equipment, and medium for fine-grained edge extraction and classification of buildings provided in this application extract fine-grained building edges from remote sensing images and classify buildings through dynamic equalization sampling enhancement, supervised training, model fine-tuning, and fine-grained mask fusion. The obtained building outlines are more refined. At the same time, the category dynamic equalization sampling enhancement can alleviate the long-tail distribution of samples and prevent overfitting, which can effectively improve the accuracy of building boundary extraction and classification results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for fine-grained edge extraction and classification of buildings, provided in an embodiment of this application;

[0019] Figure 2 A structural diagram of a building fine-grained edge extraction and classification device provided in an embodiment of this application;

[0020] Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] This application provides a method, apparatus, device, and medium for fine-grained edge extraction and classification of buildings, which can effectively improve the accuracy of building boundary extraction and classification results.

[0025] This application provides a method for fine-grained edge extraction and classification of buildings. See also... Figure 1 As shown, the method mainly includes the following steps:

[0026] Step S110: Acquire remote sensing images of buildings, crop the size of the remote sensing images of buildings, and construct a training sample set based on the cropped remote sensing images of buildings; the training sample set includes a training set and a test set, and the roof outlines of buildings in the training set images are instantiated and labeled according to the building type.

[0027] When performing size cropping, the cropping can be performed according to the input sample requirements of the subsequently selected instance segmentation network to meet the requirements of the model.

[0028] Step S120: Input the training set into multiple pre-selected instance segmentation networks respectively. The training set is augmented using the category dynamic equalization sampling enhancement method. The instance segmentation networks are trained using the augmented training set to obtain the first building recognition model corresponding to each instance segmentation network. The test set is then input into the corresponding first building recognition model to obtain the corresponding first recognition result output by each first building recognition model.

[0029] Step S130: The first recognition result is used as the label of the test set to expand the training set. The expanded training set is used to optimize and train each first building recognition model to obtain a second building recognition model corresponding to each first building recognition model. The test set is then input into each second building recognition model to obtain the corresponding second recognition result output by each second building recognition model.

[0030] Step S140: Perform fine-grained mask fusion processing on the corresponding second recognition result output by each second building recognition model to obtain the target recognition result; wherein, the target recognition result includes building category, confidence score, building detection box and building fine-grained mask.

[0031] The methods provided in the embodiments of this application will be described in detail below.

[0032] In one implementation, the training set is augmented using a class dynamic equalization sampling method, which may include the following steps 1-1 to 1-5:

[0033] Step 1-1: Calculate the percentage of instances corresponding to each building category in the training set. In practice, the following formula is used:

[0034]

[0035] Among them, f c c represents the percentage of instances; n represents the building category; m represents the number of categories. c This refers to the number of instances under the corresponding building category.

[0036] In one implementation, n can be selected as 4. Since it is a remote sensing image, the building categories can be divided by the roof. That is, the building categories can include flat roof, pitched roof, curved roof, and multi-wave folded plate roof.

[0037] Steps 1-2 determine the augmentation probability for each category based on the instance proportion and the preset equalization rate. In practice, the augmentation probability can be calculated using the following formula:

[0038]

[0039] Among them, P c To enhance the probability; α is the equilibrium rate, α∈[0,1).

[0040] Steps 1-3 determine the image sampling probability based on the building categories included in the training set. In practice, this can be further implemented by including steps 1-3.1 to 1-3.3:

[0041] Step 1-3.1: When the number of target objects in the images contained in the training set is 0, do not sample image I;

[0042] Step 1-3.2: When the number of target object categories in the images contained in the training set is 1, the category augmentation probability is determined as the image sampling probability corresponding to that image;

[0043] Step 1-3.3: When the number of target object categories in the images contained in the training set is greater than 1, the value with the highest category augmentation probability among the target categories contained in the image is determined as the image sampling probability corresponding to that image.

[0044] In practical applications, when the number of targets in image I is 0, image I is not sampled, i.e., P I =0; When the number of target categories in image I is 1, the sampling probability of image I is the category augmentation probability of that category, i.e., P I =P c When the number of target categories in image I is greater than 1, the sampling probability of image I is the value with the highest category enhancement probability among the target categories contained in image I, i.e., P. I =max c∈i P c , where i is the set of target categories contained in image I.

[0045] Steps 1-4: Based on the image sampling probability, source images and target images are selected sequentially in the training set. The target object is cropped out from the source image and then subjected to data augmentation processing.

[0046] In one implementation, assuming the batch size is b, the enhancement frequency P is... ISelect a source image and a target image, and crop the target image from the source image. Data augmentation processing includes at least random scaling, random rotation, and random blurring.

[0047] Steps 1-5: Randomly select a preset number of target objects based on the augmentation probability and paste them into the target image to obtain the data-augmented training set.

[0048] In practical applications, according to P c The probability of randomly selecting n (n can be 1 to the number of targets in the source image) is given. m Paste the target image into the target image, ensuring there are no overlapping pixels. Repeat this process b times to reach the batch size, then feed the data-augmented images into the network for learning.

[0049] To further ensure the optimization training of each first building recognition model using the expanded training set to obtain a second building recognition model corresponding to each first building recognition model, the learning rate can be reduced by a preset factor and the feature extraction network parameters can be fixed during the optimization training of each first building recognition model using the expanded training set. Optionally, this preset factor can be 10 times.

[0050] Furthermore, the corresponding second recognition results output by each second building recognition model are subjected to fine-grained mask fusion processing to obtain the target recognition result, which may include the following steps 2-1 to 2-4:

[0051] Step 2-1: Divide the corresponding second recognition results output by each second building recognition model according to the building category, and cluster the results in each building category based on the fine-grained building mask to obtain the target cluster corresponding to each building category;

[0052] Step 2-2: Fuse the masks in each target cluster, recalculate their confidence scores, and obtain a new fine-grained mask for the building and its corresponding confidence score.

[0053] Steps 2-3: Determine the maximum bounding rectangle of the building fine-grained mask as the building detection frame;

[0054] Steps 2-4: After traversing all target clusters and categories, the target recognition result is obtained.

[0055] The aforementioned second identification result includes building category, confidence score, building detection box, and fine-grained building mask. The corresponding second identification result output by each second building identification model is divided according to building category. Based on the fine-grained building mask, the results in each building category are clustered to obtain the target cluster corresponding to each building category. In specific implementation, this may further include the following steps:

[0056] Step 3-1: Divide the corresponding second recognition results output by each second building recognition model according to the building category;

[0057] Step 3-2: For each building category, sort them in descending order according to their confidence scores, and determine the results with the highest confidence scores as the first target cluster;

[0058] Step 3-3: Calculate the intersection-union ratio (IUR) of the fine-grained mask of the building with the second highest confidence score and the corresponding fine-grained mask of the building in the first target cluster. If the IUR is greater than the threshold, it is classified as the first target cluster. If the IUR is less than the threshold, the second target cluster is obtained.

[0059] Steps 3-4: Select the confidence scores in descending order and perform crossover and reunion calculations until all results have been traversed to determine the entire target cluster.

[0060] Specifically, in practical applications, all results can first be sorted in descending order of confidence level; the results with the highest confidence level (W1, B1, M1) are compressed into list L to form a target cluster Z1; the mask M2 of the result with the second highest confidence level (W2, B2, M2) is compared with the masks of all results in L, i.e., M1 at this point, and the intersection-over-union ratio (IoU) is calculated to find the corresponding target cluster; if the largest intersection-over-union ratio (IoU) is found... max If the result is greater than the threshold β, the match is successful, and this result is classified into the target cluster with the maximum intersection-union ratio (UU), i.e., Z1; otherwise, it becomes a new target cluster Z2 in L. After traversing all results, the target clusters of this category are obtained. Where R is the number of the target clusters, S r This represents the number of results for this target cluster.

[0061] For the i-th target cluster The resulting mask is obtained by retaining the pixels of each target pixel according to the majority rule, resulting in the fused mask M′. i Get the mask M′ i The largest bounding rectangle is the merged frame B′ i ;use Calculate the confidence level W′ after fusion. i S i For target cluster Z i The number of results, Ws For target cluster Z i The confidence level of the s-th result.

[0062] Furthermore, this application embodiment also provides a specific execution method, including the following steps S1 to S6:

[0063] Step S1: Acquire remote sensing images, crop the images to the size required by the model, and randomly divide the data into training set D. x and test set D c For training set D x The outlines of the building roofs in the image are instantiated and labeled, and the roofs are classified into four categories according to their architectural type: flat roofs, pitched roofs, curved roofs, and multi-wave folded plate roofs.

[0064] S2: Based on a pre-defined instance segmentation network, such as Mask R-CNN, a class dynamic equalization sampling enhancement method is used to learn the training set and the test set D. c The image is used to perform inference to obtain the first recognition result. The first recognition result should include the category of the detected building target, the confidence score, the building detection box, and the building mask.

[0065] Specifically, category dynamic equalization sampling enhancement methods include:

[0066] S21: Set the equalization sampling rate. Set the equalization rate α = 0.35, according to... Calculate the proportion f of instances in category c. c Where c is the category, m c This refers to the number of instances within that category;

[0067] S22: Based on the proportion of instances f c Calculate the augmentation probability for each category

[0068] S23: Calculate image I (I∈D) x The sampling probability of image I is: when the number of targets in image I is 0, image I is not sampled, i.e., P I =0; When the number of target categories in image I is 1, the sampling probability of image I is the category augmentation probability of that category, i.e., P I =P c When the number of target categories in image I is greater than 1, the sampling probability of image I is the value with the highest category enhancement probability among the target categories contained in image I, i.e., P. I =max c∈i P I , where i is the target category contained in image I;

[0069] S24: Assuming the batch size is b = 4, according to the image enhancement frequency P I Select a source image and a target image, crop the target from the source image, and perform operations such as random scaling, random rotation, and random blurring.

[0070] S25: According to P c The probability of randomly selecting n (n can be 1 to the number of targets in the source image) is given. m Paste the target into the target image, ensuring that there are no overlapping pixels between the pasted target and the target in the target image;

[0071] S26: Repeat the above operation b=4 times, and send the data-enhanced image into the network for learning.

[0072] S3: The first recognition result of the test set is used as the label of the test set and fed together with the training set into the instance segmentation network preset in S2 for fine-tuning training. The fine-tuning method includes reducing the learning rate by a factor of 10 and fixing the parameters of the feature extraction network. Inference is then performed on the images in the test set to obtain the second recognition result. The second recognition result should include the category, confidence score, building detection box, and building mask for each building.

[0073] S4: Change the preset instance segmentation network model structure, for example, to Refine Mask or Cascader Mask RCNN network, and repeat steps S2 and S3 to obtain three second recognition results.

[0074] S5: For a given image in the test set, the three second recognition results G are fed into the fine-grained mask fusion module for fusion to obtain the final recognition result G′. The final recognition result includes the category, confidence score, building bounding box, and fine-grained mask for each building. The specific representation of the result G is as follows:

[0075]

[0076] Where C j Represented as the category of the result, B j W represents the position of the predicted bounding box. j M represents the confidence score of the result. j Let J be the building mask and J be the number of results G.

[0077] The steps of the fine-grained mask fusion module are as follows:

[0078] S51: First, set the IoU threshold β = 0.55, divide the three second identification results into categories, and sort the results in each category according to their confidence level; the result G can be represented as:

[0079]

[0080] Where J = J1 + J2 + J3 + J4, J1, J2, J3 and J4 represent the number of results in the four categories, respectively.

[0081] S52: For the results under a certain category, such as the first category, sort them in descending order of confidence. Compress the results with the highest confidence (W1, B1, M1) into list L to form a target cluster Z1. Calculate the intersection-over-union ratio (IoU) of the mask M2 of the result with the second highest confidence (W2, B2, M2) with the masks of all results in L, i.e., M1 at this point, and find the corresponding target cluster; if the largest IoU is found... max If the result is greater than the threshold β, the match is successful, and this result is classified into the target cluster with the largest intersection-union ratio, i.e., Z1; otherwise, it becomes a new target cluster Z2 in L.

[0082] S53: Repeat step S52 until all results are traversed to obtain the target cluster list L1 for each category. List L1 can be represented as... Where R1 is the number of the first type of target clusters, S r The number of results under this target cluster is S1+S2+…+S for the first class. r =J1.

[0083] S54: For the results in a specific target cluster, for example Extract all the masks, retain each target pixel according to the majority rule, and obtain the fused mask M′1; take the largest bounding rectangle of mask M′1 as the fused bounding box B′1; use... Calculate the fusion confidence score W′1, where S1 is the number of results in the target cluster Z1, and W... s Let be the confidence level of the s-th result for the target cluster Z1.

[0084] S55: Repeat steps S52, S53, and S54 to iterate through all target clusters and obtain the final result of this test image. Where R is the total number of target clusters in this test image.

[0085] S6: Repeat step S5 to iterate through all images in the test set and obtain the final test results for the test set.

[0086] In summary, the fine-grained edge extraction and classification method for buildings provided in this application includes category dynamic equalization sampling enhancement, supervised training, model fine-tuning, and fine-grained mask fusion. This invention can extract and classify buildings, resulting in more refined building outlines. Simultaneously, category dynamic equalization sampling enhancement can alleviate the long-tailed distribution of samples while preventing overfitting.

[0087] Based on the above method embodiments, this application also provides a device for fine-grained edge extraction and classification of buildings, see [link to relevant documentation]. Figure 2 As shown, the device mainly includes the following parts:

[0088] The sample construction module 210 is used to acquire remote sensing images of buildings, crop the size of the remote sensing images of buildings, and construct a training sample set based on the cropped remote sensing images of buildings. The training sample set includes a training set and a test set. The roof outlines of buildings in the training set images are instantiated and labeled according to the building type.

[0089] The first recognition module 220 is used to input the training set into multiple pre-selected instance segmentation networks respectively. The training set is augmented using the category dynamic equalization sampling enhancement method. The instance segmentation network is trained with the data-enhanced training set to obtain the first building recognition model corresponding to each instance segmentation network. The test set is input into the corresponding first building recognition model to obtain the corresponding first recognition result output by each first building recognition model.

[0090] The second recognition module 230 is used to expand the training set by using the first recognition result as the label of the test set, optimize and train each first building recognition model with the expanded training set to obtain the second building recognition model corresponding to each first building recognition model, and input the test set into each second building recognition model to obtain the corresponding second recognition result output by each second building recognition model.

[0091] The fusion module 240 is used to perform fine-grained mask fusion processing on the corresponding second recognition results output by each second building recognition model to obtain the target recognition result; wherein, the target recognition result includes building category, confidence score, building detection box and building fine-grained mask.

[0092] In an optional implementation, the first identification module 220 is further configured to:

[0093] Calculate the percentage of instances corresponding to each building category in the training set;

[0094] The enhancement probability for each category is determined based on the instance proportion and the preset equalization rate;

[0095] The image sampling probability is determined based on the building categories included in the training set;

[0096] Based on the image sampling probability, source images and target images are selected sequentially in the training set. The target object is cropped out from the source image and data augmentation is performed. Data augmentation includes at least random scaling, random rotation and random blurring.

[0097] Based on the augmentation probability, a preset number of target objects are randomly selected and pasted into the target image to obtain the data-augmented training set.

[0098] In an optional implementation, the first identification module 220 is further configured to employ the following formula:

[0099]

[0100] Among them, f c c represents the percentage of instances; n represents the building category; m represents the number of categories. c This refers to the number of instances under the corresponding building category.

[0101] In an optional implementation, the first identification module 220 is further configured to:

[0102] When the number of target objects in the images contained in the training set is 0, image I is not sampled;

[0103] When the number of target object categories in the images contained in the training set is 1, the category augmentation probability is determined as the image sampling probability corresponding to that image;

[0104] When the number of target object categories in the images contained in the training set is greater than 1, the value with the highest category augmentation probability among the target categories contained in the image is determined as the image sampling probability corresponding to that image.

[0105] In an optional implementation, the second identification module 230 is further configured to:

[0106] When optimizing the training of each first building recognition model using the expanded training set, the learning rate is reduced by a preset factor and the feature extraction network parameters are fixed for optimization training.

[0107] In an optional implementation, the fusion module 240 is further configured to:

[0108] The corresponding second recognition results output by each second building recognition model are divided according to building category. Based on the fine-grained building mask, the results in each building category are clustered to obtain the target cluster corresponding to each building category.

[0109] The masks in each target cluster are fused together, and their confidence scores are recalculated to obtain a new fine-grained mask for the building and its corresponding confidence score.

[0110] The maximum bounding rectangle of the building fine-grained mask is determined as the building detection frame;

[0111] After traversing all target clusters and categories, the target recognition result is obtained.

[0112] In an optional implementation, the second identification result includes building category, confidence score, building detection box, and building fine-grained mask;

[0113] The aforementioned fusion module 240 is also used for:

[0114] The corresponding second recognition results output by each second building recognition model are divided according to the building category;

[0115] For each building category, the results are sorted in descending order of confidence score, and the results with the highest confidence score are identified as the first target cluster.

[0116] The fine-grained mask of the building with the second highest confidence score is compared with the fine-grained mask of the building in the first target cluster by cross-union ratio (CUI). If the CUI is greater than the threshold, it is classified into the first target cluster; if the CUI is less than the threshold, it is classified into the second target cluster.

[0117] The results of confidence scores are selected in descending order for intersection-union calculation until all results are traversed to determine the entire target cluster.

[0118] The building fine-grained edge extraction and classification device provided in this application has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the embodiment of the building fine-grained edge extraction and classification device can be referred to the corresponding content in the aforementioned building fine-grained edge extraction and classification method embodiment.

[0119] This application also provides an electronic device, such as... Figure 3 The diagram shows the structure of the electronic device 100, which includes a processor 31 and a memory 30. The memory 30 stores computer-executable instructions that can be executed by the processor 31. The processor 31 executes the computer-executable instructions to implement any of the above-mentioned fine-grained edge extraction and classification methods for buildings.

[0120] exist Figure 3 In the illustrated embodiment, the electronic device further includes a bus 32 and a communication interface 33, wherein the processor 31, the communication interface 33, and the memory 30 are connected via the bus 32.

[0121] The memory 30 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 33 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 32 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus 32 can be divided into an address bus, a data bus, and a control bus. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0122] Processor 31 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 31 or by software instructions. Processor 31 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 31 reads the information in the memory and, in conjunction with its hardware, completes the steps of the building fine-grained edge extraction and classification method of the aforementioned embodiment.

[0123] This application also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-mentioned fine-grained edge extraction and classification method for buildings. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0124] The computer program product of the building fine-grained edge extraction and classification method, apparatus, device and medium provided in the embodiments of this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0125] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0126] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] In the description of this application, it should be noted that the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0128] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method of building fine-grained edge extraction and classification, characterized by, The method comprises: acquiring a building remote sensing image, performing size cropping on the building remote sensing image, and constructing a training sample set based on the cropped building remote sensing image; the training sample set comprises a training set and a test set, and the building roof profiles in the training set are instantiated and labeled according to building types; inputting the training set into a plurality of preselected instance segmentation networks respectively, performing data enhancement on the training set by using a category dynamic balance sampling enhancement method, training the instance segmentation networks by using the data-enhanced training set respectively, obtaining a first building identification model corresponding to each instance segmentation network, inputting the test set into the corresponding first building identification model, and obtaining a corresponding first identification result output by each first building identification model; expanding the training set by using the first identification result as the label of the test set, optimizing and training each first building identification model by using the expanded training set, obtaining a second building identification model corresponding to each first building identification model, inputting the test set into each second building identification model respectively, and obtaining a corresponding second identification result output by each second building identification model; performing fine-grained mask fusion processing on the corresponding second identification result output by each second building identification model to obtain a target identification result; wherein the target identification result comprises a building category, a confidence score, a building detection frame and a building fine-grained mask.

2. The building fine-grained edge extraction and classification method of claim 1, wherein, The training set is subjected to data enhancement by using a category dynamic balance sampling enhancement method, which comprises: calculating an instance proportion corresponding to each building category in the training set; determining a category enhancement probability of each category according to the instance proportion and a preset balance rate; determining a picture sampling probability according to the building categories contained in the training set; based on the picture sampling probability, sequentially selecting a source picture and a target picture in the training set, cropping a target object in the source picture, and performing data enhancement processing, wherein the data enhancement processing at least comprises random scaling, random rotation and random blurring; based on the enhancement probability, randomly selecting a preset number of target objects to paste into the target picture to obtain a data-enhanced training set.

3. The building fine-grained edge extraction and classification method of claim 2, wherein, The instance proportion corresponding to each building category in the training set is calculated by using the following formula: wherein, is the instance proportion; c is the building category, and n is the number of categories; refers to the number of instances under the corresponding building category.

4. The building fine-grained edge extraction and classification method of claim 2, wherein, determining a picture sampling probability according to the building categories contained in the training set, which comprises: when the number of target objects in the pictures contained in the training set is 0, the picture I is not sampled; when the number of target object categories in the pictures contained in the training set is 1, the category enhancement probability is determined as the picture sampling probability corresponding to the picture; when the number of target object categories in the pictures contained in the training set is greater than 1, the maximum value of the category enhancement probabilities of the target categories contained in the picture is determined as the picture sampling probability corresponding to the picture.

5. The building fine-grained edge extraction and classification method of claim 1, wherein, optimizing and training each first building identification model by using the expanded training set to obtain a second building identification model corresponding to each first building identification model, which comprises: The learning rate is reduced by a preset multiple and the feature extraction network parameters are fixed for the optimization training of each first building identification model through the expanded training set.

6. The building fine-grained edge extraction and classification method of claim 2, wherein, The corresponding second identification results output by each second building identification model are subjected to fine-grained mask fusion processing to obtain a target identification result, including: The corresponding second identification results output by each second building identification model are divided according to building categories, and the results in each building category are clustered based on a building fine-grained mask to obtain a target cluster corresponding to each building category; The masks in each target cluster are fused, and their confidence is recalculated to obtain a new building fine-grained mask and a corresponding confidence score; The maximum bounding rectangle of the building fine-grained mask is determined as a building detection box; After all the target clusters and categories are traversed, the target identification result is obtained.

7. The building fine-grained edge extraction and classification method of claim 6, wherein, The second identification result includes a building category, a confidence score, a building detection box and a building fine-grained mask; The corresponding second identification results output by each second building identification model are divided according to building categories, and the results in each building category are clustered based on a building fine-grained mask to obtain a target cluster corresponding to each building category, including: The corresponding second identification results output by each second building identification model are divided according to building categories; For each building category, the results are sorted in descending order according to the confidence scores, and the result with the highest confidence score is determined as a first target cluster; The second building fine-grained mask is calculated with the corresponding building fine-grained mask in the first target cluster, and if the intersection over union is greater than a threshold, it is classified into the first target cluster, and if the intersection over union is less than the threshold, a second target cluster is obtained. The results with the confidence scores are selected in descending order, and the intersection over union is calculated until all the target clusters are determined.

8. A building fine-grained edge extraction and classification apparatus, characterized by, The device includes: A sample construction module configured to obtain a building remote sensing image, crop the size of the building remote sensing image, and construct a training sample set based on the cropped building remote sensing image; the training sample set includes a training set and a test set, and the building roof outlines in the training set are instance-labeled according to building types; A first identification module configured to input the training set into a plurality of instance segmentation networks selected in advance, perform data enhancement on the training set by using a category dynamic balance sampling enhancement method, train the instance segmentation networks through the data-enhanced training set, obtain a first building identification model corresponding to each instance segmentation network, input the test set into the corresponding first building identification model, and obtain a corresponding first identification result output by each first building identification model; The second identification module is configured to expand the training set by taking the first identification result as a label of the test set, to perform optimized training on each first building identification model by using the expanded training set, to obtain a second building identification model corresponding to each first building identification model, and to input the test set into each second building identification model to obtain a corresponding second identification result output by each second building identification model. The fusion module is configured to perform fine-grained mask fusion processing on the corresponding second identification result output by each second building identification model to obtain a target identification result, wherein the target identification result includes a building category, a confidence score, a building detection frame, and a building fine-grained mask.

9. An electronic device, comprising: The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when invoked and executed by the processor, cause the processor to implement the building fine-grained edge extraction and classification method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when invoked and executed by the processor, cause the processor to implement the building fine-grained edge extraction and classification method in any one of claims 1 to 7.

Images