Building semantic segmentation sample set error sample elimination method based on confidence learning
Through a confidence learning-based method, using deep learning image recognition residual network and K-fold cross-validation, the error samples in the semantic segmentation sample set are automatically eliminated, which solves the problem of low sample set accuracy and efficiency in the existing technology, and realizes the High precision and efficient sample set production.
Patent Information
- Application Number
- CN202410983697.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The existing technology is difficult to efficiently and automatically produce a massive high-precision building semantic segmentation sample set, and there is a problem of error samples, resulting in a decrease in the generalization ability of deep learning models.
Using a confidence learning method, a deep learning image recognition residual network model is constructed by extracting the feature images of the semantic segmentation samples of the building, combining K-fold cross-validation and confidence threshold method to automatically remove error samples, and improve the accuracy of the sample set. and efficiency.
It realizes efficient and automated semantic segmentation sample set production of massive buildings, improves the accuracy of the sample set and the generalization ability of the model, and reduces the cost of manual labeling and screening.
Smart Images

Figure CN118918416A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of building recognition, and in particular relates to a method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning. Background Art
[0002] As one of the important surface cover elements, accurate building extraction plays an important role in urban planning, population estimation, economic activity distribution, disaster reporting, illegal construction monitoring, etc. With the combination of remote sensing big data and artificial intelligence, especially the successful application of deep learning semantic segmentation technology to remote sensing image interpretation, accurate extraction of large-scale buildings has become possible. However, remote sensing image building interpretation based on deep learning requires a large number of high-precision sample sets for training semantic segmentation models.
[0003] Traditional remote sensing intelligent high-precision building interpretation sample sets rely on manual visual judgment and annotation to outline the precise vector boundaries of buildings, and then cut and produce samples after rasterization. This method is time-consuming, labor-intensive, and inefficient, and cannot meet the needs of remote sensing big data analysis and intelligent information extraction. Historical building result vector data and crowd-sourced open map data provide rich clue data for the production of building semantic segmentation sample sets. However, due to the inaccurate correspondence between the production time of vector result data and the imaging time of remote sensing images, and the existence of missing and mislabeled vector result data, if the sample set is directly automatically cut and produced, the accuracy of the sample set is low and cannot be used to train semantic segmentation models.
[0004] Moreover, the semantic segmentation of remote sensing images is a dense prediction task, and the training samples have high pixel-level accuracy requirements. In addition, in actual engineering projects, high-precision building semantic segmentation sample sets are mainly produced manually, which consumes a lot of manpower and material resources and has a high production cost. Errors and noise problems are common in semantic segmentation samples. Due to data overfitting, deep learning networks are weak in solving such problems, resulting in a decrease in generalization ability. Therefore, eliminating erroneous samples is an important means to improve model accuracy and generalization.
[0005] In response to the above problems, how to make full use of historical building vector data and crowd-source open map data, automatically cut and produce massive semantic segmentation sample sets, eliminate erroneous samples in the sample sets, obtain large-scale high-precision building semantic segmentation sample sets, and provide data support for training and building high-performance remote sensing intelligent building interpretation models is an issue that needs to be solved urgently. Summary of the invention
[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for eliminating erroneous samples in a building semantic segmentation sample set based on confidence learning. The method utilizes confidence learning to eliminate erroneous samples in a building semantic segmentation sample set, and can be applied to the fields of large-scale high-precision building semantic segmentation sample set production, building remote sensing intelligent interpretation, etc.
[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:
[0008] A method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning comprises the following steps:
[0009] Step 1: Cut the historical building result vector and high-resolution image into S initial building semantic segmentation samples, where the initial building semantic segmentation samples include image patches and label patches;
[0010] Step 2: extracting each sample image feature map of the initial building semantic segmentation sample, wherein the sample image feature map at least includes a building existence index feature image, a building morphology feature index image, and a building geometry heat map feature index image;
[0011] Step 3, taking a single pixel neighborhood in each of the initial building semantic segmentation samples as a processing unit, converting the initial building semantic segmentation samples into neighborhood image block recognition samples, each of the neighborhood image block recognition samples including a neighborhood image block and a center pixel label;
[0012] Step 4. Construct a confidence learning model with the deep learning image recognition residual network as the base classifier; the deep learning image recognition residual network includes multiple residual module layers, multiple maximum pooling layers, 1 fully connected layer and 1 softmax layer. The number of residual module layers is determined according to the amount of samples to be processed. When the sample size is large, more residual module layers are required. The residual module layer is used to extract features, and the fully connected layer and softmax layer are used for category determination. Each residual module layer contains 2 weighted convolution layers and 2 Relu activation functions.
[0013] Step 5: The neighborhood image block recognition samples are subjected to K-fold cross validation to estimate the joint distribution of noise labels and true labels;
[0014] Step 6: Use the joint distribution of the noise label and the true label to find the wrong label, and estimate the wrong samples in the initial building semantic segmentation sample set according to the confidence threshold method based on the proportion of the wrong label in the initial building semantic segmentation sample.
[0015] Furthermore, the method further includes step 7: evaluating the sample elimination method according to the precision and recall of the erroneous samples.
[0016] Furthermore, the precision and recall of the erroneous samples evaluate the sample elimination method, specifically including:
[0017] Perform manual visual judgment on the S initial building semantic segmentation samples in step 1 to find out the wrong samples and the correct samples as the reference truth value for the accuracy evaluation of this method. Compare the wrong samples in the initial building semantic segmentation sample set found in step 6 with the reference truth value to calculate the recall rate and precision rate. The recall rate Recall calculation formula is:
[0018]
[0019] Among them, ManEorry represents the number of samples found to be erroneous semantic segmentation by manual visual inspection, and TP represents the number of samples found to be erroneous semantic segmentation by both the erroneous sample elimination method of the building semantic segmentation sample set and manual visual inspection.
[0020] The precision calculation formula is:
[0021]
[0022] Among them, CleanEorry represents the number of erroneous semantic segmentation samples found by the erroneous sample elimination method of the semantic segmentation sample set of this building.
[0023] Furthermore, the step 1 cuts the historical building achievement vector and the high-resolution image into S initial building semantic segmentation samples, and the initial building semantic segmentation samples include image patches and label patches, including:
[0024] Step 1.1, obtain high-resolution image data and building vector data of the corresponding area, perform registration and geometric precision correction, and convert the vector data into raster data with the same spatial resolution as the image;
[0025] Step 1.2: Use the raster data as the true value of the label, and compare the image and the true value of the label. Use a sliding window of m*m pixel size to move up, down, left, and right in m pixel steps to batch cut the image and the true value of the label. The size of m is generally a multiple of 512. Obtain semantic segmentation samples including image patches and label patches, and delete samples in which all pixels in the label patch are background. Obtain S initial building semantic segmentation samples, each of which includes an image patch of m*m pixel size and a label patch of m*m pixel size.
[0026] Furthermore, extracting the sample image feature map of each of the initial building semantic segmentation samples in step 2 includes:
[0027] Based on the contrast texture features extracted from the gray-level co-occurrence matrix, a building existence index BPindex is constructed, and a building existence index feature image with building texture features is extracted from the image patch of the initial building semantic segmentation sample;
[0028] Based on morphological white top-hat transform and black top-hat transform, a building morphological feature index BMFindex is constructed, and a building morphological feature index image with building morphological features is extracted from the image patch of the initial building semantic segmentation sample;
[0029] Corner point features are detected based on the Shi-Tomasi operator, and straight line segment features are extracted using the EDLines algorithm. A building geometric heat map feature index BGHotindex with mutual constraints between corner points and straight line segments is constructed. A building geometric heat map feature index image with geometric features of building corner points and edge lines is extracted from the image patch of the initial building semantic segmentation sample.
[0030] Furthermore, extracting the sample image feature map of each of the initial building semantic segmentation samples in step 2 specifically includes:
[0031] Step 2.1, for the cropped m*m pixel size building semantic segmentation sample image patch, take the average of the three bands, linearly reduce the gray level to LG, LG can be 32, 64, and use the 5*5 neighborhood size as the sliding window to traverse all pixels. The outermost circle pixels of the semantic segmentation sample image need to be copied and expanded outward twice, and the gray level co-occurrence matrix GLCM of 12 displacement vectors is calculated in the neighborhood sliding window. The contrast texture eigenvalue is calculated based on each gray level co-occurrence matrix GLCM;
[0032] Step 2.2: Take the maximum value of the 12 contrast eigenvalues of all displacement vectors as the building presence index BPindex value of the center pixel in the neighborhood sliding window, as shown in the following formula:
[0033] BPindex=max(window=5,V d ,CON d )
[0034] Among them, max means taking the maximum value operation, window=5 means the size of the domain sliding window is 5*5 pixels, V d is the displacement vector used to calculate the gray-level co-occurrence matrix GLCM, and the value of d ranges from 1 to 12, representing a total of 12 displacement vectors; CON d V d Corresponding to the calculated contrast texture eigenvalue, the building presence index BPindex is all CON d The maximum value of CON d The calculation formula is:
[0035]
[0036] Where row, col are the values of discrete gray levels in the row and column directions in GLCM, ranging from 1 to 64, and P(row, col) is the value corresponding to the row and column col in GLCM;
[0037] Step 2.3: For the cropped m*m pixel size building semantic segmentation sample image patch, take the average value of the three bands, perform morphological white hat transformation and black hat transformation, and take the average value of the difference between the white hat transformation and the black hat transformation of multiple line structure elements as the building morphological feature index BMFindex. The calculation formula is as follows:
[0038]
[0039] Represents a linear structure element with size and direction, is the total number of sizes and directions of line structure elements, with values ranging from 0 to L, directions ranging from 0 to F, and F values depending on the specific resolution of the image used, where the calculation formula for the white hat transform and is:
[0040]
[0041] open(A,line l,f ) indicates the use of line structure element line l,f Perform an open operation on the image A after taking the average value, close(A, line l,f ) indicates the use of line structure element line l,f Perform a closing operation on the image A after taking the average value;
[0042] Step 2.4: For the cropped m*m pixel size building semantic segmentation sample image patch, the three bands are averaged to convert it into a single-band grayscale image, and the Shi-Tomasi algorithm is used to extract the corner points, and the EDLines line detection algorithm is used to extract the straight line segments on the image. Then, according to the distance and direction relationship between the straight line segments, the right-angle features on the image are extracted. Then, a variety of geometric feature self-inspection and mutual inspection screening mechanisms are designed to obtain the corner points, straight line segments and right-angle geometric features of the building, and the natural neighbor interpolation method is used to calculate the building geometric heat map feature index BGHotindex of the sample image for the high-quality building geometric features after screening.
[0043] Furthermore, the step 2.4 specifically includes:
[0044] Step 2.4.1, use Shi-Tomasi algorithm to detect corner points, constrain the minimum distance between two feature points and the local maximum condition to detect the corner points on the sample image;
[0045]
[0046] Where I(x, y) is the image gray level of the local window, I(x + u, y + v) is the image gray level after translation, ω(x, y) is the window function, the window size is WS, u and v are the moving distances of the window in the x and y directions. In corner detection, the value of E(u, v) is maximized. After the first-order Taylor expansion, I(x + u, y + v) is expressed as I(x, y) + I x u + I y v, where I x and I y are the derivatives along the x and y directions, calculated using the sobel operator. E(u, v) is expressed as:
[0047]
[0048] Let The autocorrelation matrix Mat determines the value of E(u, v). Use Mat to find the corners. Mat is a quadratic function of I x and I y The eigenvalues of Mat are γ1 and γ2;
[0049] The stability of the corners is related to the smaller eigenvalue of the matrix Mat. Use the smallest eigenvalue as the score SC:
[0050] SC = min(γ1, γ2)
[0051] The result of the Shi-Tomasi corner detection is a grayscale image with these scores SC. Set a threshold TS. When SC > TS, the corresponding pixels are the corners;
[0052] Step 2.4.2: Take the average of multiple bands of the initial building semantic segmentation sample image and convert it into a single-band grayscale image. Use the ED edge detector of the EDLines algorithm to generate a set of clean and continuous edge pixel chains for the single-band grayscale image. Then, extract straight lines from the edge segments according to the least squares principle, and use the Helmholtz principle to eliminate the detected incorrect straight line segments;
[0053] Step 2.4.3: For the straight line segments detected by the EDLines algorithm, when there is an intersection between two straight line segments and the direction Asp satisfies 80° < Asp ≤ 90°, extract the right-angle features on the sample image;
[0054] Step 2.4.4: Design a screening mechanism for self-checking and mutual-checking of the three geometric features of corners, straight line segments, and right angles to obtain the geometric features of corners, straight line segments, and right angles of buildings on the sample image:
[0055] A. Perform self-check on Shi-Tomasi corner points. If corner point P j The surrounding area D j If there are more than σ corner points in the memory, then the corner point P j Keep it, otherwise delete it;
[0056]
[0057] in is the current set of corner points, is the number of corner points around the current corner point, σ is the number threshold;
[0058] B. Perform length self-check on the straight line segments detected by EDLines and delete those whose length is less than the threshold L. T The straight line segment, L T Set according to the image resolution and the minimum size of buildings in the image;
[0059] C. Check each other between the diagonal point and the straight line segment. J If there is no straight line segment within the radius, the corner point is a weak corner point, and the D of the straight line segment L If there is no corner point within the radius, the straight line segment is a pseudo straight line segment, and the weak corner points and pseudo straight line segments are deleted;
[0060] D. The foot of a right angle detected based on EDLines straight line segment is in D Z There must be at least one corner point within the radius, otherwise the right angle will be deleted;
[0061] Step 2.4.5: After screening, the high-quality corner points, straight line segments, and right-angle geometric features are converted into binary images, with the pixel values of corner points, straight line segments, and right angles being 255 and the background pixel value being 0. The building geometric heat map feature index BGHotindex of the sample image is then calculated using the natural neighbor interpolation method.
[0062]
[0063] Among them BGHotindex (a,b) is the geometric heat value of the building at coordinates a and b, For the The true value of the binary image of the corner points involved in the interpolation, for The corresponding weights, For the The true value of the binary graph of the straight line segments involved in the interpolation, for The corresponding weights, For the The true value of the rectangular binary graph involved in the interpolation, for The corresponding weights, HJ, HL, and HZ, are the number of corner points in the corner binary map, the number of straight line segments in the straight line binary map, and the number of right angles in the right angle binary map. The truth value V and the weight W are determined by the Thiessen polygons;
[0064]
[0065] Where Area(a,b) is the area of the new unit centered at (a,b), and the new unit is the newly established Thiessen polygon. It is the area of the intersection between the new cell centered at (a,b) and the old cell centered at (a,b), where the old cell is the original Thiessen polygon.
[0066] Furthermore, in step 3, the single pixel neighborhood in each of the initial building semantic segmentation samples is used as a processing unit to convert the initial building semantic segmentation samples into neighborhood image block recognition samples, each of which includes a neighborhood image block and a center pixel label, including:
[0067] Step 3.1, for each building semantic segmentation sample, use the n*n neighborhood size as the sliding window. The size of n is determined by the image resolution, and is generally 3, 5, 7, 9, 11, 13, or 15. The sliding window moves from left to right and from top to bottom, and the distance of each movement is q.
[0068] Step 3.2, superimpose the original sample image piece, the building existence index feature image, the building morphological feature index feature image and the building geometric heat map feature index feature image to obtain six bands of data, and cut the superimposed six bands of data with an n*n neighborhood size sliding window to obtain a neighborhood image block, which is composed of the original image block, the building existence index feature image, the building morphological feature index feature image and the building geometric heat map feature index feature image superimposed. The label of the neighborhood image block sample is the category label of the pixel at the center position of the neighborhood sliding window. One building semantic segmentation sample can generate R neighborhood image block recognition samples;
[0069] Step 3.3, all initial building semantic segmentation samples are converted into neighborhood image block recognition samples. After the conversion, R*S neighborhood image block recognition samples are generated. The first R neighborhood image block recognition samples correspond to the first one in the initial semantic segmentation sample set, and the z*Rth to (z+1)*Rth neighborhood image block recognition samples correspond to the z+1th in the initial semantic segmentation sample set. The value range of z is 0 to S-1.
[0070] Furthermore, in step 5, the neighborhood image block recognition samples use a K-fold cross validation method to estimate the joint distribution of noise labels and true labels, including:
[0071] Define the noise label as That is, all neighborhood image blocks identify sample labels, which contain both correct and incorrect labels; the true label is defined as lb * , but in fact the true label will not be obtained. The true label is estimated through K-fold cross validation, K = 2, 3, 4, 5, and the total number of categories is CL.
[0072] Step 5.1: Perform K-fold cross validation on all neighborhood image block recognition sample data, α represents the sample, β represents the category, calculate the probability P[α][β] of the αth sample in the βth category, and then calculate the average probability t[β] under each category β as the confidence threshold. Finally, for the neighborhood image block recognition sample α, its true label lb * is the maximum probability argmax in the βth category β The labels corresponding to P[α][β], and P[α][β]>t[β];
[0073] Step 5.2: Calculate the count matrix Similar to a confusion matrix;
[0074] Step 5.3, calibrate the count matrix so that the total count is the same as the total number of sample labels identified in all neighborhood image blocks. The calibrated count matrix The calculation formula is:
[0075]
[0076] in Identify sample labels for all neighborhood patches The total number of samples, is the count matrix after calibration;
[0077] Step 5.4: Estimating Noise Labels and the true label lb * The joint distribution of The joint distribution The calculation formula is:
[0078]
[0079] Step 6: Use the joint distribution of the noise label and the true label to find the wrong label, and estimate the wrong samples in the initial building semantic segmentation sample set according to the confidence threshold method based on the proportion of the wrong label in the initial building semantic segmentation sample.
[0080] Step 6.1, based on the joint distribution of noise labels and true labels At the same time, Prune by Class (PBC) and Prune by Noise Rate (PBNR) are used to filter out the error samples.
[0081] Step 6.2: For the R*S neighborhood image block recognition samples, determine whether each sample is an error sample through the step 6.1. If the proportion of error samples in the z*Rth to (z+1)*Rth neighborhood image block recognition samples exceeds the set threshold θ, it indicates that the z+1th sample in the initial building semantic segmentation sample set is an error sample.
[0082] The remarkable effects of the present invention are:
[0083] (1) The present invention performs sliding window cropping on building vector data and remote sensing images to obtain a large number of initial building semantic segmentation samples, and then uses the confidence learning algorithm to establish a model to achieve the search and elimination of erroneous samples, thereby improving the efficiency and accuracy of large-scale building semantic segmentation sample set production. This effectively solves the problem that the existing production of massive high-precision remote sensing semantic segmentation sample sets relies on manual labeling and manual screening and refinement. Compared with the production of high-precision samples by manual visual labeling, the production efficiency and accuracy of massive building semantic segmentation samples are greatly improved.
[0084] (2) A deep learning image recognition residual network is adopted as the base classifier of confidence learning. The pixel neighborhood image block is taken as the unit. The neighborhood image block contains the building texture features, morphological features and geometric features extracted by BPindex, BMFindex and BGHotindex. The initial building semantic segmentation samples are converted into image block recognition samples that can be processed by confidence learning. Then, the accuracy of the building semantic segmentation samples is determined by finding the erroneous neighborhood image block recognition samples and the threshold method. The method of the present invention is scientific, reasonable and feasible.
[0085] (3) The method of the present invention has reference significance for the production of massive high-precision remote sensing scene recognition samples, target detection samples, and remote sensing change detection samples. In addition, the method of the present invention can also be applied to the production of large-scale high-precision remote sensing semantic segmentation sample sets such as medical image segmentation and natural image segmentation, and the field of intelligent interpretation of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS
[0086] Figure 1 It is a schematic diagram of the structure of the present invention;
[0087] Figure 2 It is the high-resolution remote sensing image and Open Street Map building vector data of the present invention;
[0088] Figure 3 is the initial building semantic segmentation sample set of the present invention;
[0089] Figure 4 These are 12 displacement vector diagrams designed by the present invention;
[0090] Figure 5 This is a structural diagram of the residual network for deep learning image recognition of the present invention;
[0091] Figure 6 This is a result graph of some erroneous building semantic segmentation samples found by the present invention. DETAILED DESCRIPTION
[0092] The specific implementation manner and working principle of the present invention are further described in detail below with reference to the accompanying drawings.
[0093] See also Figures 1 to 6 As shown in FIG. 1 , a method for removing erroneous samples from a building segmentation sample set based on confidence learning is shown in FIG. The specific steps are as follows:
[0094] Step 1: Cut the historical building result vector and high-resolution image into S initial building semantic segmentation samples, where the initial building semantic segmentation samples include image patches and label patches. The steps include:
[0095] Step 1.1, obtain high-resolution image data and building vector data of the corresponding area, perform registration and geometric precision correction, and convert the vector data into raster data with the same spatial resolution as the image;
[0096] Step 1.2: Use the raster data as the true value of the label, and compare the image and the true value of the label. Use a sliding window of m*m pixel size to move up, down, left, and right in m pixel steps to batch cut the image and the true value of the label. The size of m is generally a multiple of 512. Obtain semantic segmentation samples including image patches and label patches, and delete samples in which all pixels in the label patch are background. Use a 512*512 sliding window to cut to obtain a large number of initial building semantic segmentation samples, and obtain S initial building semantic segmentation samples. Each initial building semantic segmentation sample includes an image patch of m*m pixel size and a label patch of m*m pixel size.
[0097] In this specific embodiment: the high-resolution image used is 0.5-meter resolution, located in a certain city, and the imaging time is January 2022. The building vectors at the corresponding time are derived from Open Street Map, and the vectors and images are geometrically corrected to convert the building vectors into 0.5-meter resolution raster data. Figure 2 As shown, the image covers an area of 146.93 square kilometers, and there are obvious omissions and mislabeling in the building vectors.
[0098] The image and building raster data are cropped with a 512*512 sliding window, and the sliding window moving step is 512 pixels, that is, adjacent sliding windows do not overlap, and each sliding window cannot be full of background pixels. After cropping, 2242 initial building semantic segmentation samples are obtained, such as Figure 3 Each sample shown includes a 512*512 pixel image patch and a 512*512 pixel building binary grid label patch. Some samples have obvious errors.
[0099] Step 2: extracting the sample image feature maps of the initial building semantic segmentation samples, wherein the sample image feature maps at least include a building existence index feature image, a building morphology feature index image, and a building geometry heat map feature index image. Including:
[0100] Based on the contrast texture features extracted from the gray-level co-occurrence matrix, a building existence index BPindex is constructed, and a building existence index feature image with building texture features is extracted from the image patch of the initial building semantic segmentation sample;
[0101] Based on morphological white top-hat transform and black top-hat transform, a building morphological feature index BMFindex is constructed, and a building morphological feature index image with building morphological features is extracted from the image patch of the initial building semantic segmentation sample;
[0102] Corner point features are detected based on the Shi-Tomasi operator, and straight line segment features are extracted using the EDLines algorithm. A building geometric heat map feature index BGHotindex with mutual constraints between corner points and straight line segments is constructed. A building geometric heat map feature index image with geometric features of building corner points and edge lines is extracted from the image patch of the initial building semantic segmentation sample.
[0103] The calculation methods of the BPindex, BMFindex and BGHotindex are as follows:
[0104] Step 2.1, for the cropped 512*512 pixel building semantic segmentation sample image patch, take the average of the three bands, linearly reduce the grayscale to LG, LG can be 32, 64, in this specific example: linearly reduce the grayscale to 64, traverse all pixels with a 5*5 neighborhood size as the sliding window, the outermost circle pixels of the semantic segmentation sample image need to be copied and expanded outward twice, and the gray level co-occurrence matrix (GLCM) of 12 displacement vectors is calculated in the neighborhood sliding window, and the contrast texture eigenvalue can be calculated based on each GLCM.
[0105] Step 2.2: Take the maximum value of the 12 contrast eigenvalues of all displacement vectors as the BPindex value of the center pixel in the neighborhood sliding window. As shown in the following formula:
[0106] BPindex=max(window=5,V d ,CON d )
[0107] Among them, max means taking the maximum value operation, window=5 means the size of the domain sliding window is 5*5 pixels, V d is the displacement vector used to calculate GLCM. The value of d ranges from 1 to 12, representing a total of 12 displacement vectors. The step size and direction of each displacement vector are detailed in Figure 4 ;CON d V d Corresponding to the calculated contrast texture feature value, BPindex is all CON d The maximum value of CON d The calculation formula is:
[0108]
[0109] Where row, col are the values of discrete gray levels in the row and column directions in GLCM, ranging from 1 to 64, and P(row, col) is the value corresponding to the row and col columns in GLCM.
[0110] Step 2.3: For the cropped 512*512 pixel building semantic segmentation sample image, take the average of the three bands, perform morphological white-hat transformation (whitetophat, enhance bright targets such as buildings) and black-hat transformation (blacktophat, enhance dark targets such as vegetation and water), and take the average of the difference between white-tophat and black-tophat of multiple line structure elements as BMFindex. The calculation formula is as follows:
[0111]
[0112] line l,fRepresents a line structure element with a size of l and a direction of f. L and F are the total number of sizes and directions of the line structure element, with values ranging from 0 to L and values ranging from 0 to F. The resolution of the high-resolution image used in this example is 0.5 meters. Considering that the length of the building on the 0.5-meter image ranges from about 20 to 400 pixels, l∈(20, 40, 60, 80, 100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300, 320, 340, 360, 380, 400), f∈(0 degrees, 45 degrees, 90 degrees, 135 degrees), that is, the total number of sizes of the line structure element L = 20, and the total number of directions F = 4 (horizontal, vertical, and two oblique directions). The calculation formulas for whitetophat and blacktophat are:
[0113]
[0114] open(A,line l,f ) indicates the use of line structure element line l,f Perform an open operation on the image A after taking the average value, close(A, line l,f ) indicates the use of line structure element line l,f Perform a closing operation on the image A after taking the average value.
[0115] Step 2.4: For the cropped 512*512 pixel building semantic segmentation sample image, the three bands are averaged to convert into a single-band grayscale image. The Shi-Tomasi algorithm is used to extract corner points, and the EDLines line detection algorithm is used to extract straight line segments on the image. Then, according to the distance and direction relationship between the straight line segments, the right-angle features on the image are extracted. Then, a variety of geometric feature self-checking and mutual-checking screening mechanisms are designed to obtain the corner points, straight line segments and right-angle geometric features of the building. The natural neighbor interpolation method is used to calculate the BGHotindex of the sample image for the high-quality geometric features after screening. The specific process is as follows:
[0116] Step 2.4.1, use Shi-Tomasi algorithm to detect corner points, constrain the minimum distance between two feature points and the local maximum condition to detect the corner points on the sample image;
[0117]
[0118] Where I(x, y) is the image grayscale of the local window, I(x + u, y + v) is the grayscale of the translated image, ω(x, y) is the window function. In this example, the image resolution is 0.5 meters, so the window size WS = 5 pixels. u and v are the moving distances of the window in the x and y directions. In corner detection, the value of E(u, v) is maximized. After the first-order Taylor expansion of I(x + u, y + v), it is expressed as I(x, y) + I x u + I y v, where I x and I y are the derivatives along the x and y directions, calculated using the sobel operator. Further, E(u, v) can be expressed as:
[0119]
[0120] Let The autocorrelation matrix Mat determines the value of E(u, v). Use Mat to find the corners. Mat is a quadratic function of I x and I y The eigenvalues of Mat are γ1 and γ2.
[0121] The stability of the corners is related to the smaller eigenvalue of the matrix Mat. Use the smallest eigenvalue as the score SC:
[0122] SC = min(γ1, γ2)
[0123] The result of the Shi - Tomasi corner detection is a grayscale image with these scores SC. Set a threshold TS = 0.4. When SC > TS, the corresponding pixels are the corners.
[0124] Step 2.4.2: For the small piece of the building semantic segmentation sample image with a size of 512 * 512 pixels after cropping, take the average of the three bands and convert it into a single - band grayscale image. Use the ED (edge drawing) edge detector of the EDLines algorithm to generate a set of clean and continuous edge pixel chains for the single - band grayscale image. Then, extract straight lines from the edge segments according to the least - squares principle and eliminate the detected wrong straight line segments using the Helmholtz principle;
[0125] Step 2.4.3: For the straight line segments detected by the EDLines algorithm, when there is an intersection between two straight line segments and the direction Asp satisfies 80° < Asp ≤ 90°, extract the right - angle feature on the sample image;
[0126] Step 2.4.4: Design a screening mechanism for self - inspection and mutual - inspection of the three geometric features of corners, straight line segments, and right - angles to obtain the geometric features of corners, straight line segments, and right - angles of the building on the sample image:
[0127] A. Perform self-check on Shi-Tomasi corner points, such as corner point P j The surrounding area D j If there are more than σ corner points in the memory, then the corner point P j Keep it, otherwise delete it.
[0128]
[0129] in is the current set of corner points, is the number of corner points around the current corner point, σ is the number threshold, in this example σ=2, D j =200 pixels.
[0130] B. Perform length self-check on the straight line segments detected by EDLines and delete those whose length is less than the threshold L. T In this example, L is set based on the image used and the minimum size of the building. T =20 pixels.
[0131] C. Check each other between the diagonal point and the straight line segment. J If there is no straight line segment within the radius, the corner point is a weak corner point, and the D of the straight line segment L If there is no corner point within the radius, the straight line segment is a pseudo straight line segment. Delete the weak corner points and pseudo straight line segments. In this example, D J = 20 pixels, D L =20 pixels.
[0132] D. The foot of the right angle detected based on the EDLines straight line segment is at D Z There must be at least one corner point within the radius, otherwise the right angle is deleted. In this example, D Z =10 pixels.
[0133] Step 2.4.5: After screening, the high-quality corner points, straight line segments, and right-angle geometric features are converted into binary images. The pixel values of corner points, straight line segments, and right angles are 255, and the background pixel value is 0. Then, the BGHotindex of the sample image is calculated using the natural neighbor interpolation method.
[0134]
[0135] Among them BGHotindex (a,b) is the geometric heat value of the building at coordinates a and b, For the The true value of the binary image of the corner points involved in the interpolation, for The corresponding weights, For the The true value of the binary graph of the straight line segments involved in the interpolation, for The corresponding weights, For the The true value of the rectangular binary graph involved in the interpolation, for The corresponding weights, HJ, HL, and HZ are respectively the number of corner points in the corner point binary map, the number of straight line segments in the straight line segment binary map, and the number of right angles in the right angle binary map. The true value V and the weight W are determined by the Thiessen polygons.
[0136]
[0137] Where Area(a,b) is the area of the new unit centered at (a,b), and the new unit is the newly created Thiessen polygon It is the area of the intersection between the new cell centered at (a,b) and the old cell centered at (a,b), where the old cell is the original Thiessen polygon.
[0138] Step 3: Using a single pixel neighborhood in each of the initial building semantic segmentation samples as a processing unit, the initial building semantic segmentation samples are converted into neighborhood image block recognition samples, each of which includes a neighborhood image block and a center pixel label.
[0139] Step 3.1, for each building semantic segmentation sample, use the n*n neighborhood size as the sliding window. The size of n is determined by the image resolution, and is generally 3, 5, 7, 9, 11, 13, or 15. The sliding window moves from left to right and from top to bottom, and the distance of each movement is q.
[0140] Step 3.2, superimpose the original sample image piece, the building existence index feature image, the building morphological feature index feature image and the building geometric heat map feature index feature image to obtain six bands of data, and cut the superimposed six bands of data with an n*n neighborhood size sliding window to obtain a neighborhood image block, which is composed of the original image block, the building existence index feature image, the building morphological feature index feature image and the building geometric heat map feature index feature image superimposed. The label of the neighborhood image block sample is the category label of the pixel at the center position of the neighborhood sliding window. One building semantic segmentation sample can generate R neighborhood image block recognition samples;
[0141] Step 3.3, all initial building semantic segmentation samples are converted into neighborhood image block recognition samples. After the conversion, R*S neighborhood image block recognition samples are generated. The first R neighborhood image block recognition samples correspond to the first one in the initial semantic segmentation sample set, and the z*Rth to (z+1)*Rth neighborhood image block recognition samples correspond to the z+1th in the initial semantic segmentation sample set. The value range of z is 0 to S-1.
[0142] In this specific embodiment: the neighborhood size n is set to 5, the moving distance of the neighborhood sliding window is 2 pixels, the row number or column number of the center pixel of the neighborhood sliding window is a multiple of 5, the neighborhood sliding window cannot move out of the semantic segmentation sample, the label of the 5*5 neighborhood image block recognition sample is the label of the center pixel of the neighborhood, and finally each building semantic segmentation sample of 512*512 pixel size can be converted to generate 92415 neighborhood image block recognition samples, and all building semantic segmentation samples can generate 207194430 neighborhood image block recognition samples.
[0143] Step 4: Construct a confidence learning model with a deep learning image recognition residual network as the base classifier;
[0144] The deep learning image recognition residual network includes multiple residual module layers, multiple maximum pooling layers, 1 fully connected layer and 1 softmax layer. The number of residual module layers is determined by the number of samples to be processed. When the sample size is large, more residual module layers are required. Each residual module layer contains 2 weighted convolution layers and 2 Relu activation functions. The residual module layer is used to extract features, and the fully connected layer and softmax layer are used for category determination.
[0145] In this specific embodiment, the deep learning image recognition residual network is as follows Figure 4 As shown in the figure, it includes 5 residual module layers, 2 maximum pooling layers, 1 fully connected layer and 1 softmax layer. The residual module layer is used to extract features, and the fully connected layer and softmax layer are used for category determination. Each residual module layer contains 2 weighted convolution layers and 2 Relu activation functions. The network is simpler, has a small amount of calculation, and reduces data redundancy.
[0146] A NVIDIA GPU graphics card (NVIDIA GeForceRTX 2080Ti) is used to train the deep learning image recognition residual network model. The loss function is set to binary cross entropy, the optimizer is stochastic gradient descent (SGD), the learning rate is 0.01, the confidence learning framework library used is cleanlab version 2.0, and the deep learning framework is pytorch1.2.0.
[0147] Step 5: The neighborhood image block recognition samples are subjected to K-fold cross validation to estimate the joint distribution of noise labels and true labels;
[0148] In this specific embodiment: for the 207194430 neighborhood image block recognition samples generated in step 3, a K-fold cross validation method is used to estimate the joint distribution of the noise label and the true label;
[0149] Define the noise label as That is, all neighborhood image blocks identify sample labels, which contain both correct and incorrect labels; the true label is defined as lb* , but in fact the true label will not be obtained. The true label is estimated through K-fold cross validation, K = 2, 3, 4, 5, and the total number of categories is CL.
[0150] Step 5.1: Perform K-fold cross validation on all neighborhood image block recognition sample data, α represents the sample, β represents the category, calculate the probability P[α][β] of the αth sample in the βth category, and then calculate the average probability t[β] under each category β as the confidence threshold. Finally, for the neighborhood image block recognition sample α, its true label lb * is the maximum probability argmax in the βth category β The labels corresponding to P[α][β], and P[α][β]>t[β]; in this specific embodiment: K is set to 3, the total number of categories is CL=2, and the values of β are 1 and 2.
[0151] Step 5.2: Calculate the count matrix Similar to a confusion matrix;
[0152] Step 5.3, calibrate the count matrix so that the total count is the same as the total number of sample labels identified in all neighborhood image blocks. The calibrated count matrix The calculation formula is:
[0153]
[0154] in Identify sample labels for all neighborhood patches The total number of samples, is the count matrix after calibration;
[0155] Step 5.4: Estimating Noise Labels and the true label lb * The joint distribution of The joint distribution The calculation formula is:
[0156]
[0157] Step 6: Use the joint distribution of noise labels and true labels to find wrong labels, and estimate the wrong samples in the initial building semantic segmentation sample set according to the confidence threshold method based on the proportion of wrong labels in the initial building semantic segmentation samples;
[0158] The process of determining the wrong samples in the initial building semantic segmentation samples in step 6 is as follows:
[0159] Step 6.1: Based on the joint distribution of noise labels and true labels At the same time, Prune by Class (PBC) and Prune by Noise Rate (PBNR) are used to filter out the error samples.
[0160] Step 6.2: For the R*S neighborhood image block recognition samples, determine whether each sample is an error sample through the step 6.1. If the proportion of error samples in the z*Rth to (z+1)*Rth neighborhood image block recognition samples exceeds the set threshold θ, it indicates that the z+1th sample in the initial building semantic segmentation sample set is an error sample.
[0161] In this specific embodiment, for 207194430 neighborhood image block recognition samples, whether each sample is an error sample is determined by step 6.1. If the proportion of error samples in the 92415*zth to 92415*(z+1)th neighborhood image block recognition samples exceeds the set threshold value θ=90%, that is, the correct building semantic segmentation samples are required, and the pixel-level accuracy exceeds 90%, then it indicates that the z+1th sample in the initial building semantic segmentation sample set is an error sample, and the value of z is 0 to 2241;
[0162] Through step 6.2, this embodiment finds 532 erroneous building semantic segmentation samples, such as Figure 6 The schematic diagram of the erroneous building semantic segmentation samples found by the present invention shows that the method of the present invention can determine the erroneous semantic segmentation samples by quickly finding the erroneous labels of the neighborhood image blocks through confidence learning.
[0163] Possibly, step 7 is also included: evaluating the sample elimination method according to the precision and recall of the erroneous samples. Specifically including:
[0164] Perform manual visual judgment on the S initial building semantic segmentation samples in step 1 to find out the wrong samples and the correct samples as the reference truth value for the accuracy evaluation of this method. Compare the wrong samples in the initial building semantic segmentation sample set found in step 6 with the reference truth value to calculate the recall rate and precision rate. The recall rate Recall calculation formula is:
[0165]
[0166] Among them, ManEorry represents the number of samples found to be erroneous semantic segmentation by manual visual inspection, and TP represents the number of samples found to be erroneous semantic segmentation by both the erroneous sample elimination method of the building semantic segmentation sample set and manual visual inspection.
[0167] The precision calculation formula is:
[0168]
[0169] Among them, CleanEorry represents the number of erroneous semantic segmentation samples found by the erroneous sample elimination method of the semantic segmentation sample set of this building.
[0170] In this specific embodiment: the number of erroneous semantic segmentation samples found by the erroneous sample elimination method of the building semantic segmentation sample set is 532, and the manual visual kernel finds 604 erroneous samples, of which 498 are found as erroneous semantic segmentation samples by both the erroneous sample elimination method of the building semantic segmentation sample set and the manual visual kernel, so the recall rate is 498 / 604*100%=82.45%, and the precision rate is 498 / 532*100%=93.6%.
[0171] The method of the present invention designs a deep learning image recognition residual network as a base classifier. On the basis of a confidence learning framework, a single-pixel neighborhood image block in a building semantic segmentation sample is used as a processing unit. The color features, texture features, morphological features and geometric features of the building are considered, and a deep learning residual image classification and recognition module is embedded. By estimating the joint distribution of noise labels and potential true labels of single-pixel neighborhood image blocks, and finding and filtering out erroneous samples, the erroneous samples in a building semantic segmentation sample set are accurately, effectively and quickly eliminated, thereby obtaining a large-scale high-precision building semantic segmentation sample set.
[0172] It can be seen that the method of the present invention is capable of finding and eliminating erroneous samples from a building semantic segmentation sample set with noise samples. For the production of large-scale remote sensing intelligent interpretation sets, the method of the present invention has significant advantages over manual visual annotation in producing high-precision samples, and greatly improves the production efficiency and accuracy of massive remote sensing semantic segmentation samples.
[0173] The technical solution provided by the present invention is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
Claims
1. A method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning, characterized by: The steps include: Step 1: Cut the historical building result vector and high-resolution image into S initial building semantic segmentation samples, where the initial building semantic segmentation samples include image patches and label patches; Step 2: extracting each sample image feature map of the initial building semantic segmentation sample, wherein the sample image feature map at least includes a building existence index feature image, a building morphology feature index image, and a building geometry heat map feature index image; Step 3: using a single pixel neighborhood in each of the initial building semantic segmentation samples as a processing unit, converting the initial building semantic segmentation samples into neighborhood image block recognition samples, each of the neighborhood image block recognition samples including a neighborhood image block and a center pixel label; Step 4: Construct a confidence learning model with a deep learning image recognition residual network as the base classifier; Step 5: The neighborhood image block recognition samples are subjected to K-fold cross validation to estimate the joint distribution of noise labels and true labels; Step 6: Use the joint distribution of the noise label and the true label to find the wrong label, and estimate the wrong samples in the initial building semantic segmentation sample set according to the confidence threshold method based on the proportion of the wrong label in the initial building semantic segmentation sample.
2. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 1, characterized in that: The method further includes step 7: evaluating the sample elimination method according to the precision and recall of the erroneous samples.
3. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 2, characterized in that: The precision and recall of the erroneous samples evaluate the sample elimination method, specifically including: Perform manual visual judgment on the S initial building semantic segmentation samples in step 1 to find out the wrong samples and the correct samples as the reference truth value for the accuracy evaluation of this method. Compare the wrong samples in the initial building semantic segmentation sample set found in step 6 with the reference truth value to calculate the recall rate and precision rate. The recall rate Recall calculation formula is: Among them, ManEorry represents the number of samples found to be erroneous semantic segmentation by manual visual inspection, and TP represents the number of samples found to be erroneous semantic segmentation by both the erroneous sample elimination method of the building semantic segmentation sample set and manual visual inspection. The precision calculation formula is: Among them, CleanEorry represents the number of erroneous semantic segmentation samples found by the erroneous sample elimination method of the semantic segmentation sample set of this building.
4. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 1, characterized in that: The step 1 cuts the historical building result vector and the high-resolution image into S initial building semantic segmentation samples, and the initial building semantic segmentation samples include image patches and label patches, including: Step 1.1, obtain high-resolution image data and building vector data of the corresponding area, perform registration and geometric precision correction, and convert the vector data into raster data with the same spatial resolution as the image; Step 1.2: Use the raster data as the label truth value, compare the image and the label truth value, use a sliding window of m*m pixel size to batch cut the image and the label truth value according to the m pixel step size, obtain semantic segmentation samples containing image patches and label patches, and delete the samples in which all pixels in the label patch are background, and obtain S initial building semantic segmentation samples, each of which contains an image patch of m*m pixel size and a label patch of m*m pixel size.
5. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 1 or 4, characterized in that: The step 2 of extracting the sample image feature map of each of the initial building semantic segmentation samples includes: Based on the contrast texture features extracted from the gray-level co-occurrence matrix, a building existence index BPindex is constructed, and a building existence index feature image with building texture features is extracted from the image patch of the initial building semantic segmentation sample; Based on morphological white top-hat transform and black top-hat transform, a building morphological feature index BMFindex is constructed, and a building morphological feature index image with building morphological features is extracted from the image patch of the initial building semantic segmentation sample; Corner point features are detected based on the Shi-Tomasi operator, and straight line segment features are extracted using the EDLines algorithm. A building geometric heat map feature index BGHotindex with mutual constraints between corner points and straight line segments is constructed. A building geometric heat map feature index image with geometric features of building corner points and edge lines is extracted from the image patch of the initial building semantic segmentation sample.
6. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 5, characterized in that: The step 2 of extracting the sample image feature map of each of the initial building semantic segmentation samples specifically includes: Step 2.1, for the cropped m*m pixel size building semantic segmentation sample image patch, take the average of the three bands, linearly reduce the gray level to LG, traverse all pixels with a 5*5 neighborhood size as a sliding window, calculate the gray level co-occurrence matrix GLCM of 12 displacement vectors in the neighborhood sliding window, and calculate the contrast texture eigenvalue based on each gray level co-occurrence matrix GLCM; Step 2.2: Take the maximum value of the 12 contrast eigenvalues of all displacement vectors as the building presence index BPindex value of the center pixel in the neighborhood sliding window, as shown in the following formula: BPindex=max(window=5,V d ,CON d ) Among them, max means taking the maximum value operation, window=5 means the size of the domain sliding window is 5*5 pixels, V d is the displacement vector used to calculate the gray-level co-occurrence matrix GLCM, and the value of d ranges from 1 to 12, representing a total of 12 displacement vectors; CON d V d Corresponding to the calculated contrast texture eigenvalue, the building presence index BPindex is all CON d The maximum value of CON d The calculation formula is: Where row, col are the values of discrete gray levels in the row and column directions in GLCM, ranging from 1 to LG, and P(row, col) is the value corresponding to the row and column col in GLCM; Step 2.3: For the cropped m*m pixel size building semantic segmentation sample image patch, take the average value of the three bands, perform morphological white hat transformation and black hat transformation, and take the average value of the difference between the white hat transformation and the black hat transformation of multiple line structure elements as the building morphological feature index BMFindex. The calculation formula is as follows: line ,f It represents a linear structure element with a size of l and a direction of f. L and F are the total number of sizes and directions of the line structure elements. l is within 0 to L, and f is a direction within 0 to F. The values of L and F depend on the specific resolution of the image used. The calculation formulas for white hat transform witetophat and black hat transform blacktophat are: open(A,line l,f ) indicates the use of line structure element line l,f Perform an open operation on the image A after taking the average value, close(A, line l,f ) indicates the use of line structure element line l,f Perform a closing operation on the image A after taking the average value; Step 2.4: For the small pieces of building semantic segmentation sample images with a pixel size of m*m after cutting, the average value of the three bands is taken and converted into a single-band grayscale image. The Shi-Tomasi algorithm is used to extract corner points, and the EDLines line detection algorithm is used to extract line segments on the image. Then, according to the distance and direction relationship between the line segments, the right-angle features on the image are extracted. Furthermore, a screening mechanism for self-checking and mutual-checking of various geometric features is designed to obtain the corner points, line segments, and right-angle geometric features of the building. The building geometric heat map feature index BGHotindex of the sample image is calculated using the natural neighbor interpolation method for the high-quality building geometric features after screening.
7. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 6, characterized in that: The specific steps of Step 2.4 are as follows: Step 2.4.1: The Shi-Tomasi algorithm is used to detect corner points, and the minimum distance between two feature points and the local maximum condition are used for constraint to detect the corner points on the sample image. Where I(x,y) is the image grayscale of the local window, I(x+u,y+v) is the image grayscale after translation, ω(x,y) is the window function, the window size is WS, u and v are the distances the window moves in the x and y directions, and the value of E(u,v) is maximized in corner detection. I(x+u,y+v) is expressed as I(x,y)+I after first-order Taylor expansion. x u+I y v, where I x and I y is the derivative along the x and y directions, calculated using the Sobel operator, E(u,v) is expressed as: make The autocorrelation matrix Mat determines the value of E(u,v). Mat is used to find the corner points. Mat is I x and I y The quadratic function of Mat has eigenvalues γ1 and γ2; The stability of the corner points is related to the smaller eigenvalue of the matrix Mat. The smallest eigenvalue is used as the score SC: SC = min(γ1, γ2) The result of the Shi-Tomasi corner detection is a grayscale image with these scores SC. A threshold TS is set, and the pixel corresponding to SC > TS is the corner point. Step 2.4.2: The average value of multiple bands of the initial building semantic segmentation sample image is taken and converted into a single-band grayscale image. The ED edge detector of the EDLines algorithm is used to generate a set of clean and continuous edge pixel chains for the single-band grayscale image. Then, lines are extracted from the edge segments according to the least squares principle, and the wrongly detected line segments are eliminated using the Helmholtz principle. Step 2.4.3: For the line segments detected by the EDLines algorithm, when there is an intersection between two line segments and the direction Asp satisfies 80° < Asp ≤ 90°, the right-angle features on the sample image are extracted. Step 2.4.4: A screening mechanism for self-checking and mutual-checking of the three geometric features of corner points, line segments, and right angles is designed to obtain the corner points, line segments, and right-angle geometric features of the building on the sample image: A. Perform self-check on Shi-Tomasi corner points. If corner point P j The surrounding area D j If there are more than σ corner points in the memory, then the corner point P j Keep it, otherwise delete it; in is the current set of corner points, is the number of corner points around the current corner point, σ is the number threshold; B. Perform length self-check on the straight line segments detected by EDLines and delete those whose length is less than the threshold L. T The straight line segment, L T Set according to the image resolution and the minimum size of buildings in the image; C. Check each other between the diagonal point and the straight line segment. J If there is no straight line segment within the radius, the corner point is a weak corner point, and the D of the straight line segment L If there is no corner point within the radius, the straight line segment is a pseudo straight line segment, and the weak corner points and pseudo straight line segments are deleted; D. There should be at least one corner point within the range of radius D from the foot of a right angle detected based on the EDLines line segments. Otherwise, the right angle is deleted. Step 2.4.5: For the high-quality corner points, line segments, and right-angle geometric features after screening, they are respectively converted into binary images. The pixel values of the corner points, line segments, and right angles are 255, and the background pixel value is 0. Then, the building geometric heat map feature index BGHotindex of the sample image is calculated using the natural neighbor interpolation method. Among them BGHotindex (a,b) is the geometric heat value of the building at coordinates a and b, For the The true value of the binary image of the corner points involved in the interpolation, for The corresponding weights, For the The true value of the binary graph of the straight line segments involved in the interpolation, for The corresponding weights, For the The true value of the rectangular binary graph involved in the interpolation, for The corresponding weights, HJ, HL, and HZ, are the number of corner points in the corner binary map, the number of straight line segments in the straight line binary map, and the number of right angles in the right angle binary map. The truth value V and the weight W are determined by the Thiessen polygons; Where Area(a,b) is the area of the new cell centered at (a,b). is the area of the intersection between the new cell centered at (a,b) and the old cell centered at (a,b).
8. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 1, characterized in that: In Step 3, each single-pixel neighborhood in the initial building semantic segmentation sample is used as a processing unit, and the initial building semantic segmentation sample is converted into a neighborhood image block recognition sample. Each neighborhood image block recognition sample includes a neighborhood image block and a central pixel label, which specifically includes: Step 3.1: For each building semantic segmentation sample, a sliding window with a neighborhood size of n*n is used. Step 3.2, superimpose the original sample image piece, the building existence index feature image, the building morphological feature index feature image and the building geometric heat map feature index feature image to obtain six bands of data, and cut the superimposed six bands of data with an n*n neighborhood size sliding window to obtain a neighborhood image block. The label of the neighborhood image block sample is the category label of the pixel at the center position of the neighborhood sliding window. One building semantic segmentation sample can generate R neighborhood image block recognition samples; Step 3.3, all initial building semantic segmentation samples are converted into neighborhood image block recognition samples. After the conversion, R*S neighborhood image block recognition samples are generated. The first R neighborhood image block recognition samples correspond to the first one in the initial semantic segmentation sample set, and the z*Rth to (z+1)*Rth neighborhood image block recognition samples correspond to the z+1th in the initial semantic segmentation sample set. The value range of z is 0 to S-1.
9. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 1, characterized in that: In step 5, the neighborhood image block recognition samples are estimated using a K-fold cross-validation method to estimate the joint distribution of noise labels and true labels, including: Define the noise label as That is, all neighborhood image blocks identify sample labels, which contain both correct and incorrect labels; the true label is defined as lb * , the true label is estimated by K-fold cross validation, the total number of categories is CL, Step 5.1: Perform K-fold cross validation on all neighborhood image block recognition sample data, α represents the sample, β represents the category, calculate the probability P[α][β] of the αth sample in the βth category, and then calculate the average probability t[β] under each category β as the confidence threshold. Finally, for the neighborhood image block recognition sample α, its true label lb * is the maximum probability argmax in the βth category β The labels corresponding to P[α][β], and P[α][β]>t[β]; Step 5.2: Calculate the count matrix Step 5.3, calibrate the count matrix so that the total count is the same as the total number of sample labels identified in all neighborhood image blocks. The calibrated count matrix The calculation formula is: in Identify sample labels for all neighborhood patches The total number of samples, is the count matrix after calibration; Step 5.4: Estimating Noise Labels and the true label lb * The joint distribution of The joint distribution The calculation formula is:
10. The method for eliminating erroneous samples from a building semantic segmentation sample set based on confidence learning according to claim 8, characterized in that: The step 6, using the joint distribution of the noise label and the true label to find the wrong label, for each initial building semantic segmentation sample, according to the proportion of the wrong label in the initial building semantic segmentation sample, according to the confidence threshold method, estimates the wrong sample in the initial building semantic segmentation sample, specifically includes: Step 6.1: Based on the joint distribution of noise labels and true labels At the same time, Prune byClass and Prune byNoise Rate are used to filter out the error samples. Step 6.2: For the R*S neighborhood image block recognition samples, determine whether each sample is an error sample through the step 6.
1. If the proportion of error samples in the z*Rth to (z+1)*Rth neighborhood image block recognition samples exceeds the set threshold θ, it indicates that the z+1th sample in the initial building semantic segmentation sample set is an error sample.
Citation Information
Patent Citations
High-resolution image weak supervision building extraction method combining pixel semantic association and boundary attention
CN112052783A
Method for constructing large-scale remote sensing image semantic segmentation model training sample set
CN112884791A
Remote sensing image building semantic segmentation edge optimization method based on multitask CNN+GCN
CN113449640A
Super-resolution building fine identification method based on multi-scale feature deconvolution
CN113469052A
Ultrahigh-resolution remote sensing image earthquake damage building detection method
CN113688696A
