Method, device, equipment and medium for recognizing building roof outlines from remote sensing images

Abstract

The present invention relates to the field of image processing technology and provides a method, apparatus, device, and medium for recognizing building roof outlines using remote sensing images. The method comprises: acquiring remote sensing image data; inputting the remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model; wherein the building roof outline optimization recognition model is obtained by training a pre-constructed generative adversarial network using remote sensing image sample data; and the loss function of the generative adversarial network is determined based on a boundary loss function. The present invention improves the accuracy of the building roof outline recognition model by incorporating a boundary loss function into the building roof outline optimization recognition model to constrain the regularized generation of building roof mask boundaries.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and medium for recognizing the outline of a building roof using remote sensing images. Background Art

[0002] Traditional pixel-based automatic building roof extraction methods focus on utilizing visual features, such as corner and edge features detected by the Scale-invariant Feature Transform (SIFT) and Canny edge detection algorithms to extract buildings. Object-based image analysis methods proposed by Geographic Object-Based Image Analysis (GEOBIA) are limited by multi-scale segmentation methods and the high heterogeneity of spatial and spectral targets in high-resolution image semantics, making it difficult to obtain complete building roof outlines. Manually designing and selecting empirical features for building roofs can only achieve good results under specific requirements or specific data. Due to the spatial heterogeneity of building features distributed across regions, such extraction methods based on manual feature design generally lack good universal representation and generalization capabilities. Traditional remote sensing image target recognition methods have difficulty effectively obtaining complete physical image primitives for building recognition.

[0003] Deep learning network models produce inaccurate segmentation results at boundaries, particularly in low-contrast boundary areas. This problem can also lead to secondary problems, such as inaccurate boundary delineation when manually constructing samples. Secondly, the roof grid boundary is composed of multiple pixels across its width, resulting in the extracted polygonal contours of building roof instances often exhibiting a jagged or speckled appearance during the "raster-to-vector" conversion process, which does not conform to industry standards for vectorized mapping of building roof edges in Geographic Information Systems (GIS). Finally, because the geometric spectral characteristics of roofs are easily confused by side elevations, trees, and shadows, this method is prone to false detections. Deep learning network models are computationally intensive and extremely complex, making it difficult for users to interpret their decision-making mechanisms even when the model has a clear network structure. This makes subsequent optimization and generalization of the model difficult. Furthermore, to maintain high-precision recognition results, deep learning network models also require a certain number and quality of samples. Currently, the spatial distribution and types of building rooftop samples are insufficiently representative of the coupled factors of different regional geographic environments, and the sample library construction process lacks effective theoretical guidance. This leads to overfitting of the model to local architectural features or certain types of architectural features, thereby reducing the generalization ability of deep learning network models when applied on a large scale. In addition, the model cannot quantify the type and number of training samples, nor does it clearly define which types of samples (features) contribute to the parameterization process of model training. This can lead to repeated and ineffective labeling of some samples that are unbalanced for training. The "black box" sample construction process lacks inherent mechanism guidance, resulting in a sharp increase in the workload of manual labeling and inaccurate targets.

[0004] Existing building roof recognition methods suffer from low accuracy in large-scale processing. Summary of the Invention

[0005] The present invention provides a method, device, equipment and medium for recognizing building roof outlines in remote sensing images, which are used to solve the problem of low accuracy in large-scale image processing. By adding a boundary loss function to the building roof outline optimization recognition model to constrain the regular generation of the building roof mask boundary, the accuracy of the building roof outline recognition model is improved.

[0006] The present invention provides a method for recognizing the outline of a building roof using remote sensing images, comprising:

[0007] Collect remote sensing image data;

[0008] Inputting the remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model;

[0009] Among them, the building roof outline optimization recognition model is obtained by training a pre-built generative adversarial network using remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function.

[0010] In one embodiment, the generative adversarial network includes a generator and a discriminator, the generator includes a bounding box generator and a mask generator, and the discriminator includes a bounding box discriminator and a mask discriminator;

[0011] The formula of the generator's loss function is:

[0012]

[0013] Among them, L Generator represents the loss function of the generator, L cls represents the category loss function of the generator, L bbox represents the bounding box loss function of the generator, L mask represents the mask loss function of the generator, represents the loss function of the bounding box generator, represents the loss function of the mask generator.

[0014] In one embodiment, the mask loss formula of the generator is:

[0015]

[0016] Among them, L mask represents the mask loss function of the generator, L BR represents the boundary loss function; λ and η represent the hyperparameters of the constraint loss; y represents the ground truth label value of the building roof, and x represents the predicted value of the building roof; m 2 Indicates that the mask has an output of m*m dimensions.

[0017] In one embodiment, the boundary loss function is determined based on the following steps:

[0018] Using boundary models in the spatial and spectral domains, the grayscale value distribution of the edge transition zone is determined; the edge transition zone is used to characterize the area formed by mixed pixels on both sides of the building roof boundary;

[0019] Determining pixel values ​​along a direction of the building roof edge and pixel values ​​perpendicular to a direction of the building roof edge based on the grayscale value distribution of the edge transition region;

[0020] Constructing a first loss function based on pixel values ​​along a direction of an edge of a building roof, and constructing a second loss function based on pixel values ​​perpendicular to a direction of an edge of a building roof;

[0021] A boundary loss function is determined based on the first loss function and the second loss function.

[0022] In one embodiment, the formula of the boundary loss function is:

[0023]

[0024]

[0025] L BR =αL BH +βL BV ;

[0026] Among them, N1 represents the number of edge pixels along the edge direction of the building roof, T represents the number of edge pixels perpendicular to the edge direction of the building roof, represents the predicted probability value of the i-th pixel of the image along the direction of the building roof edge, represents the predicted probability value of the i-1th pixel of the image along the direction of the edge of the building roof, represents the predicted probability value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the predicted probability value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image along the edge of the building roof, represents the true value of the i-1th pixel of the image along the edge of the building roof, α and β represent the hyperparameters of the constraint loss, and L BH Represents the first loss function, L BV represents the second loss function.

[0027] In one embodiment, the loss function of the discriminator is formulated as follows:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] Among them, L Discriminator represents the loss function of the discriminator, represents the loss function of the bounding box discriminator, represents the loss function of the mask discriminator, represents the ground truth value corresponding to the i-th bounding box, represents the ground truth value corresponding to the i-th binary mask prediction, G m (ROI i ) represents the i-th binary mask prediction; N2 represents the batch size; G b (ROI i ) represents the predicted value of the i-th bounding box region of interest; D b () represents the probability that the bounding box is true, D m () represents the probability that the predicted mask is true.

[0034] In one embodiment, the building roof profile optimization recognition model is obtained based on the following steps:

[0035] Collect remote sensing image sample data;

[0036] Preprocessing the remote sensing image sample data, wherein the preprocessing includes image fusion, image framing, linear stretching and image filtering;

[0037] Extracting sample features of pre-processed remote sensing image sample data to generate a training sample set;

[0038] The training sample set is used to train a pre-constructed generative adversarial network to obtain the building roof outline optimization recognition model.

[0039] The present invention also provides a remote sensing image building roof outline recognition device, comprising:

[0040] Acquisition module, used to collect remote sensing image sample data;

[0041] A building roof outline recognition module is used to input the remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model;

[0042] Among them, the building roof outline optimization recognition model is obtained by training a pre-built generative adversarial network using remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function.

[0043] The present invention also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described methods for recognizing building roof outlines using remote sensing images is implemented.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for recognizing the roof outline of a building using remote sensing images as described above is implemented.

[0045] The present invention provides a method, apparatus, device, and medium for recognizing building roof outlines from remote sensing images. These methods collect remote sensing image data, input the data into a building roof outline optimization recognition model, and obtain the building roof outline output by the model. The model is obtained by training a pre-constructed generative adversarial network using remote sensing image sample data. The loss function of the generative adversarial network is determined based on a boundary loss function. By incorporating a boundary loss function into the model to constrain the regularized generation of building roof mask boundaries, the present invention improves the accuracy of the building roof outline recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flow chart of the method for recognizing the outline of a building roof using remote sensing images provided by the present invention;

[0048] Figure 2 It is a structural schematic diagram of the building roof outline optimization recognition model provided by the present invention;

[0049] Figure 3 Schematic diagram of the structure of the bounding box generator and the bounding box discriminator provided by the present invention;

[0050] Figure 4 Schematic diagram of the structure of the mask generator and mask discriminator provided by the present invention;

[0051] Figure 5 It is a schematic diagram of the spectrum curve of the edge transition area along the boundary direction of the building roof and perpendicular to the boundary direction in the remote sensing image data provided by the present invention;

[0052] Figure 6 This is a schematic diagram of the production process of the training sample set provided by the present invention;

[0053] Figure 7 It is a flow chart of the construction and optimization method of the remote sensing image building roof outline recognition model provided by the present invention;

[0054] Figure 8 This is a schematic diagram of the structure of the remote sensing image building roof outline recognition device provided by the present invention;

[0055] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0057] The following combination Figures 1-9 The present invention describes the method, device, equipment and medium for recognizing the roof outline of a building using remote sensing images.

[0058] Specifically, the present invention provides a method for recognizing the roof outline of a building using remote sensing images. Figure 1 .

[0059] The embodiment of the present invention provides a method for recognizing building roof outlines using remote sensing images, comprising:

[0060] S100, collects remote sensing image data;

[0061] It should be noted that remote sensing image data refers to films or photos that record the size of electromagnetic waves of various ground objects. Remote sensing image data has the characteristics of spatial resolution, spectral resolution, time series, and multi-source data fusion, and can conduct all-round monitoring and observation of the earth's surface.

[0062] Remote sensing image data is collected via platforms such as satellites, aircraft, and drones, or via ground-based surveying instruments. Optionally, the remote sensing image data includes the Inria Aerial Image Labeling Dataset (Inria) and remote sensing image data provided by the International Society for Photogrammetry and Remote Sensing (ISPRS), specifically including multispectral and panchromatic image data.

[0063] Furthermore, to improve the accuracy of building rooftop profile recognition, remote sensing image data needs to be preprocessed after acquisition. The remote sensing image data in the embodiments of the present invention is remote sensing image data that is compatible with neural networks. Preprocessing involves converting high-resolution remote sensing images into image data suitable for thematic feature recognition methods based on deep learning algorithms. For example, preprocessing remote sensing image data can be performed using image fusion, image framing, linear stretching, and image filtering.

[0064] S200, inputting remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model;

[0065] Among them, the building roof outline optimization recognition model is obtained by training the pre-built generative adversarial network with remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function.

[0066] The pre-processed remote sensing image data is input into the trained building roof outline optimization recognition model, and the building roof outline recognition result is output.

[0067] The optimized rooftop recognition model uses a generative adversarial network (GAN) to optimize the generation of original recognition masks for building roofs. GANs can better learn the distribution of data, resulting in more realistic and clear segmentation of building rooftops in input remote sensing imagery. Furthermore, during the training process of the GAN, the adversarial loss used in the GAN complements the standard pixel-level loss used in convolutional neural networks, improving the training efficiency of the model.

[0068] The loss function of the generative adversarial network in the embodiment of the present invention adds a boundary loss function, which can constrain the regular generation of the mask boundary of the building roof, and through the iterative feedback of the discriminator, make the model training results closer to the ground truth label, thereby improving the recognition accuracy of the building roof contour optimization recognition model.

[0069] The method for recognizing building roof outlines using remote sensing images provided by an embodiment of the present invention involves collecting remote sensing image data; inputting the remote sensing image data into a building roof outline optimization recognition model to obtain the building roof outline output by the building roof outline optimization recognition model; wherein the building roof outline optimization recognition model is obtained by training a pre-constructed generative adversarial network using remote sensing image sample data; and the loss function of the generative adversarial network is determined based on a boundary loss function. By incorporating a boundary loss function into the building roof outline optimization recognition model, the present invention constrains the regularized generation of building roof mask boundaries, thereby improving the accuracy of the building roof outline recognition model.

[0070] Based on the above embodiment, the generative adversarial network includes a generator and a discriminator, the generator includes a bounding box generator and a mask generator, and the discriminator includes a bounding box discriminator and a mask discriminator;

[0071] The formula of the generator's loss function is:

[0072]

[0073] Among them, L Generator represents the loss function of the generator, L cls represents the category loss function of the generator, L bbox represents the bounding box loss function of the generator, L mask represents the mask loss function of the generator, represents the loss function of the bounding box generator, represents the loss function of the mask generator.

[0074] The Multi-scale Boundary Regulation Generative Network (MSBR-GNet) is a generative adversarial network consisting of a generator and a discriminator. The generator learns regularized building roof masks, while the discriminator distinguishes between ideal building masks. During training, the generator and discriminator compete with each other, ultimately reaching a Nash equilibrium to produce the optimal output. The generator ultimately outputs a building roof mask that is closer to the ground truth, with clear boundaries and corners.

[0075] It should be noted that the building roof mask can be understood as the building roof outline.

[0076] The generator includes a bounding box generator and a mask generator:

[0077] like Figure 2 、 3 As shown, the bounding box generator uses the bounding box head of the multi-scale convolutional neural network (MS-CNN) as the input of the bounding box generator of MSBR-GNet. It then uses the region of interest (RoI) feature to extract the feature map. The feature map passes through the fully connected layer and outputs the class and bounding box prediction. During the training process, the predicted bounding box and the ground truth bounding box label are selected and sent to the bounding box discriminator.

[0078] like Figure 2 、 4As shown, the mask generator uses the MS-CNN mask head as input. This network consists of four convolutional layers, a transposed convolutional layer, and a 1×1 convolutional layer. It takes the MS-CNN RoI features as input, processes them through the convolutional layers to generate feature maps, and outputs a predicted mask for each feature map. In this embodiment of the present invention, the predicted mask size is 512×512. During training, logs are extracted from the predicted class channel and sent to the mask discriminator.

[0079] The discriminator includes a bounding box discriminator and a mask discriminator:

[0080] like Figure 2 、 3 As shown, the purpose of the bounding box discriminator is to receive the feature predictions of the region of interest (prRoI) from the bounding box generator and judge their quality. Since it is difficult for the discriminator to evaluate the bounding box prediction of an object by observing 4 coordinates. Therefore, the solution of an embodiment of the present invention is to send the feature map of the bounding box prediction and its ground truth as a pair of false and real sample labels to the bounding box discriminator. To this end, an embodiment of the present invention uses precise RoI pooling on the feature map output of the Feature Pyramid Network (FPN) backbone to extract the region of interest for bounding box prediction. The discriminator network consists of 5 convolutional layers, and each layer is followed by a batch normalization (BN) layer and an activation function LeakyReLU layer. It accepts a multidimensional image of size 512×512 and outputs a score between [0, 1] to indicate the quality of the prediction result, where the higher the score, the better the prediction result.

[0081] like Figure 2 、 4 As shown in the figure, the predicted mask and the ground truth mask (GT-Mask) are input to the mask discriminator, and both are multiplied with the RoI feature map of size 512×512. The mask discriminator network has 5 convolutional layers, and its structure is similar to that of an encoder. As the convolution operation leads to the problem of layer-by-layer resolution reduction, in order to preserve the geometric details of the building roof, the features of each convolutional layer are connected to a single output in a manner similar to a residual link, and used to calculate the adversarial loss.

[0082] The generator loss function of the rooftop optimization and recognition model is a combination of the generator loss and the discriminator loss. After each batch of training data is fed into the model, the loss function outputs a predicted value through forward propagation. The loss function then calculates the difference between the predicted value and the true value, which is the loss value. After obtaining the loss value, the model updates various parameters through backpropagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the rooftop optimization and recognition model approaches the true value, thus achieving the learning goal.

[0083] The loss function of the generator consists of the category loss function of the generator, the bounding box loss function of the generator, the mask loss function of the generator, the loss function of the bounding box generator, and the loss function of the mask generator.

[0084] This embodiment of the present invention constructs a generator and a discriminator within a generative adversarial network to learn regularized building roof masks and distinguish ideal building masks. During the training of the optimized roof outline recognition model, the generator and discriminator compete with each other, ultimately reaching a Nash equilibrium to produce the optimal output, thereby improving the accuracy of the optimized roof outline recognition model in identifying building roofs.

[0085] Based on the above embodiment, the formula of the generator's mask loss is:

[0086]

[0087] Among them, L mask represents the mask loss function of the generator, L BR represents the boundary loss function; λ and η represent the hyperparameters of the constraint loss, y represents the ground truth label value of the building roof, x represents the predicted value of the building roof, and m 2 Indicates that the mask has an output of m*m dimensions.

[0088] represents the mask loss of MS-CNN.

[0089] Optionally, λ=1, η=0.5.

[0090] L BR Used to optimize the generation of building roof profiles.

[0091] In the process of predicting the mask of the building roof, due to the edge transition zone formed by the mixed pixels on both sides of the roof boundary, the segmentation result of the model is inaccurate at the boundary. In order to improve the accuracy of the mask boundary extraction of the building roof, the mask loss function (L mask ), add the boundary loss function (L BR ) to constrain the generation of building roof mask boundaries.

[0092] The embodiment of the present invention constrains the regular generation of the mask boundary of the building roof by adding a boundary loss function term to the mask loss of the generator, and makes the model training result closer to the ground truth label through iterative feedback of the discriminator.

[0093] Based on the above embodiment, the boundary loss function is determined based on the following steps:

[0094] S201, using boundary models in the spatial domain and the spectral domain, determining the grayscale value distribution of the edge transition region; the edge transition region is used to represent the region formed by mixed pixels on both sides of the building roof boundary;

[0095] S202, determining pixel values ​​along the edge direction of the building roof and pixel values ​​perpendicular to the edge direction of the building roof based on the grayscale value distribution of the edge transition area;

[0096] S203, constructing a first loss function based on pixel values ​​along the edge direction of the building roof, and constructing a second loss function based on pixel values ​​perpendicular to the edge direction of the building roof;

[0097] S204: Determine a boundary loss function based on the first function and the second loss function.

[0098] The boundary statistical patterns in the spatial and spectral domains describe the fluctuation range of pixels in the image edge transition area in the value domain and the distribution characteristics in the spatial domain, and the boundary model can be established based on this. Figure 5 As shown in the figure, it can be seen that in the edge transition area (ΔSV, ΔSH), the grayscale (R, G, B channels) distribution fluctuation along the building roof boundary direction (ΔSH) is small, and the grayscale (R, G, B channels) distribution fluctuation perpendicular to the building roof boundary direction (ΔSV) is large. AB , ΔSV AD , ΔSV BC , ΔSV CD They represent the vertical direction of the AB side, the vertical direction of the AD side, the vertical direction of the BC side, and the vertical direction of the CD side respectively.

[0099] The fluctuation range of pixel values ​​in the edge transition area along the direction of the building roof edge and perpendicular to the direction of the building roof edge is statistically analyzed, as shown in Tables 1 and 2. AB , ΔSH BC , ΔSH CD , ΔSH AD They represent the directions along the AB side, the BC side, the CD side, and the AD side, respectively.

[0100] Table 1 Pixel value fluctuation range along the building roof edge direction (ΔSH)

[0101]

[0102] Table 2 Pixel value fluctuation range in the direction perpendicular to the building roof edge (ΔSV)

[0103]

[0104]

[0105] According to statistical analysis, ΔSH values ​​along the roof edge generally range from 5 to 30, with a mean of 13.4 and a minimum of 6, with small fluctuations and a variance of 34.9. ΔSV values ​​perpendicular to the roof edge, on the other hand, are generally above 30, with a mean of 74.6 and a maximum of 163, and with large fluctuations and a variance of 1835.2.

[0106] An embodiment of the present invention proposes to construct a boundary loss function based on a boundary model in the spatial domain and the spectral domain; by constructing a first loss function and a second loss function, the determination of the boundary loss function is further refined, and the model's recognition accuracy of the building roof boundaries in remote sensing image data is improved.

[0107] Based on the above embodiment, the formula of the boundary loss function is:

[0108]

[0109]

[0110] L BR =αL BH +βL BV ;

[0111] Among them, N1 represents the number of edge pixels along the edge direction of the building roof, T represents the number of edge pixels perpendicular to the edge direction of the building roof, represents the predicted probability value of the i-th pixel of the image along the direction of the building roof edge, represents the predicted probability value of the i-1th pixel of the image along the direction of the edge of the building roof, represents the predicted probability value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the predicted probability value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image along the edge of the building roof, represents the true value of the i-1th pixel of the image along the edge of the building roof, α and β represent the hyperparameters of the constraint loss, and L BH Represents the first loss function, L BV represents the second loss function.

[0112] In the embodiment of the present invention, an initial value α=β=0.5 is set, and the initial value is related to the spatial resolution and boundary strength of the image.

[0113] In the embodiment of the present invention, a loss function L is first constructed for the pixel values ​​along the edge of the building roof. BH (First loss function), and then construct the loss function L for the pixel value perpendicular to the edge of the building roof BV (Second loss function), and finally establish the boundary loss function L based on the boundary model BR .

[0114] The embodiment of the present invention establishes a boundary loss function based on the statistical model of the grayscale value distribution in the edge transition zone, combines geological mechanisms, and gives full play to the advantages of knowledge-driven and data-driven methods, thereby enhancing the interpretability of the model.

[0115] Based on the above embodiment, the formula of the loss function of the discriminator is:

[0116]

[0117]

[0118]

[0119]

[0120]

[0121] Among them, L Discriminator represents the loss function of the discriminator, represents the loss function of the bounding box discriminator, represents the loss function of the mask discriminator, represents the ground truth value corresponding to the i-th bounding box, represents the ground truth value corresponding to the i-th binary mask prediction, G m (ROI i ) represents the i-th binary mask prediction, N2 represents the batch size; G b (ROI i ) represents the predicted value of the i-th bounding box region of interest; D b () represents the probability that the image is real, D m () represents the probability that the predicted mask is true.

[0122] During training, Enhanced D b the ability to distinguish between real and false bounding boxes, Enhanced D m The ability to distinguish between real and false masks.

[0123] The embodiment of the present invention constructs a loss function of the discriminator to quantitatively evaluate the output results of the discriminator, thereby improving the model's prediction accuracy for building roof profiles.

[0124] Based on the above embodiment, the building roof profile optimization recognition model is obtained based on the following steps:

[0125] S205, collecting remote sensing image sample data;

[0126] S206, preprocessing the remote sensing image sample data, the preprocessing including image fusion, image framing, linear stretching and image filtering;

[0127] S207, extracting sample features of the pre-processed remote sensing image sample data to generate a training sample set;

[0128] S208: Using the training sample set to train the pre-built generative adversarial network, a building roof outline optimization recognition model is obtained.

[0129] like Figure 6 、 7 As shown in Figure 2, the process of making a training sample set includes:

[0130] (1) Collecting remote sensing image sample data; the remote sensing image sample data in this embodiment includes multispectral image data and panchromatic image data.

[0131] (2) Preprocessing the collected remote sensing image sample data, wherein the preprocessing includes: image fusion, image segmentation, linear stretching and image filtering.

[0132] Image fusion: refers to the fusion of remote sensing images of different spatial resolutions and spectral resolutions of the same area to obtain images that meet both high spatial resolution and high spectral resolution. Image fusion is used to combine high spatial and multispectral information to obtain a fused multispectral image, retaining the spatial information from the high-resolution panchromatic image and the spectral characteristics of the lower-resolution multispectral image. Among them, image fusion methods include panchromatic-multispectral fusion method and Gram-Schmidt image fusion method (Gram-Schmidt Pan Sharpening, GS). Optionally, since the Gram-Schmidt image fusion method has relatively high performance in maximizing image clarity and minimizing spectral distortion, and there is no limit on the number of bands that need to be fused, the embodiment of the present invention adopts the Gram-Schmidt image fusion method to perform image fusion on remote sensing image data.

[0133] Image Framing: Fused remote sensing images are rich in spatial and spectral information, but also have large file sizes. Excessive data input during network model training can reduce the training and prediction efficiency of deep learning algorithms. Furthermore, the fully connected layers of neural networks require fixed-size inputs. If the inputs of the training images vary in size, the model will process the input images to the same size before entering the fully connected layer. This will cause the size of the feature vectors extracted by the corresponding convolutional layer to change, affecting the final prediction accuracy. Therefore, after considering the load capacity, training efficiency, and training image information of the neural network algorithm, the fused remote sensing images need to be framed. For example, remote sensing images of different sizes can be segmented into multiple images of the same size, such as splitting a large image into multiple images of 500*500 pixels.

[0134] Optionally, the image framing may be performed based on the load capacity of the neural network algorithm, training efficiency, and training image information.

[0135] Linear stretching: Remote sensing images are affected by many factors during the imaging process, and their final visual effects can vary greatly. Some images are blurred due to insufficient contrast, and some images have better visual effects but cannot highlight the required thematic information. High-quality data is the driving force behind the development of deep learning applications. In order to make remote sensing information more accurate, it is particularly important to analyze and study remote sensing image enhancement methods. Since the original image is a 16-bit TIFF image, considering the data capacity, the embodiment of the present invention uses a 2%-98% maximum and minimum value stretching to quantize the 16-bit image to 8 bits. For example, the low grayscale value is set to 2%, and the high grayscale value is set to 98%, and the interval is stretched. The implementation steps are as follows: calculate the grayscale histogram of the image to be stretched; traverse the grayscale histogram, and when the cumulative number of low grayscale pixels reaches 2%, set the grayscale value as the starting grayscale value; continue to accumulate the number of pixels, and when the number of pixels accumulates to 98%, set the grayscale value to the ending grayscale value; use the obtained starting grayscale value and ending grayscale value to normalize the grayscale values ​​of all pixels; reconvert the normalized pixel values ​​to the range of 0-255 or 0-65535. If the converted value exceeds the mapping interval, set the corresponding maximum or minimum value. By linearly stretching the remote sensing image, the 16-bit image can be quantized to 8 bits. Based on this, the conflict between image processing speed and efficiency can be alleviated, and the linear stretching process can also obtain better visual effects, making subsequent processing and analysis more convenient.

[0136] Image filtering: While effectively suppressing the noise in the target and background, the integrity of the detailed features of the geometric and topological structure of the processed image target also needs to be considered during the image preprocessing process. Image filtering can better retain the detailed features of the image while suppressing the noise of the target image. It is an indispensable operation in image preprocessing. The quality of its processing effect will directly affect the effectiveness and reliability of subsequent image processing and analysis. Optionally, the image filtering method adopted in the embodiment of the present invention is a bilateral filtering processing method. Bilateral filtering is a nonlinear filtering that combines the spatial proximity and pixel value similarity of the image for compromise processing, while considering spatial information and grayscale similarity. The spatial function ensures that only pixels in the adjacent area have an impact on the center point, and the grayscale similarity function ensures that only pixels with grayscale values ​​close to the center pixel will be used for blurring operations. Since the grayscale value at the boundary varies greatly, the surrounding pixels will not affect the edge pixels too much, so the edge preservation effect of this method is very good.

[0137] (3) Use image annotation tools to extract sample features from the preprocessed remote sensing image sample data and analyze the data, including target location, target mask, target label, and original image. The sample features are divided into two categories: geometric features and spectral features. Geometric features include U-shaped, Z-shaped, elliptical, combined, rectangular, H-shaped, L-shaped, and T-shaped; spectral features include yellow, gray, green, black, blue, red, white, and brown. The labeled sample data is converted into Json format using the image annotation tool. The Json file is parsed to generate the corresponding building roof feature attributes and mask information, and a standard COCO format training sample set is constructed.

[0138] The pre-built generative adversarial network is trained using a training sample set to obtain an optimized recognition model for building roof contours.

[0139] Furthermore, image annotation tools were used to extract sample features from the preprocessed remote sensing image sample data and analyze the data to construct a validation sample set and a test sample set. The validation sample set was used to verify the optimized recognition model for building roof outlines, while the test sample set was used to test the optimized recognition model for building roof outlines, including testing the refined recognition of building roofs.

[0140] The embodiment of the present invention constructs a training sample set that is adapted to the building roof image target, has high universality, and integrates the concept of regional coupling through sample feature extraction.

[0141] Optionally, the training sample set is verified through a spectral sample ablation experiment, so as to more efficiently make the building roof profile optimization recognition model have both regional coupling and global universality.

[0142] It should be noted that when extracting building roof feature information from sample images, the following points should be noted: (1) Pay attention to the semantic granularity division of objects in multi-angle images and the correspondence between objects in the images. (2) When extracting building roof feature information, the multi-scale characteristics of the target objects should be taken into account, and small-scale building roof features should be retained so that the model can learn building roof features at multiple scales. (3) When extracting building roof feature information, avoid areas blocked by shadows or vegetation to prevent interference with building roof recognition results.

[0143] The embodiment of the present invention improves the accuracy of the building roof outline optimization recognition model by constructing a training sample set and training the building roof outline optimization recognition model through the training samples.

[0144] In order to further analyze and illustrate the remote sensing image building roof outline recognition method provided by the embodiment of the present invention, the embodiment of the present invention performs building roof outline recognition on WorldView-3 high-resolution remote sensing data of the Beijing urban sub-center.

[0145] The remote sensing image data includes panchromatic images with a spatial resolution of 0.3 m and multispectral images with a spatial resolution of 1.24 m. The longitude and latitude range is nwLat = 39°96', nwLong = 116°63', seLat = 39°84', seLong = 116°78'.

[0146] The experimental hardware and software environment was Ubuntu 16.04, implemented in PyTorch 0.4.1 with CUDA 9.0, and run on a single NVIDIA GeForce GTX 1060 graphics processing unit (GPU). In all experiments, we used a weight decay of 0.0001 and a momentum of 0.9, along with the Stochastic Gradient Descent (SGD) optimizer. Furthermore, we used the same learning rate of 0.001 for both the generator and the discriminator.

[0147] The hyperparameters of the rooftop profile optimization recognition model are trained using the constructed training sample set. Based on the model evaluation results, the hyperparameters are adjusted to the optimal level to ensure that the rooftop profile optimization recognition model achieves good recognition performance and accuracy. Model training primarily involves data input (creating a model training database) and network training.

[0148] Furthermore, using the training sample set to train the pre-built generative adversarial network includes: data input and network training.

[0149] (1) Data input: The training sample set is input into the constructed building roof outline optimization recognition model. In the process of inputting the training sample set into the network, the R, G, and B corresponding three channels of the training sample are used as the input of the model.

[0150] (2) Network training: The MSBR-GNet is trained using the formula of the generator's mask loss. The generator and discriminator are trained alternately. First, the generator is frozen and the discriminator is trained. Before the discriminator is trained to find out how to distinguish between real data and fake data, it first learns how to identify the generator's flaws. Similarly, the discriminator is frozen while the generator is trained. As training progresses, both the generator network and the discriminator network become more powerful, and eventually the generator will be able to produce predictions that are very close to the ground truth. Hyperparameters need to be set before network training. The main parameters of the model include: anchor box size, the number of samples passed to the program for training at a single time (Batch Size), the number of iterations (Epoch), the learning rate, etc. The optimal model hyperparameters are determined based on parameter adjustment experience and multiple experimental results. The anchor box size should be selected according to the size of the image and the target. The anchor box sizes corresponding to each feature map of P2-P6 are (32, 64, 128, 256, 512) respectively. Considering computer load and operational efficiency, this experiment requires dividing the training samples into multiple smaller batches and inputting them into the network. Optimizing the batch size is crucial for the performance and speed of the network model. Too large a batch size can lead to a lack of randomness in the training samples, trapping them in a local optimum. Too small a batch size can prevent the network from converging and slow down training. An epoch is the process of completing feature learning after all training samples are input into the network. Training samples require multiple iterations for the network model to converge. The learning rate is a crucial hyperparameter for deep learning. It controls the speed at which the neural network weights are adjusted based on the loss gradient. A smaller learning rate results in a slower descent of the loss gradient and a longer convergence time. Excessively large learning rates can easily skip the optimal value. Based on parameter tuning experience and multiple experiments, this experiment uses a batch size of 4, an epoch of 30, and a learning rate of 0.0001.

[0151] The training process was repeated based on the optimal hyperparameters, and the output of the trained model was saved. The trained rooftop profile optimization model was then used to identify objects in the test sample set. The recognition results showed relatively regular edges, with a structural similarity metric (SSIM) reaching 0.8696.

[0152] like Figure 8 As shown, an embodiment of the present invention further provides a remote sensing image building roof outline recognition device, comprising:

[0153] Acquisition module 801, for acquiring remote sensing image data;

[0154] The building roof outline recognition module 802 is used to input the remote sensing image data into the building roof outline optimization recognition model to obtain the building roof outline output by the building roof outline optimization recognition model;

[0155] Among them, the building roof outline optimization recognition model is obtained by training the pre-built generative adversarial network with remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function.

[0156] The remote sensing image building roof outline recognition device provided by an embodiment of the present invention collects remote sensing image data; inputs the remote sensing image data into a building roof outline optimization recognition model to obtain the building roof outline output by the building roof outline optimization recognition model. The building roof outline optimization recognition model is obtained by training a pre-constructed generative adversarial network using remote sensing image sample data; the loss function of the generative adversarial network is determined based on a boundary loss function. By establishing the building roof outline optimization recognition model to identify building roofs, the embodiment of the present invention improves the accuracy of building roof recognition.

[0157] In one embodiment, the building roof outline recognition module 802 specifically includes: the generative adversarial network includes a generator and a discriminator, the generator includes a bounding box generator and a mask generator, and the discriminator includes a bounding box discriminator and a mask discriminator; the loss function of the generator is formulated as follows:

[0158]

[0159] Among them, L Generator represents the loss function of the generator, L cls represents the category loss function of the generator, L bbox represents the bounding box loss function of the generator, L mask represents the mask loss function of the generator, represents the loss function of the bounding box generator, represents the loss function of the mask generator.

[0160] In one embodiment, the building roof outline recognition module 802 specifically includes: the mask loss formula of the generator is:

[0161]

[0162] Among them, L mask represents the mask loss function of the generator, L BR represents the boundary loss function; λ and η represent the hyperparameters of the constraint loss; y represents the ground truth label value of the building roof, and x represents the predicted value of the building roof; m 2Indicates that the mask has an output of m*m dimensions.

[0163] In one embodiment, the building roof outline recognition module 802 specifically includes:

[0164] Using boundary models in the spatial and spectral domains, the grayscale value distribution of the edge transition zone is determined; the edge transition zone is used to characterize the area formed by mixed pixels on both sides of the building roof boundary;

[0165] Determining pixel values ​​along a direction of the building roof edge and pixel values ​​perpendicular to a direction of the building roof edge based on the grayscale value distribution of the edge transition region;

[0166] Constructing a first loss function based on pixel values ​​along a direction of an edge of a building roof, and constructing a second loss function based on pixel values ​​perpendicular to a direction of an edge of a building roof;

[0167] A boundary loss function is determined based on the first loss function and the second loss function.

[0168] In one embodiment, the building roof outline recognition module 802 specifically includes: the formula of the boundary loss function is:

[0169]

[0170]

[0171] L BR =αL BH +βL BV ;

[0172] Among them, N1 represents the number of edge pixels along the edge direction of the building roof, T represents the number of edge pixels perpendicular to the edge direction of the building roof, represents the predicted probability value of the i-th pixel of the image along the direction of the building roof edge, represents the predicted probability value of the i-1th pixel of the image along the direction of the edge of the building roof, represents the predicted probability value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the predicted probability value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image along the edge of the building roof, represents the true value of the i-1th pixel of the image along the edge of the building roof, α and β represent the hyperparameters of the constraint loss, and L BH Represents the first loss function, L BV represents the second loss function.

[0173] In one embodiment, the building roof outline recognition module 802 specifically includes: the formula of the loss function of the discriminator is:

[0174]

[0175]

[0176]

[0177]

[0178]

[0179] Among them, L Discriminator represents the loss function of the discriminator, represents the loss function of the bounding box discriminator, represents the loss function of the mask discriminator, represents the ground truth value corresponding to the i-th bounding box, represents the ground truth value corresponding to the i-th binary mask prediction, G m (ROI i ) represents the i-th binary mask prediction; N2 represents the batch size; G b (ROI i ) represents the predicted value of the i-th bounding box region of interest; D b () represents the probability that the bounding box is true, D m () represents the probability that the predicted mask is true.

[0180] In one embodiment, the building roof outline recognition module 802 specifically includes:

[0181] Collect remote sensing image sample data;

[0182] Preprocessing the remote sensing image sample data, wherein the preprocessing includes at least one of image fusion, image framing, linear stretching, and image filtering;

[0183] Extracting sample features of pre-processed remote sensing image sample data to generate a training sample set;

[0184] The training sample set is used to train a pre-constructed generative adversarial network to obtain the building roof outline optimization recognition model.

[0185] Figure 9 An example of a physical structure diagram of an electronic device is shown below. Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a method for recognizing the roof outline of a building from a remote sensing image, the method comprising:

[0186] Collect remote sensing image data;

[0187] Inputting the remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model;

[0188] Among them, the building roof outline optimization recognition model is obtained by training a pre-built generative adversarial network using remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function.

[0189] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program code.

[0190] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing image building roof outline recognition method provided by the above methods, the method comprising:

[0191] Collect remote sensing image data;

[0192] Inputting the remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model;

[0193] Among them, the building roof outline optimization recognition model is obtained by training a pre-built generative adversarial network using remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0195] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for recognizing building roof outlines from remote sensing images, characterized in that: include: Collect remote sensing image data; Inputting the remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model; The building roof outline optimization recognition model is obtained by training a pre-built generative adversarial network using remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function; The generative adversarial network includes a generator and a discriminator, the generator includes a bounding box generator and a mask generator, and the discriminator includes a bounding box discriminator and a mask discriminator; The boundary loss function is determined based on the following steps: Using boundary models in the spatial domain and the spectral domain, the grayscale value distribution of the edge transition area is determined; the edge transition area is used to characterize the area formed by mixed pixels on both sides of the building roof boundary; Determining pixel values ​​along a direction of the building roof edge and pixel values ​​perpendicular to a direction of the building roof edge based on the grayscale value distribution of the edge transition region; Constructing a first loss function based on pixel values ​​along a direction of an edge of a building roof, and constructing a second loss function based on pixel values ​​perpendicular to a direction of an edge of a building roof; Based on the first loss function and the second loss function, a boundary loss function is determined; the formula of the boundary loss function is: L BR =αL BH +βL BV ; Among them, N1 represents the number of edge pixels along the edge direction of the building roof, T represents the number of edge pixels perpendicular to the edge direction of the building roof, represents the predicted probability value of the i-th pixel of the image along the direction of the building roof edge, represents the predicted probability value of the i-1th pixel of the image along the direction of the edge of the building roof, represents the predicted probability value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the predicted probability value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image along the edge of the building roof, represents the true value of the i-1th pixel of the image along the edge of the building roof, α and β represent the hyperparameters of the constraint loss, and L BH Represents the first loss function, L BV represents the second loss function.

2. The method for recognizing building roof outlines from remote sensing images according to claim 1, characterized in that: The formula of the generator's loss function is: Among them, L Generator represents the loss function of the generator, L cls represents the category loss function of the generator, L bbox represents the bounding box loss function of the generator, L mask represents the mask loss function of the generator, represents the loss function of the bounding box generator, represents the loss function of the mask generator.

3. The method for recognizing building roof outlines from remote sensing images according to claim 2, wherein: The formula for the mask loss of the generator is: Among them, L mask represents the mask loss function of the generator, L BR represents the boundary loss function; λ and η represent the hyperparameters of the constraint loss, y represents the ground truth label value of the building roof, x represents the predicted value of the building roof, and m 2 Indicates that the mask has an output of m*m dimensions.

4. The method for recognizing building roof outlines from remote sensing images according to claim 1, wherein: The formula of the loss function of the discriminator is: Among them, L Discriminator represents the loss function of the discriminator, represents the loss function of the bounding box discriminator, represents the loss function of the mask discriminator, represents the ground truth value corresponding to the i-th bounding box, represents the ground truth value corresponding to the i-th binary mask prediction, G m (ROI i ) represents the i-th binary mask prediction; N2 represents the batch size; G b (ROI i ) represents the predicted value of the i-th bounding box region of interest; D b () represents the probability that the bounding box is true, D m () represents the probability that the predicted mask is true.

5. The method for recognizing building roof outlines from remote sensing images according to claim 1, wherein: The building roof profile optimization recognition model is obtained based on the following steps: Collect remote sensing image sample data; Preprocessing the remote sensing image sample data, wherein the preprocessing includes image fusion, image framing, linear stretching and image filtering; Extracting sample features of preprocessed remote sensing image sample data to generate a training sample set; The training sample set is used to train a pre-constructed generative adversarial network to obtain the building roof outline optimization recognition model.

6. A device for recognizing building roof outlines using remote sensing images, characterized in that: include: Acquisition module, used to collect remote sensing image data; A building roof outline recognition module is used to input the remote sensing image data into a building roof outline optimization recognition model to obtain a building roof outline output by the building roof outline optimization recognition model; The building roof outline optimization recognition model is obtained by training a pre-built generative adversarial network using remote sensing image sample data; the loss function of the generative adversarial network is determined based on the boundary loss function; The generative adversarial network includes a generator and a discriminator, the generator includes a bounding box generator and a mask generator, and the discriminator includes a bounding box discriminator and a mask discriminator; The boundary loss function is determined based on the following steps: Using boundary models in the spatial and spectral domains, the grayscale value distribution of the edge transition zone is determined; the edge transition zone is used to characterize the area formed by mixed pixels on both sides of the building roof boundary; Determining pixel values ​​along a direction of the building roof edge and pixel values ​​perpendicular to a direction of the building roof edge based on the grayscale value distribution of the edge transition region; Constructing a first loss function based on pixel values ​​along a direction of an edge of a building roof, and constructing a second loss function based on pixel values ​​perpendicular to a direction of an edge of a building roof; Based on the first loss function and the second loss function, a boundary loss function is determined; the formula of the boundary loss function is: L BR =αL BH +βL BV ; Among them, N1 represents the number of edge pixels along the edge direction of the building roof, T represents the number of edge pixels perpendicular to the edge direction of the building roof, represents the predicted probability value of the i-th pixel of the image along the direction of the building roof edge, represents the predicted probability value of the i-1th pixel of the image along the direction of the edge of the building roof, represents the predicted probability value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the predicted probability value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-1th pixel of the image perpendicular to the edge of the building roof, represents the true value of the i-th pixel of the image along the edge of the building roof, represents the true value of the i-1th pixel of the image along the edge of the building roof, α and β represent the hyperparameters of the constraint loss, and L BH Represents the first loss function, L BV represents the second loss function.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the remote sensing image building roof outline recognition method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the remote sensing image building roof outline recognition method according to any one of claims 1 to 5 is implemented.

Images