Roof photovoltaic identification and potential evaluation method based on improved TransUNet semantic segmentation

Abstract

The invention discloses a roof photovoltaic identification and potential evaluation method based on improved TransUNet semantic segmentation. The method comprises the following steps: obtaining a building roof satellite image data set; cleaning and classifying the images in the data set according to whether roof features exist or not, and dividing out images with roofs; according to the roof image, roof features are extracted through the constructed and trained TransUNet semantic segmentation model, and a building roof area is identified and area division is carried out; and matching the processed image with the subject name, and positioning the subject to which the building belongs. According to the method, an innovative solution of a remote sensing technology and a machine learning method is integrated, the generalization ability of roof recognition semantic segmentation can be effectively improved, the roofs are accurately recognized and divided, and the method aims at achieving rapid and accurate evaluation of large-scale urban building roofs.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a rooftop photovoltaic recognition and potential assessment method based on improved TransUNet semantic segmentation. Background Art

[0002] As a renewable energy source, solar energy has become one of the most important renewable energy sources in the world. The development of rooftop photovoltaics is a long-term planning project. It is necessary to give priority to the development of rooftops with rich roof resources and good lighting conditions. Rooftops that do not meet the conditions for installing photovoltaics or rooftops that are difficult to recover costs will not be developed. It is necessary to have a systematic understanding and planning of rooftops. Learning image recognition semantic segmentation can help understand and process large amounts of image data. In the face of the lack of generalization ability of traditional methods for rooftop recognition, the semantic segmentation effect of rooftops in different cities and regions is poor. The key to promoting the construction of photovoltaic roofs is to accurately identify and divide roofs. Traditional rooftop recognition methods are difficult to achieve ideal results. Summary of the invention

[0003] Purpose of the invention: In view of the fact that in the prior art, when segmenting a large number of remote sensing images collected from different geographical locations or cities, the data volume is large and the resolution is low, and the pre-trained deep learning model does not always output satisfactory predictions, a rooftop photovoltaic identification and potential assessment method based on improved TransUNet semantic segmentation is provided, which integrates innovative solutions of remote sensing technology and machine learning methods, can effectively improve the generalization ability of rooftop recognition semantic segmentation, accurately identify and divide roofs, and aims to achieve rapid and accurate assessment of roofs of large-scale urban buildings.

[0004] Technical solution: To achieve the above purpose, the present invention provides a method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation, comprising the following steps:

[0005] S1: Get the satellite image dataset of building roofs;

[0006] S2: Clean and classify the images in the dataset according to whether they have roof features, and separate the images with roofs;

[0007] S3: Based on the images with roofs, the TransUNet semantic segmentation model is built and trained to extract roof features, identify the roof area of ​​the building and divide it into regions according to the roof structure, material, and color;

[0008] S4: Match the image processed in step S3 with the subject name to locate the subject to which the building belongs.

[0009] Furthermore, in step S2, The network is used as an image classification model to classify images. The network structure contains 50 convolutional layers, pooling layers, fully connected layers and residual modules, where each residual module consists of two convolutional layers and a skip connection; the skip connection directly bypasses the input signal through a certain number of convolutional layers and pooling layers, and adds the input signal to the output signal after the skip connection to form a residual signal.

[0010] Furthermore, the construction of the TransUNet semantic segmentation model in step S3 is specifically as follows:

[0011] A1: Combine the residual network with the U-Net model to form a residual U structure;

[0012] A2: Stacking residual U structures with different depths, including the down-sampling encoding part and the up-sampling decoding part, to build a deeper U 2 -Network model;

[0013] A3: Introduce the Transformer module with multi-head self-attention mechanism into U 2 -Net network model to obtain the TransUNet semantic segmentation model.

[0014] Furthermore, the encoder of the U-Net model in step A1 is composed of five parts. The encoder extracts features of the input image and reduces the resolution through four downsampling blocks and a channel adjustment module. Each module contains two 3×3 convolutional layers, and the downsampling block also includes an additional 2×2 pooling layer, which helps to further reduce the size of the feature map; the decoder is composed of four parts, which are responsible for the resolution restoration and feature integration of the feature map. Each part contains an upsampling layer and two 3×3 convolutional layers; a jump connection is used between the decoder and the encoder, allowing information to be directly passed from the encoding stage to the decoding stage, so as to retain and utilize more feature information, which helps to improve the accuracy and stability of the segmentation model;

[0015] A new type of residual module is used in the residual network structure. The output of the first convolutional layer of the residual module is directly added to the final output of the encoder. By introducing a shortcut connection, the output of the previous layer is directly added to the input of the current layer.

[0016] Furthermore, the U of the Transformer module containing the multi-head self-attention mechanism introduced in step A3 2-Net network model, the network structure divides the input features into 5 branches for processing. The first 3 branches use the self-attention mechanism to generate Query, Key, and Value vectors respectively, which are used to calculate the correlation between each pixel and all other pixels in the feature map. The decoder branch combines the backbone network with the self-attention mechanism to add the input features to the Query vector to better understand the relationship and contextual information between pixels in the input image and increase the receptive field of feature extraction; the last two branches focus on the height and width information of the input image, and introduce this information into the self-attention mechanism in a weighted manner.

[0017] Furthermore, in step A3, the multi-head self-attention mechanism establishes a feature map for useful target features, thereby suppressing unimportant features. The self-attention mechanism is calculated as follows:

[0018] Given an input tensor , Query, Key and Value are convolved 1×1, and three 1×1 convolutions are used to project the feature map for query, key and value embedding: Query, Key, Valu , where d is the size of each head embedded; then Query, Key and Value are flattened and transposed and converted into a sequence of size n×d, where n=HW; and then multiplied with the learnable matrix to obtain 3 tensors Q, K, V, where , Q and K perform dot product to obtain the similarity of the current two targets; at the same time, in order to solve the degradation problem of the model, the similarity matrix X is obtained using equation (1);

[0019] ;

[0020] in, ; In equation (1), the transpose of Q and K is multiplied to obtain the similarity between pixels. Similarly, the self-attention is multiplied by the transpose of R of the input image to obtain ; is the context position in the input image, After the position is determined, the content of the enriched context is added to obtain the content information of the context according to the position; and then, the weighted attention feature map A is calculated as shown in equation (2); in equation (2), X is a similarity matrix, is the dimension of K; dividing X by is to perform a feature scaling operation on X to prevent its variance from being too large, resulting in gradient disappearance; it is normalized by Softmax to obtain the attention weight; finally, the result is weighted on the tensor V to obtain the feature A;

[0021] ;

[0022] There is usually information loss in the convolutional layer, which makes it difficult for the model to recover the position of the target from the feature space. Therefore, the present invention uses 2D relative position encoding for pairwise attention before Softmax. To add relative height and width information, in pixels With pixels Use formula (3) between

[0023] ;

[0024] in, is the Query vector of the pixel, is the pixel's Key vector, and

[0025] is the relative width , relative height Similar to the proposed self-attention mechanism, the relative width and height are calculated after the low-dimensional projection, so the self-attention formula for relative position embedding is:

[0026] ;

[0027] at this time, is a matrix of relative position logarithms along the height and width dimensions, satisfying and ; To avoid calculating pairs , the present invention considers the information of height and width respectively and When and .

[0028] The decoder stage of the TransUNet semantic segmentation model introduces an improved multi-scale fusion supervision structure. The design principle is: in the existing U-Net semantic segmentation network, the encoder is responsible for feature extraction and downsampling, while the decoder is responsible for feature fusion and upsampling. However, as the depth of the convolutional layer increases, the resolution of the feature map decreases, resulting in the loss of semantic information of low-level features such as edge details, causing the feature image to lose low-level features such as edge details. In the encoder part of TransUNet, a hybrid encoding method of CNN and Transformer with self-attention mechanism is adopted. However, Transformer is more focused on establishing long-term dependencies and may ignore low-level details and edge features. Although Skip jump connection is used in the decoding stage to partially compensate for this problem, the effect of improving the edge accuracy of the target area is limited.

[0029] In semantic segmentation models, more dense integration of information from features at different scales usually leads to higher segmentation accuracy. In order to solve the problem of missing low-level semantic features in the encoding and decoding process of the TransUNet network model, a multi-scale fusion supervision structure is introduced in its decoding stage. This structure supervises and utilizes the feature maps of each scale level in the decoding stage during network training. In this way, the network can more effectively integrate and utilize semantic information at different scale levels, thereby improving the ability to capture low-level details and edge features in the image, thereby improving the precision and accuracy of semantic segmentation.

[0030] A side output layer (SL) is added after the output of each layer in the decoding stage. Each side output layer consists of a 3×3 convolution layer, an upsampling layer and a Sigmoid activation function. The number of hidden feature channels output by the decoder is adjusted to correspond to the number of segmentation categories. At the same time, the generated segmentation probability map is upsampled by bilinear interpolation to restore it to the same size as the input image. After this processing, a segmentation probability map matching the size of the input image is obtained, which is used to show the probability of each pixel belonging to each category. A fusion layer (FL) is added to the final output part of the decoder. The fusion layer consists of a 3×3 convolution layer and a Sigmoid activation function, which is responsible for fusing the outputs of each side output layer to obtain the final segmentation result.

[0031] The improved multi-scale fusion supervision structure makes the network approximate multiple independent multi-scale prediction models by simultaneously training the five side outputs and the five results output by the fusion layer. This can make full use of the information of multiple scale features to improve the performance of the segmentation model, and this improvement does not introduce a large number of additional parameters.

[0032] The multi-scale fusion supervision structure simultaneously supervises the network output and the feature maps of the intermediate layers by simultaneously training the five side output feature maps of the decoder and these feature maps and the fusion results. The loss function of the network is designed to maximize the utilization of feature information at each scale and level, thereby improving the overall performance of the segmentation model. The loss function of the network is defined as follows:

[0033] ;

[0034] There are five side output layers in the network decoding stage, so the value of M in the above formula is 5; represents the loss of the output layer on the network side, Represents the loss of the feature map of the final fusion output.

[0035] When using the BCE loss function to optimize the network, the foreground samples are usually ignored and the background samples dominate, resulting in insufficient model feature learning and poor segmentation effect. The existing loss function also has the Dice loss function, which is expressed as follows:

[0036] ;

[0037] Among them, A is the positive class (foreground) predicted by the model, B is the positive class (foreground) of the true label, and are their sizes (number of pixels) respectively. This loss function not only focuses on whether the prediction result is correctly classified as foreground or background, but also considers the spatial overlap of the prediction, which can better guide the model to learn accurate segmentation boundaries and shapes. Compared with the BCE loss function, the Dice loss function can more effectively solve the problem of foreground samples being ignored and background samples being dominant, because it directly measures the spatial overlap of the predicted and true segmentation results, improves the model's ability to learn segmentation boundaries and details, and thus improves the segmentation effect. However, the Dice loss function is not friendly to model training. When the target area in the image is small, some pixel errors will cause large fluctuations in the loss value, affecting the training and convergence of the model.

[0038] To solve the above problems, the present invention proposes a hybrid balanced cross entropy loss function to optimize the network. and The common definitions are as follows:

[0039] ;

[0040] Where: N represents the total number of samples; and Respectively represent The label matrix and prediction matrix of samples; is the weight coefficient of the loss function; and They represent the balanced cross entropy loss function and the smoothed Dice loss function respectively, and the formulas are as follows:

[0041]

[0042] ;

[0043] In formula (8) is the pixel coordinate; The size of the image; H and W are the height and width;

[0044] and Represent the pixel values ​​of the label image and the predicted image respectively; is the introduced balance coefficient to prevent the model from having poor performance on categories with fewer samples during training; It is a minimum value, in order to prevent the extreme case where the denominator is 0, it plays the role of smoothing loss and gradient.

[0045] Furthermore, the prediction stage of the TransUNet semantic segmentation model in step A3 introduces a dislocation overlap prediction mechanism, that is, when scaling and cropping the remote sensing image, the image H and W directions are introduced with a segmentation stride, and the image of size (H, W) is segmented into a number of image blocks according to the set block size (h, w) and stride. The number of image blocks is:

[0046] ;

[0047] Before the segmentation step is used, the model can only see some independent areas, and the observation field of the image is small. After the segmentation step is introduced, the observation field of the model becomes larger, and the overall grasp of the image features is improved. After the image blocks processed by the misaligned image are sent to the model for prediction, the predicted probability images of each block can be overlapped and spliced; in the overlapping splicing process, the pixels of the non-overlapping part come from the prediction results of each block; for the pixels of the overlapping part, the prediction results of the overlapping area need to be merged; this merger usually uses weighted average or other statistical methods to ensure that the prediction results of different parts can be smoothly connected together to form the final full-image segmentation probability image. The overlay operation can improve the model's "observation" of the overall information of the remote sensing image, improve the accuracy of image segmentation, and improve the overall visualization effect.

[0048] Furthermore, the step S4 is specifically as follows:

[0049] B1: Division of urban building connectivity domain, including:

[0050] (1) Read the image, through In The function converts the image into a three-dimensional matrix;

[0051] (2) By Function converts image color space;

[0052] (3) By The function performs binarization on the image;

[0053] (4) By Function creates a labeled graph;

[0054] (5) By The function extracts connected domains. It analyzes each region of the image, marks the background as 0, and marks other regions with positive integers starting from 1. Finally, the marking results are returned to the label , and returns the information of the found connected domain: ,Label , , Center Point

[0055] B2: Obtaining the names of urban buildings

[0056] By point of interest ( , ) data to obtain the subject name information. The point of interest is the subject registration place on the map. Select the map conduct Data acquisition, The categories are divided into six categories: shopping malls, communities, factories, schools, governments, and hospitals. The obtained contents are coordinates, names and other information.

[0057] The present invention also provides a rooftop photovoltaic identification and potential assessment system based on improved TransUNet semantic segmentation, the system comprising a memory and a processor; wherein,

[0058] The memory is used to store computer program instructions that can be executed on the processor;

[0059] The processor is used to execute the steps of the rooftop photovoltaic identification and potential assessment method based on improved TransUNet semantic segmentation when running the computer program instructions.

[0060] The present invention also provides a computer storage medium, which stores a program for a method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation. When the program for a method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation is executed by at least one processor, the steps of the method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation are implemented.

[0061] Beneficial effects: Compared with the prior art, the present invention adopts an improved TransUNet semantic segmentation model, and uses the advantage of Transformer to establish long-term dependencies to improve the accuracy of satellite remote sensing image segmentation. An effective multi-head self-attention mechanism is used in the Transformer encoding and decoding process to focus on target features to establish feature mapping, thereby suppressing unimportant features. Secondly, in order to improve the segmentation accuracy of image edges, an improved multi-scale fusion supervision structure is introduced in the decoder stage of TransUNet, and the network is trained using a fusion supervision loss function. At the same time, in order to solve the problem of feature missing caused by satellite remote sensing image segmentation, a dislocation overlap prediction mechanism is proposed in the prediction stage of the model. The introduction of a hybrid model can optimize the building outline, eliminate noise information, and extract regular building roof outlines to support subsequent solar energy potential calculations. The area, orientation and type are calculated based on the extracted building roof data, and the roof morphology changes are simulated through grid division, paving the way for subsequent roof photovoltaic potential analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the process of the present invention;

[0063] Figure 2 Schematic diagram of the residual module structure;

[0064] Figure 3 Display graph for connected domain image;

[0065] Figure 4 It is the structural diagram of the U-Net model;

[0066] Figure 5 It is the residual U structure diagram;

[0067] Figure 6 This is the structural diagram of the U2-Net model;

[0068] Figure 7 This is a structural diagram of the Transformer module with a multi-head self-attention mechanism;

[0069] Figure 8 This is a schematic diagram of the structure of the combination of U-Net and Transformer;

[0070] Fig. 9 Schematic diagram of the multi-scale fusion supervision structure in the model. DETAILED DESCRIPTION

[0071] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0072] like Figure 1 As shown, the present invention provides a rooftop photovoltaic identification and potential assessment method based on improved TransUNet semantic segmentation, comprising the following steps:

[0073] S1: Obtain building roof satellite image dataset: Download high-resolution building roof satellite image dataset from public maps, and obtain roof dataset information based on satellite images;

[0074] S2: Clean and classify the images in the dataset according to whether they have roof features, and separate the images with roofs for identification;

[0075] S3: Based on the images with roofs, the TransUNet semantic segmentation model is built and trained to extract roof features, identify the roof area of ​​the building and divide it into regions according to the roof structure, material, and color;

[0076] S4: Match the image processed in step S3 with the subject name to locate the subject to which the building belongs.

[0077] The specific requirements for data collection in step S1 of this embodiment are as follows:

[0078] For the purpose of identifying and dividing building roofs, satellite images of building roofs are collected. To ensure the accuracy of the identification results, the following standards must be followed: the satellite images are as clear as possible, and when the data set is annotated, the number of data sets accounts for no less than 10% of the original data, and the number of each category is balanced; geographic location information is included to facilitate roof location positioning; satellite images are regular and consistent in size to facilitate splicing and overall visualization; satellite image data within administrative divisions are collected, including different types of scenes. This satellite image is a high-resolution satellite image that can provide clear geographic information and can be used for map making, road network analysis, terrain analysis and many other purposes.

[0079] Data cleaning in step S2 of this embodiment is an important part of the data analysis process. Data quality is directly related to the model effect. Since the acquired satellite image contains data within the entire administrative area, including a large number of images without buildings, it may take a long time for the entire model to perform roof recognition, and it may also cause the recognition model to misrecognize fields with regular shapes. Therefore, it is necessary to design a model for classifying satellite images to clean the data. Satellite images are divided into two categories, those containing building roofs and those not containing building roofs. The building satellite image classification model used in this embodiment is based on deep learning technology and applies convolutional neural networks ( ) is combined with remote sensing image processing technology to extract spatial features in satellite images and convert them into high-level feature expressions for building classification.

[0080] Based on the above content, step S2 is specifically as follows:

[0081] 1) Dataset preparation: collect image data and prepare labels. Labels usually refer to the category to which an image belongs. The purpose of this model is to separate buildings with roofs from buildings without roofs, so the image classification model is divided into two categories, images with buildings and images without buildings. The acquired satellite images are classified. Among them, images with buildings are often rich in colors and the image memory size is often large. Therefore, the images are pre-classified, and the images with the memory size in the middle part are manually labeled.

[0082] 2) Feature extraction: Feature extraction refers to the process of converting an image into digital features. The network is used for image classification model, The structure of contains 50 convolutional layers, pooling layers, fully connected layers and residual modules, where each residual module consists of two convolutional layers and a skip connection. The skip connection directly bypasses the input signal through a certain number of convolutional layers and pooling layers, and adds the input signal to the output signal after the skip connection to form a residual signal. The residual module structure is as follows Figure 2 shown.

[0083] 3) Model training: The features extracted from satellite images are input into the classification model for training. The whole training process is carried out for 500 , that is, the entire training data set was trained for 500 rounds. After repeated training and tuning, the final model achieved a high accuracy on the test set, with a classification accuracy of 91.78%, a roof accuracy of 85.32%, and a non-roof accuracy of 95.81%. The accuracy directly affects the actual application effect and reliability of the model. Therefore, it is necessary to continuously optimize the model during the training process to improve the classification accuracy to meet the needs of practical applications. The confusion matrix of the roof classification model is shown in Table 1.

[0084] Table 1 Confusion matrix of roof classification model

[0085]

[0086] Model testing, after the model training is completed, the model needs to be tested to evaluate its performance. The test set is a data set that has not been touched during the model training process, which can be used to test the generalization ability of the model. Some indicators are usually used to evaluate the performance of the model on the test set, such as accuracy, recall rate and precision. Accuracy refers to the ratio of the number of correctly classified samples to the total number of samples, recall rate refers to the ratio of the number of correctly classified positive samples to the number of true positive samples, and precision refers to the ratio of the number of correctly classified positive samples to the number of all samples classified as positive examples. These indicators can help evaluate the classification performance of the model in different situations, thereby guiding subsequent model selection and application. In the satellite image test data of other cities in this embodiment, the accuracy rate of buildings is 84.23%, and the accuracy rate of no buildings is 92.72%.

[0087] The construction of the TransUNet semantic segmentation model in step S3 of this embodiment is specifically as follows:

[0088] A1: Combine the residual network with the U-Net model to form a residual U structure;

[0089] A2: Stacking residual U structures with different depths, including the down-sampling encoding part and the up-sampling decoding part, to build a deeper U-shaped semantic segmentation network U 2 -Net;

[0090] A3: Introduce the Transformer module with multi-head self-attention mechanism into U 2 -Net network model to obtain the TransUNet semantic segmentation model.

[0091] In step A1, the U-Net model was improved, refer to Figure 4 The encoder of the U-Net model consists of five parts. The encoder extracts the features of the input image and reduces the resolution through four downsampling blocks and a channel adjustment module. Each module contains two 3×3 convolutional layers, and the downsampling block also includes an additional 2×2 pooling layer, which helps to further reduce the size of the feature map; the decoder consists of four parts, which are responsible for the resolution restoration and feature integration of the feature map. Each part contains an upsampling layer and two 3×3 convolutional layers; the decoder and the encoder use jump connections to allow information to be passed directly from the encoding stage to the decoding stage, so as to retain and utilize more feature information, which helps to improve the accuracy and stability of the segmentation model;

[0092] As the number of network layers increases, the model will begin to degrade, the performance of the network begins to decline, and the loss is difficult to converge. A new type of residual module is used in the residual network structure. Figure 2 , the output of the first convolutional layer of each module is directly added to the final output of the encoder. Specifically, the residual network structure introduces a shortcut connection, that is, the output of the previous layer is directly added to the input of the current layer. This allows the network to learn the residual function (that is, the difference between the current layer output and the ideal target) more effectively. This method makes it easier to learn each layer, because the network can focus more on learning the residual instead of directly learning complex mapping relationships, effectively solving the degradation problem of deep networks. Based on the residual network and combined with U-Net, the following is proposed Figure 5 The residual U structure shown.

[0093] Each residual U structure in step A2 is actually an improved U-Net, which is very similar to the residual block. It constructs a deeper U-shaped semantic segmentation network U by stacking residual U structures with different depths, including the down-sampled encoding part and the up-sampled decoding part. 2 -Net model, such as Figure 6 As shown;

[0094] A new type of residual module is used in the residual U structure. The output of the first convolutional layer of each module is directly added to the final output of the encoder, which is similar to the shortcut connection in ResNet. This operation ensures that the contextual semantic information of the feature map will not be lost due to the stacking of convolutional layers during the downsampling process. At the same time, the U-shaped stacking of the residual U structure enables the entire network to obtain a larger receptive field and richer global features and semantic information at each level. The network can extract semantic features from feature maps with information of any scale and effectively alleviate the problem of detail loss caused by direct upsampling of feature maps.

[0095] In step A3, the self-attention mechanism of the Transformer module is improved, and an effective multi-head self-attention mechanism is applied to the network in encoding and decoding. Figure 7 As shown in Figure 2, (a) is the Transformer module, (b) and (c) are the multi-head self-attention encoding-decoding branches. Figure 7 , introducing the Transformer module with multi-head self-attention mechanism 2-Net network model, the network structure divides the input features into 5 branches for processing. The first 3 branches use the self-attention mechanism to generate Query, Key, and Value vectors respectively, which are used to calculate the correlation between each pixel and all other pixels in the feature map. This method can increase the receptive field of the network in the feature extraction process; the decoder branch combines the backbone network with the self-attention mechanism to add the input features to the Query vector to better understand the relationship and contextual information between each pixel in the input image, and increase the receptive field of feature extraction; the last two branches focus on the height and width information of the input image, and introduce this information into the self-attention mechanism in a weighted manner. This helps the network learn the relative position relationship and content between each feature in the remote sensing image, thereby improving the accuracy and efficiency of semantic segmentation and position perception.

[0096] The multi-head self-attention mechanism establishes feature mapping for useful target features, thereby suppressing unimportant features. The self-attention mechanism is calculated as follows:

[0097] Given an input tensor , Query, Key and Value are convolved 1×1, and 3 1×1 convolutions are used to project the feature map for query, key and value embedding: Query, Key, Value , where d is the size of each head embedded; then Query, Key and Value are flattened and transposed and converted into a sequence of size n×d, where n=HW; and then multiplied with the learnable matrix to obtain 3 tensors Q, K, V, where , Q and K perform dot product to obtain the similarity of the current two targets; at the same time, in order to solve the degradation problem of the model, the similarity matrix X is obtained using equation (1);

[0098] ;

[0099] in, ; In equation (1), the transpose of Q and K is multiplied to obtain the similarity between pixels. Similarly, the self-attention is multiplied by the transpose of R of the input image to obtain ; is the context position in the input image, After the position is determined, the content of the enriched context is added to obtain the content information of the context according to the position; and then, the weighted attention feature map A is calculated as shown in equation (2); in equation (2), X is a similarity matrix, is the dimension of K; dividing X by is to perform a feature scaling operation on X to prevent its variance from being too large, resulting in gradient disappearance; it is normalized by Softmax to obtain the attention weight; finally, the result is weighted on the tensor V to obtain the feature A;

[0100] ;

[0101] There is usually information loss in the convolutional layer, which makes it difficult for the model to recover the position of the target from the feature space. Therefore, the present invention uses 2D relative position encoding for pairwise attention before Softmax. To add relative height and width information, in pixels With pixels Use formula (3) between

[0102] ;

[0103] in, is the Query vector of the pixel, is the pixel's Key vector, and is the relative width , relative height Similar to the proposed self-attention mechanism, the relative width and height are calculated after the low-dimensional projection, so the self-attention formula for relative position embedding is:

[0104] ;

[0105] at this time, is a matrix of relative position logarithms along the height and width dimensions, satisfying and ; To avoid calculating pairs , the present invention considers the information of height and width respectively and When and .

[0106] The introduction of the Transformer encoder structure with self-attention mechanism improves the global feature extraction capability of the CNN network by introducing the improved Transformer encoder structure in the encoding stage of CNN. The network can obtain the long-term dependencies of image features, which effectively improves the performance of the model. The TransUNet semantic segmentation model is obtained by mixing the improved U-Net and Transformer models. In the encoding stage, TransUNet uses the feature map patch extracted by the convolutional neural network and encoded and tokenized as the input of the Transformer to obtain the global dependencies of image features; on the other hand, in the decoding stage, the output features of the Transformer are upsampled and combined with the feature maps of each level output by CNN to achieve accurate positioning and segmentation of the target area. The specific scheme of combining U-Net and Transformer can be referred to as follows. Figure 8 The network structure shown.

[0107] In step A3, the decoder stage of the TransUNet semantic segmentation model introduces an improved multi-scale fusion supervision structure. The design principle is: in the existing U-Net semantic segmentation network, the encoder is responsible for feature extraction and downsampling, while the decoder is responsible for feature fusion and upsampling. However, as the depth of the convolutional layer increases, the resolution of the feature map decreases, resulting in the loss of semantic information of low-level features such as edge details, causing the feature image to lose low-level features such as edge details. In the encoder part of TransUNet, a hybrid encoding method of CNN and Transformer with self-attention mechanism is adopted. However, Transformer is more focused on establishing long-term dependencies and may ignore low-level details and edge features. Although Skip jump connection is used in the decoding stage to partially compensate for this problem, the effect of improving the edge accuracy of the target area is limited.

[0108] In semantic segmentation models, more dense integration of information from features at different scales usually leads to higher segmentation accuracy. In order to solve the problem of missing low-level semantic features in the encoding and decoding process of the TransUNet network model, a multi-scale fusion supervision structure is introduced in its decoding stage. This structure supervises and utilizes the feature maps of each scale level in the decoding stage during network training. In this way, the network can more effectively integrate and utilize semantic information at different scale levels, thereby improving the ability to capture low-level details and edge features in the image, thereby improving the precision and accuracy of semantic segmentation. The specific structure diagram is shown in the figure below. Fig. 9 As shown in the dashed box.

[0109] Reference Fig. 9, after each layer output in the decoding stage, a side output layer (SL) is added. Each side output layer consists of a 3×3 convolution layer, an upsampling layer and a Sigmoid activation function. The number of hidden feature channels output by the decoder is adjusted to correspond to the number of segmentation categories. At the same time, the generated segmentation probability map is upsampled by bilinear interpolation to restore it to the same size as the input image. After this processing, a segmentation probability map matching the size of the input image is obtained, which is used to show the probability of each pixel belonging to each category. A fusion layer (FL) is added to the final output part of the decoder. The fusion layer consists of a 3×3 convolution layer and a Sigmoid activation function, which is responsible for fusing the outputs of each side output layer to obtain the final segmentation result.

[0110] The improved multi-scale fusion supervision structure makes the network approximate multiple independent multi-scale prediction models by simultaneously training the five side outputs and the five results output by the fusion layer. This can make full use of the information of multiple scale features to improve the performance of the segmentation model, and this improvement does not introduce a large number of additional parameters.

[0111] The multi-scale fusion supervision structure simultaneously supervises the network output and the feature maps of the intermediate layers by simultaneously training the five side output feature maps of the decoder and these feature maps and the fusion results. The loss function of the network is designed to maximize the utilization of feature information at each scale and level, thereby improving the overall performance of the segmentation model. The loss function of the network is defined as follows:

[0112]

[0113] The network decoding stage has five side output layers, such as Fig. 9 As shown in SL1, SL2...SL5, the value of M in the above formula is 5; represents the loss of the output layer on the network side, Represents the loss of the feature map of the final fusion output.

[0114] When using the BCE loss function to optimize the network, the foreground samples are often ignored and the background samples dominate, resulting in insufficient model feature learning and poor segmentation results. The existing loss function also has the Dice loss function, which is expressed as follows:

[0115] ;

[0116] Among them, A is the positive class (foreground) predicted by the model, B is the positive class (foreground) of the true label,

[0117] and are their sizes (number of pixels) respectively. This loss function not only focuses on whether the prediction result is correctly classified as foreground or background, but also considers the spatial overlap of the prediction, which can better guide the model to learn accurate segmentation boundaries and shapes. Compared with the BCE loss function, the Dice loss function can more effectively solve the problem of foreground samples being ignored and background samples being dominant, because it directly measures the spatial overlap of the predicted and true segmentation results, improves the model's ability to learn segmentation boundaries and details, and thus improves the segmentation effect. However, the Dice loss function is not friendly to model training. When the target area in the image is small, some pixel errors will cause large fluctuations in the loss value, affecting the training and convergence of the model.

[0118] To solve the above problems, this embodiment proposes a hybrid balanced cross entropy loss function to optimize the network training. and The common definitions are as follows:

[0119] ;

[0120] Where: N represents the total number of samples; and Respectively represent The label matrix and prediction matrix of samples; is the weight coefficient of the loss function; and They represent the balanced cross entropy loss function and the smoothed Dice loss function respectively, and the formulas are as follows:

[0121]

[0122] ;

[0123] In formula (8) is the pixel coordinate; The size of the image; H and W are the height and width;

[0124] and Represent the pixel values ​​of the label image and the predicted image respectively; is the introduced balance coefficient to prevent the model from having poor performance on categories with fewer samples during training; It is a minimum value, in order to prevent the extreme case where the denominator is 0, it plays the role of smoothing loss and gradient.

[0125] The prediction stage of the TransUNet semantic segmentation model in step A3 introduces a misaligned overlapping prediction mechanism, specifically:

[0126] Due to the different resolutions of remote sensing satellite images and the fixed network structure of the model, the image size of the input neural network is generally 224×224, 512×512 or 600×600, etc., which requires the remote sensing image to be scaled and cropped. To avoid distortion, the aspect ratio of the image should be maintained when adjusting the size. At the same time, it is usually necessary to preprocess and standardize the image to ensure that all images have a consistent data range and format, which may include operations such as pixel value normalization (for example, scaling the pixel value to between 0 and 1 or using a specific mean and standard deviation for standardization). However, these operations will destroy the semantic information of the image to a certain extent. In response to the above problems, this embodiment proposes a dislocation overlap prediction mechanism, that is, when scaling and cropping the remote sensing image, the H and W directions of the image introduce a segmentation step (stride), and the image of size (H, W) is divided into several image blocks according to the set block size (h, w) and stride. The number of image blocks is:

[0127] ;

[0128] Before the segmentation step is used, the model can only see some independent areas, and the observation field of the image is small. After the segmentation step is introduced, the observation field of the model becomes larger, and the overall grasp of the image features is improved. After the image blocks processed by the misaligned image are sent to the model for prediction, the predicted probability images of each block can be overlapped and spliced; in this process, the pixels of the non-overlapping part come from the prediction results of each block; for the pixels of the overlapping part, the prediction results of the overlapping area need to be merged; this merger usually uses weighted average or other statistical methods to ensure that the prediction results of different parts can be smoothly connected together to form the final full-image segmentation probability image. The overlay operation can improve the model's "observation" of the overall information of the remote sensing image, improve the accuracy of image segmentation, and improve the overall visualization effect.

[0129] Based on the above content, step S3 mainly involves the following parts:

[0130] 1. Data labeling: Data labeling is to label satellite images with various types of roofs, add labels to the data to be labeled, and form readable data codes that meet the model training requirements. In order to meet the requirements of data labeling, a pre-trained model is used to pre-label the image. For places where the labeling is inaccurate or erroneous, a series of green points (positive points) and red points (negative points) are used to accurately adjust the edge of the target object. In order to ensure the training effect, the balance between various types of data and the sufficiency of various types of training samples should be fully guaranteed. Various types of typical roofs are selected for labeling, and the number of labels is not less than 10% of the satellite image to be identified. In this embodiment, urban buildings are mainly divided into five types of roofs: residential roofs, commercial park roofs, urban high-rise roofs, industrial plant roofs, and other roofs.

[0131] 2. Selection and improvement of the semantic segmentation method model of the image. This embodiment combines the residual structure with the U2-Net model to adapt to the task of roof classification, recognition and segmentation. Combining U-Net on the basis of the residual network is actually an improved U-Net, but its expression is very similar to the residual block. By nesting the residual U structures of different depths in the downsampling stage of the encoder and the upsampling stage of the decoder, a deeper U-shaped semantic segmentation network U is formed. 2 -Net, in which the output of the first convolutional layer is directly integrated into the final output of the encoder to ensure that contextual semantic information is not lost during the downsampling process, thereby effectively maintaining the integrity of the feature map. At the same time, the U-shaped structure constructed by connecting multiple residual U structures in series enhances the entire network's ability to capture a wider range of perception and multi-level semantic information, enabling it to effectively extract semantic features from feature maps containing information of any scale. This design also effectively alleviates the problem of detail loss that may be caused by directly upsampling feature maps.

[0132] 3. Improvement of Transformer self-attention mechanism, using efficient Multi-Head Self-Attention (MHSA) mechanism, and applying it to the network in encoding and decoding. Since satellite remote sensing images are highly structured data, most pixels in the local high-resolution feature map outside the image boundary are similar, so adding paired multi-head self-attention mechanisms in the encoder and decoder can solve the inefficiency and redundancy of its calculation.

[0133] 4. The introduction of the Transformer encoder structure with self-attention mechanism, by embedding the improved Transformer mechanism in the encoding stage of CNN, enhances the model's ability to grasp the global features of the image, effectively improves its understanding of long-term dependencies, and thus significantly improves the performance of the semantic segmentation model. This fusion method creates an improved TransUNet model, combining the advantages of U-Net and Transformer, and improves each module to adapt to the characteristics of large amounts of remote sensing image data, bringing significant progress in the task of remote sensing image segmentation.

[0134] 5. Improvement of the decoder. In view of the problem of boundary information loss and different resolutions of remote sensing images in the TransUNet semantic segmentation model proposed above, the present invention introduces a multi-scale feature fusion supervision structure to enhance the segmentation ability of the model. At the same time, the present invention designs an optimized fusion supervision loss function to effectively guide model learning and optimize feature fusion at various scales, and proposes a misaligned overlapping prediction method in the prediction stage to solve the above problems and improve the segmentation effect of the model.

[0135] Step S4 is specifically as follows:

[0136] B1: Division of urban building connectivity domain:

[0137] For the results obtained after semantic segmentation, in order to further process and analyze them, it is necessary to group the pixels into different connected domains. A connected domain refers to an area composed of pixels with the same pixel value, and the pixels in this area can be connected to each other by continuous straight lines, curves or other means. For a connected domain, it is usually necessary to further analyze and process it, such as calculating the area, perimeter, center point position, etc. of the connected domain. Therefore, dividing the connected domain is an important step after semantic segmentation. An image connected domain refers to a set of pixels composed of adjacent pixels. Image connected domain detection is an important issue in image processing, which can be used to separate different objects, calculate the number and size of objects, etc. In an image, a connected domain is usually used to separate objects. For a binary image, a connected domain can be composed of pixels with a pixel value of 1. For a grayscale image, a connected domain can be composed of pixels with the same or very close pixel values. In image processing, connected domain labels are usually used to represent different connected domains, and each connected domain is assigned a unique label.

[0138] In this embodiment, the connected domain division specifically includes:

[0139] (1) Read the image, through In The function converts the image into a three-dimensional matrix;

[0140] (2) By Function converts image color space;

[0141] (3) By The function performs binarization on the image;

[0142] (4) By Function creates a labeled graph;

[0143] (5) By The function extracts connected domains. It analyzes each region of the image, marks the background as 0, and marks other regions with positive integers starting from 1. Finally, the marking results are returned to the label , and returns the information of the found connected domain: ,Label , , Center Point Its connected domain image is as follows Figure 3 shown.

[0144] B2: Obtaining the names of urban buildings

[0145] After semantic segmentation of satellite images of urban buildings, in order to accurately locate the subject to which the building belongs, it is necessary to match the building with the subject name, and thus obtain the subject name information. The subject name information is obtained through the point of interest ( , ) data is obtained, and points of interest are the registered places of entities on the map, such as stores, gas stations, hospitals, bus stations, etc. Data can enable research on spatiotemporal behavior, urban planning, geographic information, etc. In this example, maps are selected conduct Data acquisition, The categories are divided into six categories: shopping malls, communities, factories, schools, governments, and hospitals; the contents obtained are coordinates, names and other information.

[0146] This embodiment also provides a rooftop photovoltaic identification and potential assessment system based on improved TransUNet semantic segmentation, the system comprising a network interface, a memory and a processor; wherein the network interface is used to realize the reception and transmission of signals in the process of sending and receiving information between other external network elements; the memory is used to store computer program instructions that can be run on the processor; the processor is used to execute the steps of the above-mentioned consensus method when running the computer program instructions.

[0147] The present embodiment also provides a computer storage medium, which stores a computer program, and the method described above can be implemented when the processor executes the computer program. The computer readable medium can be considered to be tangible and non-temporary. Non-limiting examples of non-temporary tangible computer-readable media include non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tapes or hard drives) and optical storage media (such as CDs, DVDs or Blu-ray discs), etc. The computer program includes processor executable instructions stored on at least one non-temporary tangible computer-readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, a device driver that interacts with a specific device of a special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. A rooftop photovoltaic identification and potential assessment method based on improved TransUNet semantic segmentation, characterized in that: The steps include: S1: Get the satellite image dataset of building roofs; S2: Clean and classify the images in the dataset according to whether they have roof features, and separate the images with roofs; S3: Based on the images with roofs, the roof features are extracted by building and training the TransUNet semantic segmentation model, and the roof area of ​​the building is identified and divided into regions; S4: Match the image processed in step S3 with the subject name to locate the subject to which the building belongs.

2. According to claim 1, a rooftop photovoltaic identification and potential assessment method based on improved TransUNet semantic segmentation is characterized in that: In step S2, The network is used as an image classification model to classify images. The network structure contains 50 convolutional layers, pooling layers, fully connected layers and residual modules, where each residual module consists of two convolutional layers and a skip connection; the skip connection directly bypasses the input signal through a certain number of convolutional layers and pooling layers, and adds the input signal to the output signal after the skip connection to form a residual signal.

3. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 1 is characterized in that: The construction of the TransUNet semantic segmentation model in step S3 is as follows: A1: Combine the residual network with the U-Net model to form a residual U structure; A2: Stack the residual U structures with different depths, including the down-sampling encoding part and the up-sampling decoding part, to construct U 2 -Network model; A3: Introduce the Transformer module with multi-head self-attention mechanism into U 2 -Net network model to obtain the TransUNet semantic segmentation model.

4. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 3 is characterized in that: The encoder of the U-Net model in step A1 consists of five parts. The encoder extracts the features of the input image and reduces the resolution through four downsampling blocks and a channel adjustment module. Each module contains two 3×3 convolutional layers, and the downsampling block also includes an additional 2×2 pooling layer; the decoder consists of four parts, which are responsible for the resolution restoration and feature integration of the feature map. Each part contains an upsampling layer and two 3×3 convolutional layers; Skip connections are used between the decoder and the encoder, allowing information to be passed directly from the encoding stage to the decoding stage; A new type of residual module is used in the residual network structure. The output of the first convolutional layer of the residual module is directly added to the final output of the encoder. By introducing a shortcut connection, the output of the previous layer is directly added to the input of the current layer.

5. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 3 is characterized in that: U of the Transformer module with multi-head self-attention mechanism introduced in step A3 2 -Net network model, the network structure divides the input features into 5 branches for processing; the first 3 branches use the self-attention mechanism to generate Query, Key, and Value vectors respectively, which are used to calculate the correlation between each pixel and all other pixels in the feature map. The decoder branch is combined with the self-attention mechanism through the backbone network to add the input features to the Query vector in order to better understand the relationship and contextual information between pixels in the input image and increase the receptive field of feature extraction; the last two branches focus on the height and width information of the input image, and introduce this information into the self-attention mechanism in a weighted manner.

6. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 5 is characterized in that: In step A3, the multi-head self-attention mechanism creates feature maps for useful target features, thereby suppressing unimportant features. The self-attention mechanism is calculated as follows: Given an input tensor , Query, Key and Value are convolved 1×1, and 3 1×1 convolutions are used to project the feature map for query, key and value embedding: Query, Key, Value , where d is the size of each head embedded; then Query, Key and Value are flattened and transposed and converted into a sequence of size n×d, where n=HW; and then multiplied with the learnable matrix to obtain 3 tensors Q, K, V, where , Q and K perform dot product to obtain the similarity of the current two targets; at the same time, in order to solve the degradation problem of the model, the similarity matrix X is obtained using equation (1); ； in, ; In equation (1), the transpose of Q and K is multiplied to obtain the similarity between pixels. Similarly, the self-attention is multiplied by the transpose of R of the input image to obtain ; is the context position in the input image, After the position is determined, the content of the enriched context is added to obtain the content information of the context according to the position; and then, the weighted attention feature map A is calculated as shown in equation (2); in equation (2), X is the similarity matrix, which is the dimension of K; it is normalized by Softmax to obtain the attention weight; finally, the result is weighted on the tensor V to obtain the feature A; ； Use 2D relative position encoding for pairwise attention before Softmax To add relative height and width information, in pixels With pixels Use formula (3) between ； in, is the Query vector of the pixel, is the pixel's Key vector, and Is relatively wide , relative height The learning embedding of ; the relative width and height are calculated after the low-dimensional projection, then the self-attention formula for the relative position embedding is: ； at this time, is a matrix of relative position logarithms along the height and width dimensions, satisfying and ; To avoid calculating pairs , considering the height and width information respectively and When and .

7. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 3 is characterized in that: In step A3, an improved multi-scale fusion supervision structure is introduced in the decoder stage of the TransUNet semantic segmentation model. Specifically, a side output layer is added after each layer output in the decoding stage. Each side output layer consists of a 3×3 convolution layer, an upsampling layer, and a Sigmoid activation function. The number of hidden feature channels output by the decoder is adjusted to correspond to the number of segmentation categories. At the same time, the generated segmentation probability map is upsampled by bilinear interpolation to restore it to the same size as the input image. After this processing, a segmentation probability map matching the size of the input image is obtained, which is used to show the probability of each pixel belonging to each category. A fusion layer is added to the final output part of the decoder. The fusion layer consists of a 3×3 convolution layer and a Sigmoid activation function. It is responsible for fusing the outputs of the output layers on each side to obtain the final segmentation result.

8. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 7 is characterized in that: In step A3, based on the multi-scale fusion supervision structure, the network is trained using the fusion supervision loss function, specifically: The network is optimized using a hybrid balanced cross entropy loss function, which is defined as follows: ； Where: N represents the total number of samples; and Respectively represent The label matrix and prediction matrix of samples; is the weight coefficient of the loss function; and They represent the balanced cross entropy loss function and the smoothed Dice loss function respectively, and the formulas are as follows: ； ； In formula (8) is the pixel coordinate; The size of the image; H and W are the height and width; and Represent the pixel values ​​of the label image and the predicted image respectively; is the introduced balance coefficient; in formula (9) is a minimum value.

9. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 3 is characterized in that: The prediction stage of the TransUNet semantic segmentation model in step A3 introduces a dislocation overlap prediction mechanism, that is, when scaling and cropping the remote sensing image, the segmentation step size is introduced in the H and W directions of the image, and the image of size (H, W) is divided into several image blocks according to the set block size (h, w) and step size. The number of image blocks is: ； After the image blocks processed by the offset cutting are sent to the model for prediction, the predicted probability images of each block are overlapped and spliced; in the overlapping splicing process, the pixels of the non-overlapping part come from the prediction results of each block; for the pixels of the overlapping part, the prediction results of the overlapping area are merged to form the final full-image segmentation probability image.

10. The method for rooftop photovoltaic identification and potential assessment based on improved TransUNet semantic segmentation according to claim 1, characterized in that: The step S4 is specifically as follows: B1: Division of urban building connectivity domain, including: (1) Read the image, through In The function converts the image into a three-dimensional matrix; (2) By Function converts image color space; (3) By The function performs binarization on the image; (4) By Function creates a labeled graph; B2: Obtaining the names of urban buildings By Point of Interest Data obtains the subject name information. The point of interest is the subject registration place on the map. Select the map conduct Data acquisition, The categories are divided into six categories: shopping malls, communities, factories, schools, governments, and hospitals.

Images