A photovoltaic panel segmentation method based on deep learning technology

Abstract

A photovoltaic panel segmentation method based on deep learning technology. It mainly solves the current problems of missed detection and false detection of photovoltaic panels and unclear edge segmentation of photovoltaic panels. The invention introduces channel self-attention to effectively capture the channel information obtained by downsampling, and improves the global convolution and boundary refinement modules to effectively capture the spatial information of feature maps. Finally, the experimental results on the public photovoltaic dataset show that the invention performs well in the segmentation of photovoltaic panels in remote sensing images, and is applicable to remote sensing images of multiple different spatial resolutions.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a photovoltaic panel segmentation method based on deep learning technology. Background Art

[0002] Photovoltaics are divided into centralized photovoltaics and distributed photovoltaics according to the density of installation. Distributed photovoltaics are installed in scattered locations, usually on roofs and in small areas. Therefore, collecting information such as photovoltaic installation locations and photovoltaic areas on site requires intensive labor and a lot of time. Satellite images cover a wide range and have strong generalization, which is conducive to grasping the dynamic changes of ground phenomena. Images taken by satellites and drones also have excellent applications in roads, buildings, marine life, farmland and other fields. Therefore, the use of satellite images can solve the problem of difficulty in obtaining photovoltaic information and can update the latest photovoltaic installation status in a timely manner.

[0003] Since LiDAR (Light Detection and Ranging) can easily extract the surface of objects, it has been widely used for rooftop or photovoltaic extraction. Nguyen et al. applied light detection and ranging (LiDAR) to automatic solar photovoltaic (PV) deployment analysis at regional scale, and maintained good PV identification accuracy for low-resolution, incomplete data samples. Martín-Jiménez et al. combined LiDAR and orthophoto data processing with a new automatic method to geometrically characterize roofs with slope levels and calculate their photovoltaic power generation potential. Due to the high standards for data, these methods lack flexibility in dealing with complex rooftop extraction problems. With the rapid development of machine learning, more and more technologies are used for rooftop photovoltaic identification, and the training samples for the model are not too strict. Malof et al. used image processing methods and machine learning methods to detect rooftop photovoltaic devices in high-resolution remote sensing images, providing an effective solution for fast and scalable rooftop photovoltaic information collection. Joshi et al. used support vector machine algorithms to detect rooftops of solar photovoltaic deployment in Abu Dhabi through satellite / aerial images. However, the shapes and colors of objects of different categories have strong similarities, and the robustness of photovoltaic identification using traditional machine learning methods is not strong.

[0004] With the widespread application of deep learning in the field of remote sensing, semantic segmentation methods can quickly and accurately extract photovoltaic panels from remote sensing images, thereby realizing the extraction of photovoltaic panels. Jie et al. implemented the semantic segmentation network and edge detection network in parallel, and introduced a gating fusion module to improve the model's ability to identify small photovoltaic panels in response to the problem that small photovoltaic panels are difficult to identify. Wu Yongjing et al. used the DeepLab V3+ model with ResNeSt-50 as the backbone network to identify and segment centralized photovoltaics, and used image science to correct the segmentation results, which can improve the accuracy of photovoltaic recognition. However, the model did not pay attention to too many shallow features after simple upsampling. Therefore, the segmentation results of this method are prone to treat greenhouses and cultivated land as targets, and there is a situation where the background is fused with the background and there is no significant graphic boundary. In general, the semantic segmentation method based on deep learning has achieved good results in the study of photovoltaic extraction, but the interference of complex backgrounds in remote sensing images and the low spatial resolution of remote sensing images can easily lead to missed detection and false detection of photovoltaic panels and unclear edge segmentation of photovoltaic panels. Therefore, it is necessary to further improve the model's ability to segment photovoltaic panels. Summary of the invention

[0005] In order to overcome the shortcomings of the background technology, the present invention provides a photovoltaic panel segmentation method based on deep learning technology, which mainly solves the current problems of missed detection and false detection of photovoltaic panels and unclear edge segmentation of photovoltaic panels.

[0006] The technical solution adopted by the present invention is:

[0007] A photovoltaic panel segmentation method based on deep learning technology includes the following steps:

[0008] Step 1: The regional images taken by drones or satellites include three sub-datasets with different resolutions, which are represented by PV08, PV03 and PV01 respectively;

[0009] Step 2: All three data sets are converted into RGB images with corresponding labels. After data enhancement, all images are divided into training set, validation set, and test set in a ratio of 7:2:1.

[0010] Step 3: Based on the deep learning network model of the encoder-decoder structure, a symmetrical global convolution module and boundary refinement module are constructed. The upsampling operation of the decoder part is to first perform bilinear interpolation on the input feature map to double the size of the original feature map, and then use 3×3 convolution and ReLU functions to capture local information; the downsampling operation uses maximum pooling to half the original size;

[0011] Step 4: Model training: start model training based on the training set and validation set in the sorted photovoltaic panel data set and the constructed model;

[0012] Step 5: Input the sorted PV01 test data set, PV03 test data set, and PV08 test data set into the trained model network to obtain the predicted segmentation result map.

[0013] The boundary refinement module is a BR module fusion module, which includes multiple BR modules connected in series or densely.

[0014] In the BR module, GCM and IBRM are connected in series, and then IBRM and GCM are connected in series, and finally the two are connected in parallel.

[0015] The global convolution module uses a parallel symmetric method and different split combinations to supplement some missing global features. The size value K of the global convolution kernel is 9.

[0016] The model also includes a channel self-attention mechanism, which measures the importance of these channels by the correlation between each channel. The input feature map is passed through the channel self-attention mechanism to generate a feature map with channel self-attention weights.

[0017] It also includes a model validation step, as shown in the following formula

[0018]

[0019] The pixel accuracy PA, intersection over union (IoU), and mean intersection over union (Mean IoU) are used as evaluation criteria; assuming there are k+1 categories, i represents the true value, and j represents the predicted value P ii represents TP, i.e., the number of pixels predicted as class i; P ij represents FN, i.e., the number of pixels of class i predicted as class j; P ji represents FP, that is, the number of pixels of class j predicted as class i; P jj represents FF, i.e., the number of pixels predicted as class j;

[0020] IoU is the ratio of the intersection and union of the set of true pixel category values ​​and the pixel predicted category values. The higher the correlation between the predicted image and the label image, the higher the IoU value. The calculation formula is as follows:

[0021]

[0022] MIoU is the average value of IoU calculated on each class. The calculation expression is as follows:

[0023]

[0024] Step 4 also includes the following sub-steps: Step 4.1: compress the input images of the training set in the sorted photovoltaic data set to 256×256, and add them to the feature extraction network ResNet-101. The photovoltaic panel features are obtained through convolution and attention operations in the downsampling process, and the output of the encoder layer of each stage is processed through channel self-attention, global convolution and fused boundary refinement modules using jump connections to obtain the input feature map of the corresponding decoder layer; the feature map obtained by the encoder layer is upsampled layer by layer to obtain the prediction result map; finally, the prediction result map is channel-fused with the original feature map and placed in the fused boundary refinement module for processing to optimize the segmentation boundary, and then the final prediction result map is obtained through 1×1 convolution;

[0025] Step 4.2: Compare the predicted result image with the label of the photovoltaic panel sample, using the loss function DiceLoss. Assume that X is the real mask image and Y is the predicted mask image. DiceLoss is expressed as follows:

[0026]

[0027] Adjust the network parameters according to the loss function and return to step 4.1 for the next round of training.

[0028] The beneficial effects of the present invention are as follows: the present invention introduces channel self-attention to effectively capture the channel information obtained by downsampling, and improves the global convolution and boundary refinement modules to effectively capture the spatial information of the feature map. Finally, the experimental results on the public photovoltaic dataset show that the present invention performs well in the segmentation of photovoltaic panels in remote sensing images and is applicable to remote sensing images of multiple different spatial resolutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Examples of different types of photovoltaic panels according to an embodiment of the present invention.

[0030] Figure 2 The figure is a schematic diagram of a model structure of an embodiment of the present invention.

[0031] Figure 3 Schematic diagram of a channel self-attention mechanism according to an embodiment of the present invention.

[0032] Figure 4 Schematic diagram of a global convolution module according to an embodiment of the present invention.

[0033] Figure 5 A schematic diagram of a fusion strategy of a BR module according to an embodiment of the present invention.

[0034] Figure 6 A schematic diagram of a GCM and a fused BR module in a parallel structure of BR modules according to an embodiment of the present invention.

[0035] Figure 7 This is a graph of the prediction results of different networks on the PV01 test set.

[0036] Figure 8 This is a graph showing the prediction results of different networks on the PV03 test set.

[0037] Fig. 9 This is a graph of the prediction results of different networks on the PV08 test set. DETAILED DESCRIPTION

[0038] The present invention is further described below in conjunction with the accompanying drawings: As shown in the figure, a photovoltaic panel segmentation method based on deep learning technology includes the following steps:

[0039] Step 1: The regional images taken by drones or satellites include three sub-datasets with different resolutions, which are represented by PV08, PV03 and PV01 respectively;

[0040] The public dataset used by the CPGBNet photovoltaic panel segmentation algorithm (this application) is obtained from Zenodo. All photovoltaic samples in this dataset are collected by the Gaofen-2 and Beijing-2 satellites and drones from Jiangsu Province, China. It includes three sub-datasets with different resolutions, with spatial resolutions of 0.8 meters, 0.3 meters and 0.1 meters. The three datasets are represented by PV08, PV03 and PV01 respectively. The PV08 dataset is classified into rooftop photovoltaics and ground photovoltaics according to the photovoltaic installation area. The PV03 dataset is divided into rooftop photovoltaics, shrub photovoltaics, grassland photovoltaics, farmland photovoltaics, saline-alkali photovoltaics, and water surface photovoltaics. The PV01 dataset is divided into concrete, steel tiles and bricks according to the roof background. The photovoltaic sample size of the PV03 and PV08 datasets is 1024×1024, and the image size of the PV01 dataset is 256×256. All three datasets are converted to RGB images and have corresponding labels. After data enhancement for all images, they are divided into training set, validation set, and test set in a ratio of 7:2:1. The data structure is shown in the following table

[0041]

[0042] Step 2: All three data sets are converted into RGB images with corresponding labels. After data enhancement, all images are divided into training set, validation set, and test set in a ratio of 7:2:1.

[0043] The data set is subjected to data enhancement operations such as rotation, flipping, cropping, and adding noise by 90°, 180°, and 270°. Therefore, the imaging may be bright and dark. In order to reduce the impact of overexposure or underexposure on image quality and improve the detail performance and contrast of the image, the present invention also uses gamma transformation technology to correct the image. After data enhancement, all images are divided into training set, verification set, and test set in a ratio of 7:2:1.

[0044] Step 3: Based on the deep learning network model of the encoder-decoder structure, a symmetrical global convolution module and boundary refinement module are constructed. The upsampling operation of the decoder part is to first perform bilinear interpolation on the input feature map to double the size of the original feature map, and then use 3×3 convolution and ReLU functions to capture local information; the downsampling operation uses maximum pooling to half the original size;

[0045] The present invention is a semantic segmentation network model of an encoder-decoder structure, and the backbone network adopts ResNet-101. In order to allow the model to retain more global information and local information left by the downsampling process, the present invention constructs a global convolution module and a boundary refinement module with a symmetrical structure. Since the number of channels in the downsampling process is increasing and may contain a lot of redundant information, in order to allow the model to filter redundant channel information, the present invention will halve the number of channels of the feature map in the shallow layer of the jump connection. At the same time, in order to ensure that redundant channel information rather than important channel information is filtered, the channel self-attention mechanism will be used first to make the model pay more attention to effective channel information. Considering that the semantic information contained in the semantic information of adjacent encoder output layers is similar, if the features are fused, not only will the model not be interfered by the large gap in semantic information, but the adjacent outputs of each layer of encoders will be fused to retain the semantic information lost by downsampling to a large extent. Therefore, this paper performs feature fusion on the outputs of adjacent layers of the decoder. The model structure of the present invention is as follows Figure 2 As shown in the figure. The blue line part is the upsampling operation, which requires first performing bilinear interpolation on the input feature map to double the size of the original feature map, and then using 3×3 convolution and ReLU functions to capture local information. The red line part is the downsampling operation, which uses maximum pooling to half the original size.

[0046] The traditional global convolution module (GCM) splits the large convolution kernel k×k into k×1+1×k and only uses separable convolution layers to form a symmetrical module to reduce the number of parameters and computational cost. Figure 4As shown, GCM adopts a parallel symmetric method, and different split combinations can supplement some missing global features. Where W represents the width of the input feature map, H represents the height of the input feature map, C represents the number of input channels, and C / 2 represents the number of output channels. Sum represents the addition of matrix elements. The size value K of the global convolution kernel of the present invention is 9. The original GCM greatly reduces the number of input channels at the jump connection to a fixed value. The present invention takes into account that a large reduction in the number of channels will lead to the loss of a large amount of channel information.

[0047] Among them, a) represents the series connection of multiple BR modules, b) represents the fusion of multiple BR modules in the form of dense connections, and c) represents the internal structure of the BR module.

[0048] The traditional Boundary Refinement Module (BRM) can capture fine edge features, but it does not work well in the segmentation of photovoltaic panels with low spatial resolution. Therefore, the present invention designs a variety of BR module fusion strategies, including series connection and dense connection, such as Figure 5 As shown in the figure, the fused BR modules of different strategies are added to the back of each global convolution. The BR module is a residual structure composed of traditional small convolution and ReLU activation function. 3x3xCxC represents the height, width, number of channels and number of convolution kernels.

[0049] The present invention forms a symmetrical structure with GCM and an improved BR module (IBRM), because local information and global information are equally important. If local information is filtered from global information, the global information can be well retained, but some local information may be missed. If local information is retained first and then global information is filtered, although the complexity is increased, the outline of the global features is also more obvious. Therefore, the present invention first uses GCM and IBRM in series, then IBRM and GCM in series, and finally the two are connected in parallel to form a symmetrical structure. Figure 6 shown.

[0050] Step 4: Model training: start model training based on the training set and validation set in the sorted photovoltaic panel data set and the constructed model;

[0051] Step 5: Input the sorted PV01 test data set, PV03 test data set, and PV08 test data set into the trained model network to obtain the predicted segmentation result map.

[0052] The boundary refinement module is a BR module fusion module, which includes multiple BR modules connected in series or densely.

[0053] In the BR module, GCM and IBRM are connected in series, and then IBRM and GCM are connected in series, and finally the two are connected in parallel.

[0054] The global convolution module uses a parallel symmetric method and different split combinations to supplement some missing global features. The size value K of the global convolution kernel is 9.

[0055] The model also includes a channel self-attention mechanism, which measures the importance of these channels by the correlation between each channel. The input feature map is passed through the channel self-attention mechanism to generate a feature map with channel self-attention weights.

[0056] Channel self-attention measures the importance of each channel by the correlation between them. The input feature map is passed through the channel self-attention mechanism to generate a feature map with channel self-attention weights. Figure 3 shown.

[0057] A represents the input feature map. In order to maintain the mapping relationship between different channels, 1×1 convolution is not used to embed features. A is reshaped to obtain B, C, and D, where B corresponds to the query in the self-attention mechanism calculation process, C corresponds to the key in the self-attention mechanism calculation process, and D corresponds to the value in the self-attention mechanism calculation process. B is transformed into Where N represents a total of H×W positions. After matrix multiplication of C and B, the relationship matrix between each two channels is obtained, and then the channel attention matrix is ​​obtained through softmax normalization. The calculation formula is as follows:

[0058]

[0059] where x ji Represents the relationship between the i-th channel and the j-th channel.

[0060] After matrix multiplication with D, the relationship matrix between each channel and all channels is obtained, and added to A to obtain the feature map E with channel self-attention mechanism, which is expressed as follows:

[0061]

[0062] Where E j Represents the feature value with the weight of the self-attention mechanism in the jth channel.

[0063] It also includes a model validation step, as shown in the following formula

[0064]

[0065] The pixel accuracy PA, intersection over union (IoU), and mean intersection over union (Mean IoU) are used as evaluation criteria; assuming there are k+1 categories, i represents the true value, and j represents the predicted value P ii represents TP, i.e., the number of pixels predicted as class i; P ij represents FN, i.e., the number of pixels of class i predicted as class j; P ji represents FP, that is, the number of pixels of class j predicted as class i; P jj represents FF, i.e., the number of pixels predicted as class j;

[0066] IoU is the ratio of the intersection and union of the set of true pixel category values ​​and the pixel predicted category values. The higher the correlation between the predicted image and the label image, the higher the IoU value. The calculation formula is as follows:

[0067]

[0068] MIoU is the average value of IoU calculated on each class. The calculation expression is as follows:

[0069]

[0070] Step 4 also includes the following sub-steps: Step 4.1: compress the input images of the training set in the sorted photovoltaic data set to 256×256, and add them to the feature extraction network ResNet-101. The photovoltaic panel features are obtained through convolution and attention operations in the downsampling process, and the output of the encoder layer of each stage is processed through channel self-attention, global convolution and fused boundary refinement modules using jump connections to obtain the input feature map of the corresponding decoder layer; the feature map obtained by the encoder layer is upsampled layer by layer to obtain the prediction result map; in order to make full use of adjacent semantic information to restore the high-resolution feature map, the inputs of two adjacent decoders of the decoder are fused, and the upsampling operation is continuously performed to obtain the final segmentation prediction map;

[0071] Step 4.2: Compare the predicted result image with the label of the photovoltaic panel sample, using the loss function DiceLoss. Assume that X is the real mask image and Y is the predicted mask image. DiceLoss is expressed as follows:

[0072]

[0073] Adjust the network parameters according to the loss function and return to step 4.1 for the next round of training.

[0074] To verify the effect, the present invention uses U-Net, SegNet, PSPNet, DeepLabv3+, RefineNet and CPGBNet (the algorithm of this application) to conduct experiments on three photovoltaic data sets with different spatial resolutions, while keeping the same training set, test set and the same number of training rounds of 160. The training results are shown in the following table:

[0075]

[0076] Compared with other models, the present invention improves the accuracy of photovoltaic panel segmentation. The present invention introduces channel self-attention to effectively capture the channel information obtained by downsampling, and improves the global convolution and boundary refinement modules to effectively capture the spatial information of the feature map. Finally, the experimental results on the public photovoltaic dataset show that the present invention performs well in the segmentation of photovoltaic panels in remote sensing images and is applicable to multiple remote sensing images with different spatial resolutions.

[0077] Data collection and screening:

[0078] The dataset is subjected to data enhancement operations such as 90°, 180° and 270° rotation, flipping, cropping and adding noise. Therefore, the imaging may be bright and dark. In order to reduce the impact of overexposure or underexposure on image quality and improve the image detail performance and contrast. The present invention also uses gamma transformation technology to correct the image. After data enhancement for all images, they are divided into training set, validation set and test set in a ratio of 7:2:1. Some photovoltaic panel data are shown in Figure 2. Figure 7 shown.

[0079] Model training and validation

[0080] The training and verification of the model are started based on the training set and verification set in the organized photovoltaic panel data set and the constructed CPGBNet model. Subsequently, the photovoltaic panel data set is imported into the model and trained on the data set.

[0081] The training steps are as follows: All input images of the training set in the organized photovoltaic data set are compressed to 256×256 and added to the feature extraction network ResNet-101. The photovoltaic panel features are obtained through convolution and attention operations in the downsampling process, and the output of the encoder layer at each stage is processed through channel self-attention, global convolution and fused boundary refinement modules using jump connections to obtain the input feature map of the corresponding decoder layer. The feature map obtained by the encoder layer is upsampled layer by layer to obtain the prediction result map. In order to make full use of adjacent semantic information to restore the high-resolution feature map, the inputs of the two adjacent decoders of the decoder are fused, and the upsampling operation is continuously performed to obtain the final segmentation prediction map.

[0082] Model Validation

[0083] Input the sorted PV01 test data set, PV03 test data set, and PV08 test data set into the trained model network to obtain the predicted segmentation result map, such as Figure 7 , Figure 8 , Fig. 9 shown.

[0084] The embodiments described with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention. The embodiments should not be regarded as limiting the present invention, but any improvements made based on the spirit of the present invention should be within the scope of protection of the present invention.

Claims

1. A photovoltaic panel segmentation method based on deep learning technology, characterized in that: The following steps are included: Step 1: Regional images taken by drones or satellites, including three sub-datasets with different resolutions, represented by PV08, PV03 and PV01 respectively; Step 2: All three data sets are converted into RGB images with corresponding labels. After data enhancement, all images are divided into training set, validation set, and test set in a ratio of 7:2:

1. Step 3: Based on the deep learning network model of the encoder-decoder structure, a symmetrical global convolution module and boundary refinement module are constructed. The upsampling operation of the decoder part is to first perform bilinear interpolation on the input feature map to double the size of the original feature map, and then use 3×3 convolution and ReLU functions to capture local information; the downsampling operation uses maximum pooling to half the original size; Step 4: Model training: start model training based on the training set and validation set in the sorted photovoltaic panel data set and the constructed model; Step 5: Input the sorted PV01 test data set, PV03 test data set, and PV08 test data set into the trained model network to obtain the predicted segmentation result map; The global convolution module uses a parallel symmetric method, and different split combinations can supplement some missing global features. The size value K of the global convolution kernel is 9; Step 4 also includes the following sub-steps: Step 4.1: Compress all the input images of the training set in the sorted photovoltaic data set into , and added to the feature extraction network ResNet-101, the photovoltaic panel features are obtained through the convolution and attention operations of the downsampling process, and the output of the encoder layer of each stage is processed through the channel self-attention, global convolution and fused boundary refinement module using skip connections to obtain the input feature map of the corresponding decoder layer; in order to make full use of adjacent semantic information to restore high-resolution feature maps, the inputs of two adjacent decoders of the decoder are fused, and upsampling operations are continuously performed to obtain the final segmentation prediction map; Step 4.2: Compare the obtained prediction result image with the label of the photovoltaic panel sample, using the loss function DiceLoss. Assuming X is the real mask image and Y is the predicted mask image, DiceLoss is as follows: DiceLoss=1- Adjust the network parameters according to the loss function and return to step 4.1 for the next round of training; In the BR module, GCM and IBRM are connected in series, and then IBRM and GCM are connected in series, and finally the two are connected in parallel.

2. The photovoltaic panel segmentation method based on deep learning technology according to claim 1, characterized in that: The boundary refinement module is a BR module fusion module, which includes multiple BR modules connected in series or densely.

3. The photovoltaic panel segmentation method based on deep learning technology according to claim 1, characterized in that: The model also includes a channel self-attention mechanism, which measures the importance of these channels by the correlation between each channel. The input feature map is passed through the channel self-attention mechanism to generate a feature map with channel self-attention weights.

4. The photovoltaic panel segmentation method based on deep learning technology according to claim 1, characterized in that: It also includes a model validation step, as shown in the following formula The pixel accuracy PA, intersection over union (IoU), and mean intersection over union (Mean IoU) are used as evaluation criteria; assuming there are k+1 categories, i represents the true value, j represents the predicted value, represents TP, i.e., the number of pixels predicted as class i; represents FN, i.e., the number of pixels of class i predicted as class j; represents FP, that is, the number of pixels of class j predicted as class i; represents FF, i.e., the number of pixels predicted as class j; IoU is the ratio of the intersection and union of the set of true pixel category values ​​and the pixel predicted category values. The higher the correlation between the predicted image and the label image, the higher the IoU value. The calculation formula is as follows: MIoU is the average value of IoU calculated on each class. The calculation expression is as follows: 。

Images