Improved Unet-based Method for Segmenting the Boundary of Photovoltaic Panels in Infrared Images from the Perspective of UAVs

Abstract

The present invention discloses a photovoltaic panel semantic segmentation method based on deep learning and applied to infrared images. A photovoltaic panel dataset under infrared light conditions from the perspective of an unmanned aerial vehicle is established and preprocessed; an improved Unet semantic segmentation deep learning model is constructed; the training set is input into the improved Unet semantic segmentation deep learning model batch by batch for iteration, and the performance of the model obtained by real-time training is tested using the test set; the photovoltaic panel image under the infrared light conditions to be measured is input into the model corresponding to the minimum loss for processing and output to obtain the segmentation result. The present invention applies the method of deep learning to the boundary detection of infrared photovoltaic panels, and improves the Unet network model to propose more significant shallow features to improve the accuracy of photovoltaic panel segmentation.

Description

Technical Field

[0001] The invention relates to an image processing method belonging to the field of image processing for unmanned aerial vehicle applications, and in particular to a photovoltaic panel boundary segmentation method for an infrared image from a perspective of a unmanned aerial vehicle. Background Art

[0002] In recent years, the field of drone vision has been actively promoted and applied because the camera mounted on the drone can obtain a wider viewing angle, and the real-time and flexibility of drone vision are relatively high. At present, there have been breakthroughs in the detection of photovoltaic panels with visible light. For infrared images, due to the continuity of the thermal domain, the edge boundary is not obvious, and the boundary of the photovoltaic panel cannot be well determined. The boundary segmentation of infrared images still relies on manual work. However, there are currently few infrared monitoring personnel, which cannot meet the labeling work of massive infrared images, which will lead to a waste of human resources. In addition, the method based on traditional image processing has poor accuracy and poor robustness, and will be affected by strong light and background. In order to solve the above problems, the present invention proposes a photovoltaic panel boundary detection method in an infrared scene based on deep learning.

[0003] In order to improve the intelligence level of photovoltaic panel segmentation in infrared images from the perspective of drones, some intelligent image processing methods can be used to segment infrared images and extract photovoltaic panel areas to facilitate the positioning of the photovoltaic range. In the field of computer vision, there are endless methods for image segmentation. At present, traditional image segmentation methods are mainly suitable for situations where image features are obvious and the density is low. Moreover, with the changes in the background, the incidence of misjudgment is large and the accuracy of the segmentation results is not high. In addition, due to the continuous thermal domain characteristics of infrared images, the boundaries of infrared images are difficult to define, so there is a large uncertainty. Therefore, image segmentation based on traditional methods is not conducive to photovoltaic segmentation in infrared scenes. Summary of the invention

[0004] The present invention provides a photovoltaic panel infrared image boundary segmentation method from the perspective of a drone. The method is improved based on the deep learning model Unet, which overcomes the shortcomings of the traditional method, improves the anti-interference ability of the model and improves the segmentation accuracy.

[0005] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0006] Step S1: Collect infrared photovoltaic panel images under infrared light conditions through the perspective of the drone, establish a photovoltaic panel dataset under infrared light conditions from the perspective of the drone, and make annotations, such as Figure 3 As shown;

[0007] Step S2: Propagate the photovoltaic panel dataset forward through the super-resolution model to obtain an infrared dataset image with 2x resolution, and then perform image augmentation and divide it into a training set and a test set;

[0008] Step S3: Construct an improved Unet semantic segmentation deep learning model, and the model structure is as Figure 2 shown;

[0009] Step S4: Set the training method of the improved Unet semantic segmentation deep learning model, and specifically set hyperparameters such as the number of iterations and the learning rate;

[0010] Step S5: Use the training set as the input of the model and input it into the improved Unet semantic segmentation deep learning model batch by batch for iteration. Every 3000 iterations, use the test set to test the performance of the model obtained by real-time training; and when the number of iterations reaches the pre-set iteration threshold, stop training and take out the model corresponding to the minimum loss in the test set;

[0011] Step S6: Input the photovoltaic panel image under the infrared light condition to be measured into the model corresponding to the minimum loss for processing and output the segmentation result.

[0012] In the said Step S1: Preprocess the infrared photovoltaic panel image, and the preprocessing includes unifying the image size, marking the pixels and position information occupied by the photovoltaic panel in the image, and dividing it into a training set and a test set according to the ratio of 8:2.

[0013] In the said Step S2, use a super-resolution network to double the width and height of the infrared image to the original infrared image to obtain more abundant information, retain and generate detailed information.

[0014] In the said Step S3, use an improved Unet semantic segmentation deep learning model, as Figure 3 shown,

[0015] The improved Unet semantic segmentation deep learning model is improved on the basis of the original Unet semantic segmentation deep learning model in the following way: the output of the first convolution module in the feature extraction part is added to the output of the second convolution module in the feature extraction part after dilated convolution as the first enhanced feature map set, then the first enhanced feature map set is added to the output of the third convolution module in the feature extraction part after dilated convolution as the second enhanced feature map set, then the second enhanced feature map set is added to the output of the fourth convolution module in the feature extraction part after dilated convolution as the third enhanced feature map set, then the third enhanced feature map set is sequentially subjected to feature compression and size expansion to obtain the fourth enhanced feature map set, and finally the fourth enhanced feature map set, the output of the first convolution module in the feature extraction part, and the output after upsampling of the second-to-last convolution module in the scale reduction and feature fusion part are added together and input into the last convolution module in the scale reduction and feature fusion part.

[0016] The feature map set is composed of many superimposed feature maps.

[0017] In specific implementation, the improved Unet semantic segmentation deep learning model is carried out in the following way:

[0018] It includes a feature extraction part and a scale reduction and feature fusion part. The feature extraction part includes four consecutive convolution modules, and the scale reduction and feature fusion part also includes four consecutive convolution modules. Each convolution module is composed of two convolution layers connected in sequence, and an activation function is set after each convolution layer; a max pooling layer is connected immediately after each convolution module in the feature extraction part, and an upsampling layer is connected immediately before each convolution module in the scale reduction and feature fusion part;

[0019] The output of the fourth convolution module in the feature extraction part after passing through the max pooling layer is further processed by an additional convolution module to obtain an intermediate feature map set, and then the output of the intermediate feature map set after passing through the upsampling layer is added to the output of the fourth convolution module in the feature extraction part and input into the first convolution module in the scale reduction and feature fusion part; then the output of the first convolution module in the scale reduction and feature fusion part after passing through the upsampling layer is added to the output of the third convolution module in the feature extraction part and input into the second convolution module in the scale reduction and feature fusion part, and the output of the second convolution module in the scale reduction and feature fusion part after passing through the upsampling layer is added to the output of the second convolution module in the feature extraction part and input into the third convolution module in the scale reduction and feature fusion part;

[0020] The output of the first convolutional module in the feature extraction part is subjected to dilated convolution and then added to the output of the second convolutional module in the feature extraction part to obtain the first enhanced feature map set. Then, the first enhanced feature map set is subjected to dilated convolution and added to the output of the third convolutional module in the feature extraction part to obtain the second enhanced feature map set. Then, the second enhanced feature map set is subjected to dilated convolution and added to the output of the fourth convolutional module in the feature extraction part to obtain the third enhanced feature map set. Then, the third enhanced feature map set is sequentially subjected to feature compression and size expansion to obtain the fourth enhanced feature map set; the output of the third convolutional module in the scale reduction and feature fusion part after passing through the upsampling layer is added to the output of the first convolutional module in the feature extraction part and the fourth enhanced feature map set, and then input into the fourth convolutional module in the scale reduction and feature fusion part.

[0021] The present invention strengthens the utilization of shallow features. By adding a shallow branch and using the ability of dilated convolution to capture spatial information, the model pays more attention to information such as shallow color contours. In the loss function part, the balanced cross-entropy loss function is adopted.

[0022] The present invention improves the Unet network model and builds a deep learning network. In order to achieve the effect of real-time segmentation, a lightweight architecture is adopted.

[0023] The present invention applies the deep learning model to the problem of infrared photovoltaic panel segmentation, and makes improvements to the Unet network according to the characteristics of infrared photovoltaic panels, enabling the model to pay more attention to information such as shallow contours and colors. The applicant has found through research that deep learning can more effectively extract the thermal distribution of infrared images and greatly improve the accuracy of photovoltaic panel segmentation.

[0024] The method for segmenting the boundary of the photovoltaic panel in the infrared image from the perspective of the drone based on the improved Unet of the present application adopts a learning-based image processing method to implement the segmentation of the infrared image, improves the segmentation accuracy, and uses super-resolution to amplify the thermal distribution information of the infrared image, which helps to obtain the edge information of the photovoltaic panel under infrared conditions.

[0025] In the step S5, the iteration number threshold is set between 8000 and 10000 times.

[0026] In the step S5, Adam is used as the optimizer of the model to iteratively update the weights. The batch size is set to 8, the total number of training epochs is set to 12, the initial learning rate is 0.01, and it drops to 1 / 10 of the original at the 7th and 11th epochs. The momentum is set to 0.9, and the first 500 iterations are set as the warm-up stage, and its learning rate is 1 / 100 times of the initial learning rate. It is set to find the global minimum of the objective function according to the gradient during the training process. After each batch iteration, the weight value of the model is updated once.

[0027] In step S5, if the number of infrared image data of the drone perspective prepared is less than 500 due to the limitation of the number of the data set, the transfer learning method is adopted. First, the weights obtained from the pre-trained model on the ImageNet data set are used, and then the training set is input into the model for formal training. Transferring the weights of the model pre-trained on the ImageNet data set in this way helps the model better recognize objects. And the number of training iterations is set to about 30 to 40 times, and the Adam optimization method is used for parameter update, which can prevent overfitting caused by the too small data set.

[0028] In step S5, then test the performance of the model:

[0029] If the accuracy rate reaches 95%, it indicates that the model already has a good ability to segment the infrared image of the photovoltaic panel. Save the hyperparameter file of this training, and continue to iterate and judge whether a higher accuracy can be achieved until the accuracy rate of the model on the test set does not increase for six consecutive iteration cycles.

[0030] If the accuracy rate is lower than 95%, continue to perform iterative optimization.

[0031] Among them, the accuracy rate refers to the MPA (mean pixel accuracy of classes), and the calculation formula is as follows:

[0032]

[0033] Among them, k represents the number of classes, and p ij is the number of pixels that originally belong to the i-th class but are assigned to the j-th class. That is to say, pii represents the number of correctly classified positive examples.

[0034] For the technologies not mentioned in the present invention, refer to the prior art.

[0035] The method for segmenting the boundary of the photovoltaic panel in the infrared image from the drone perspective based on the improvement of Unet of the present invention has completed the segmentation of the photovoltaic panel in the infrared image from the drone perspective. Compared with the traditional infrared image segmentation method, this method has a great improvement in accuracy and pays more attention to the edge contour information of the infrared image. The greatest advantage of the present invention is that it overcomes the problem of difficult boundary distinction caused by the continuous thermal domain of the infrared image, which can help the positioning and anomaly detection of the photovoltaic panel.

[0036] The present invention applies the method of deep learning to the infrared photovoltaic panel boundary detection, and improves the Unet network model to propose more significant shallow features to improve the accuracy of photovoltaic panel segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1It is a schematic flowchart of the photovoltaic panel boundary segmentation method for infrared images from the perspective of an unmanned aerial vehicle (UAV) based on the improved Unet in the embodiments of the present invention;

[0038] Figure 2 It is a schematic diagram of the improved Unet network structure in the embodiments of the present invention (the blue dashed line is the improved content);

[0039] Figure 3 It is an infrared image to be measured of a photovoltaic panel from the perspective of a UAV in the embodiments of the present invention;

[0040] Figure 4 is Figure 3 the segmentation result diagram. Specific implementation manners

[0041] To better understand the present invention, the content of the present invention will be further clarified below in conjunction with embodiments, but the content of the present invention is not limited to the following embodiments.

[0042] As Figure 1 shown, the overall process of the method in this embodiment includes the following steps;

[0043] Step S1; Establish an infrared image dataset of a photovoltaic panel from the perspective of a UAV. The image samples are from the inspection UAVs at the power plant site. Infrared images with variable scenes, different illuminations, and brightness under infrared conditions are respectively selected as the training dataset. The image selection examples are as Figure 3 shown; Preprocess the selected infrared images. First, use the image annotation tool labelme to annotate the original images, and then parse the generated json annotation files to obtain the segmented images of the targets as the segmentation targets. Divide the preprocessed image dataset into a training set and a test set according to an 8:2 ratio; Finally, unify the images with uneven sizes to a size of 572×572 pixels and perform a normalization operation.

[0044] Step S2: Through the forward propagation of the super-resolution model, obtain infrared dataset images with 2x high resolution; The essence of the super-resolution network is to generate images with a larger resolution through the AI method. The same purpose can also be achieved by methods such as bilinear interpolation. The AI-based method can obtain richer inferences, which is helpful for the judgment of the boundary.

[0045] Step S3: Construct a Unet deep learning model. The Unet network is widely used in the medical field. Since this network is good at capturing minute features, it is very beneficial for obtaining some edge detail information. In order to improve the extraction of the shallow contour information of the Unet network, the present invention improves the Unet network to enhance its ability to extract shallow contour features.

[0046] The size of the input image is 572x572. First, it undergoes 2 convolutional operations with a kernel size of 3x3, and then a shallow feature map set f1 with a size of 568x568 is obtained. The number of feature maps contained in the feature map set f1 is 64, that is, 64 dimensions. Figure 2 The numbers such as 64 on each feature map set represent the number of feature maps contained in the feature map set.

[0047] After that, it is divided into three paths. Two of them are the same as the original Unet. After one downsampling and several 3x3 convolutional operations, a feature map set f2 with a scale of 280x280 is obtained. Continuing with downsampling and convolutional operations, a feature map set f3 with a scale of 136x136 is obtained, and then a feature map set f4 with a scale of 64x64 is obtained.

[0048] Until the scale is compressed to a feature map set with a scale of 32x32 After that, upsampling is performed to enlarge the scale, and a feature map set with a scale of 56x56 is obtained respectively Among them, the feature map set Fuses the features of the feature map set f4, and its expression is:

[0049]

[0050] Where Refers to two convolutional blocks (a 3x3 convolution and a Relu activation function) and an upsampling operation, Refers to the cropping operation that crops f4 from 64x64 to a scale of 56x56.

[0051] According to the feature map set In the same way, a feature map set with a scale of 104x104 is generated A feature map set with a scale of 200x200 And a feature map set with a scale of 388x388 Its expression is as follows:

[0052]

[0053]

[0054]

[0055] Where All represent the cropping operation, which respectively crop the feature maps in the feature map set from 136x136 to 104x104, from 280x280 to 200x200, and from 568x568 to 388x388.

[0056] Another path is the improved network structure of the present invention.

[0057] Perform 3x3 dilated convolution on the feature map set of 568x568 to increase the receptive field. After the first 3x3 dilated convolution and fusing the deep feature map set f2, a feature map set F1 of 280x280 is obtained. The expression of the feature map set F1 is as follows.

[0058] F1 = D(f1) + f2

[0059] Where D represents a 3x3 dilated convolution with padding.

[0060] Perform another dilated convolution on the feature map set F1 and fuse the feature layer f3 to obtain a feature map set F2 with a scale of 136x136.

[0061] F2 = D(f2) + f3

[0062] Perform another dilated convolution on the feature map set F2 and fuse the feature map set f4 to obtain a feature map set F3 with a scale of 64x64.

[0063] F3 = D(f3) + f4

[0064] After obtaining the feature map set F3 that contains rich deep semantic and shallow contour color and other information, perform feature compression on it to obtain an intermediate feature vector V. The formula is as follows.

[0065]

[0066] Where H and W represent the height and width of F3 (both are 64).

[0067] After obtaining the feature vector, expand its size to 392x392 and denote it as F4. Here, the original value expansion method is used to enlarge the size. Substantially, it is to add this feature vector as a global feature quantity to the final result.

[0068] After obtaining F4, add it to the original T4 to obtain a fused feature layer R of 392x392. Perform two U operations (a 3x3 convolution and a Relu activation function) on R, and then pass through a 1x1 convolution to finally obtain a feature map set H with a scale of 388x388 and 2 layers.

[0069] R = F4 + T4

[0070] H = G(U(R))

[0071] Where G(*) represents performing a 1x1 operation on *.

[0072] Step S4: Set the initial hyperparameters and the number of iterations of the model. Set the number of iterations to 8000 - 10000 times. The parameter update uses the Adam optimizer, and the batch size is 8. After each batch training is completed, calculate the loss value through the balanced cross-entropy loss function, and perform backpropagation to update the gradient information.

[0073] The formula for the balanced cross-entropy loss function is as follows:

[0074]

[0075]

[0076] where β is the weighting factor, which weights the positive and negative samples. represents the predicted probability value. p indicates whether the current category is the same as the true category. If it is the same, it is 1; if it is not, it is 0. cls i represents the category of the i-th pixel in the final feature map obtained by prediction, c represents the true category of this pixel, represents the balanced cross-entropy loss of a single pixel.

[0077] Step S5: Input the labeled infrared image training set in Step S1 into the constructed deep model for training;

[0078] Step S6: When the number of iterations reaches 3000 times, 6000 times, and 9000 times respectively, use the labeled test set in Step S1 to evaluate the performance of the model currently obtained in Step S5, and measure it with the MPA index;

[0079] If the accuracy rate in the test set reaches more than 95%, it means that the current network already has a good cognitive ability, and the model has the ability to segment infrared photovoltaic panel images well. Save the hyperparameter file of this training;

[0080] Step S7: When the number of iterations reaches 10000 times, stop training. Compare the model performances obtained when the number of iterations is 3000 times, 6000 times, and 9000 times respectively, select the training result with the best performance, use the weight file obtained from this training, import the weights into the model, and obtain the optimal photovoltaic panel infrared image segmentation model.

[0081] Step S8: Select a photovoltaic panel infrared image that meets the test conditions as the image to be tested (as Figure 3 shown), input it into the optimal photovoltaic panel infrared image model obtained in Step S7 to complete forward propagation, and obtain the segmentation result as Figure 4 shown.

[0082] In steps S1 and S4, if the number of the obtained infrared image datasets of the photovoltaic panels is small due to inconvenient acquisition or cumbersome annotation work, etc., the idea of transfer learning is adopted, and the weights of the trained image classification in the Imagenet dataset are imported into the model; the Imagenet dataset is the largest global image classification dataset, which contains 14 million images of 1000 categories. Through the pre-trained weights, the model can have certain cognitive ability, and then the infrared image dataset of the photovoltaic panels is used for formal training in the model. This can not only solve the problem of slow weight convergence speed caused by too small a dataset, but also save the utilization of training resources; correspondingly, to prevent overfitting caused by too small a dataset, regularization and dropout are adopted to improve the anti-overfitting ability of the network, and at the same time reduce the number of iterations. The number of training iterations is set at about 30 to 40 times, and the Adam optimization method is used as the parameter update strategy. The regularization method adopted is batch regularization.

[0083] This example is an infrared image photovoltaic panel boundary segmentation method based on the improvement of Unet, which completes the segmentation of the photovoltaic panels in the infrared images from the perspective of the drone. Compared with the Unet image segmentation method, this paper mainly increases the utilization of shallow features in the model structure, enabling the model to better focus on the edge information of the photovoltaic panels. In order to allow the shallow features to have a better receptive field, dilated convolution is adopted in the improved branch. This method has a great improvement in accuracy (the MPA of Unet reaches 89.4% in the test set of the present invention, and the MPA of the method proposed in this paper reaches 95.7%, with an increase of 6.3%). The edge segmentation is significantly better than that of the Unet image segmentation method. The greatest advantage of the present invention is that it overcomes the problem of difficult boundary distinction caused by the continuous thermal domain of the infrared image, and can be applied to fields such as photovoltaic panel positioning and defect detection in the infrared scene from the perspective of the drone.

Claims

1. An infrared image photovoltaic panel boundary segmentation method based on the improvement of Unet from the perspective of an unmanned aerial vehicle, characterized in that: The method includes the following steps: Step S1: Collect infrared photovoltaic panel images under infrared light conditions from the perspective of a drone, establish a photovoltaic panel dataset under infrared light conditions from the perspective of the drone, and make annotations; Step S2: Propagate the photovoltaic panel dataset forward through a super-resolution model to obtain an infrared dataset image with 2-fold resolution. After image augmentation, it is divided into a training set and a test set; Step S3: Construct an improved Unet semantic segmentation deep learning model; Step S4: Set the training method of the improved Unet semantic segmentation deep learning model; Step S5: Use the training set as the input of the model and input it into the improved Unet semantic segmentation deep learning model batch by batch for iteration. Every 3000 iterations, use the test set to test the performance of the model obtained by real-time training; and when the number of iterations reaches the pre-set iteration number threshold, stop training and take out the model corresponding to the minimum loss in the test set; Step S6: Input the photovoltaic panel image under the infrared light conditions to be measured into the model corresponding to the minimum loss for processing and output the segmented result; In step S3, an improved Unet semantic segmentation deep learning model is adopted. The improved Unet semantic segmentation deep learning model is improved in the following way on the basis of the original Unet semantic segmentation deep learning model: The output of the first convolutional module in the feature extraction part is added to the output of the second convolutional module in the feature extraction part after dilated convolution as the first enhanced feature map set. Then, the first enhanced feature map set is added to the output of the third convolutional module in the feature extraction part after dilated convolution as the second enhanced feature map set. Then, the second enhanced feature map set is added to the output of the fourth convolutional module in the feature extraction part after dilated convolution as the third enhanced feature map set. Then, the third enhanced feature map set is sequentially subjected to feature compression and size expansion to obtain a fourth enhanced feature map set. Finally, the fourth enhanced feature map set, the output of the first convolutional module in the feature extraction part, and the output after upsampling of the second-to-last convolutional module in the scale reduction and feature fusion part are added together and then input into the last convolutional module in the feature fusion part.

2. The method for segmenting the boundary of a photovoltaic panel in an infrared image from the perspective of a drone based on the improved Unet as claimed in claim 1, wherein: In step S1: Preprocess the infrared photovoltaic panel image. The preprocessing includes unifying the image size, marking the pixels and position information occupied by the photovoltaic panel in the image, and dividing it into a training set and a test set according to a ratio of 8:

2.

3. The method for segmenting the boundary of a photovoltaic panel in an infrared image from the perspective of a drone based on the improved Unet as claimed in claim 1, wherein: In step S2, a super-resolution network is used to double the width and height of the infrared image to twice that of the original infrared image.

4. The method for segmenting the boundary of a photovoltaic panel in an infrared image from the perspective of a drone based on the improved Unet according to claim 1, characterized in that: In step S5, the iteration number threshold is set between 8000 and 10000 times.

5. The method for segmenting the boundary of a photovoltaic panel in an infrared image from the perspective of a drone based on the improved Unet according to claim 1, wherein: In step S5, Adam is used as the optimizer of the model to iteratively update the weights. The batch size is set to 8, the total number of training epochs is set to 12, the initial learning rate is 0.01, and it drops to 1 / 10 of the original value at the 7th and 11th epochs. The momentum is set to 0.9, and the first 500 iterations are set as the warm-up stage, with its learning rate being 1 / 100 times the initial learning rate. It is set to find the global minimum according to the gradient during the training process. After each batch iteration, the weight value of the model is updated.

6. The method for segmenting the boundary of a photovoltaic panel in an infrared image from the perspective of a drone based on the improved Unet according to claim 1, characterized in that: In step S5, the method of transfer learning is adopted. First, the weights are obtained by pre-training the model on the ImageNet dataset, and then the training set is input into the model for formal training. The number of training iterations is set to about 30 - 40 times. Using the Adam optimization method for parameter update can prevent overfitting caused by too small a dataset.

7. The method for segmenting the boundary of a photovoltaic panel in an infrared image from the perspective of an unmanned aerial vehicle based on the improved Unet as claimed in claim 1, characterized in that: In step S5, test the performance of the model and then: If the accuracy rate reaches 95%, it indicates that the model already has a good ability to segment the infrared images of photovoltaic panels. Save the hyperparameter file of the training, and continue to iterate and judge whether a higher accuracy can be achieved until the accuracy rate of the model on the test set no longer rises for 6 consecutive iteration cycles. If the accuracy rate is lower than 95%, continue to perform iterative optimization.

Images