Photovoltaic panel target segmentation method and system under complex scene image

By collecting photovoltaic panel images by drones and automatically segmenting them using deep learning models, the problem of time-consuming, labor-intensive and difficult detection of photovoltaic panel inspection is solved, and efficient and all-weather photovoltaic panel monitoring and management is achieved.

CN119048934APending Publication Date: 2024-11-29XIAN INNO AVIATION TECH CO LTD

Patent Information

Application Number
CN202410954378.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing photovoltaic panel inspection methods rely on manual inspection, which is time-consuming and labor-intensive, making it difficult to achieve all-weather monitoring, and it is difficult to refinely detect small or hidden defects, affecting the efficient operation of photovoltaic power stations.

Method used

A drone is used to collect photovoltaic panel images, label the outlines using labelme tool, and deep learning model training is carried out through the yolov7-tiny architecture to realize automatic segmentation and numbering of photovoltaic panels, combining training sets and verification sets to ensure model accuracy and robustness sex.

Benefits of technology

It improves the efficiency of photovoltaic panel inspection, reduces the cost of manual inspection, realizes all-weather and full-day monitoring and refined defect detection of photovoltaic power stations, and ensures the safe operation of the power station.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048934A_ABST
    Figure CN119048934A_ABST
Patent Text Reader

Abstract

The invention discloses a photovoltaic panel target segmentation method under a complex scene image, and the method comprises the steps: shooting a photovoltaic panel image through an unmanned plane, marking the contour of a photovoltaic panel through a label tool, and storing the contour information of the photovoltaic panel in a JSON file; carrying out training on the marked photovoltaic panel data set through a yolov7-tiny framework, and achieving the segmentation of the photovoltaic panel; the puzzle of the photovoltaic panels needing to be numbered is cut into a plurality of images with the size of 640 * 640; the segmented image is input into a trained yolov7-tiny segmentation model; processing a segmentation result to obtain coordinates of four corner points of the photovoltaic panel; the image coordinates are converted into latitude and longitude coordinates through the known battery size and image coordinates. According to the scheme provided by the invention, the photovoltaic panel inspection efficiency is greatly improved, the manual inspection cost is reduced, and meanwhile, the safe operation of a photovoltaic power station is also ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle applications, and in particular to a photovoltaic panel target segmentation method and system under complex scene images. Background Art

[0002] With the rapid development of solar photovoltaic power generation technology, the scale of photovoltaic power stations continues to expand. However, due to environmental factors and long-term operation, the surface of photovoltaic panels may be dirty, damaged, aged, etc. These problems will affect the power generation efficiency and life of photovoltaic panels. Therefore, timely detection and repair of photovoltaic panel defects is crucial to ensure the efficient operation of photovoltaic power stations.

[0003] At present, the traditional inspection method of photovoltaic panels usually relies on manual inspection, and the staff needs to check the status of each photovoltaic panel one by one. This method has the following problems: First, it is time-consuming and labor-intensive. Especially for large-scale photovoltaic power stations, manual inspection requires a lot of manpower and time, and the cost is high; second, it is easily affected by factors such as weather and environment, and it is impossible to achieve all-weather and all-day monitoring; third, it is difficult to achieve refined defect detection, and the human eye sometimes has difficulty identifying tiny defects or hidden faults. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a method and system for photovoltaic panel target segmentation in complex scene images.

[0005] In a first aspect, an embodiment of the present application provides a photovoltaic panel target segmentation method in a complex scene image, the method comprising:

[0006] S1: Obtain the photovoltaic panel dataset; use a drone to take images of photovoltaic panels, use the labelme tool to mark the outlines of the photovoltaic panels, and save the outline information of the photovoltaic panels in a JSON file;

[0007] S2: Training the photovoltaic panel segmentation model; training the labeled photovoltaic panel dataset through the yolov7-tiny architecture to achieve photovoltaic panel segmentation;

[0008] S3: Cut the jigsaw puzzle of the photovoltaic panels to be numbered into multiple images of 640x640 size;

[0009] S4: Input the cut image into the trained yolov7-tiny segmentation model to obtain the segmentation result of each photovoltaic panel;

[0010] S5: Process the segmentation results to obtain the coordinates of the four corner points of the photovoltaic panel; by knowing the battery size and image coordinates, convert the image coordinates into longitude and latitude coordinates.

[0011] In one embodiment, in step S1, obtaining a photovoltaic panel data set includes:

[0012] S11: Turn on the camera on the drone;

[0013] S12: Recording a video of the photovoltaic area by flying a drone, and cutting the obtained video every 0.5 seconds to obtain an image;

[0014] S13: label the obtained image using labelme to obtain the outline of each photovoltaic panel;

[0015] S14: Save all contour coordinates in a JSON file to form a dataset.

[0016] In one embodiment, in step S2, the training using the yolov7-tiny architecture includes:

[0017] Divide the data set in the JSON file into a training set and a validation set in a ratio of 0.9:0.1;

[0018] The images and labels in the training set are fed into the yolov7-tiny network for training, and the network autonomously learns the segmentation of the photovoltaic panel area;

[0019] After the training is completed, the images in the validation set are input into the yolov7-tiny network to judge the performance of the yolov7-tiny network in photovoltaic panel area segmentation;

[0020] When the segmentation performance reaches the threshold, step S3 is performed; when the segmentation performance does not reach the threshold, the process returns to step S1 and increases the number of samples in the training set.

[0021] In one embodiment, in step S3, the jigsaw cutting includes:

[0022] Use drones to fly and photograph photovoltaic stations and perform real-time mosaics to obtain aerial photos of the entire photovoltaic station area;

[0023] The captured puzzle is cut into multiple images of 640x640 size, and the intersecting pixel area of ​​adjacent images after cutting is 32 pixels.

[0024] In one embodiment, in step S4, obtaining the segmentation result of each photovoltaic panel includes:

[0025] Input each cut image into the trained yolov7-tiny network to obtain the segmentation result predicted by the model;

[0026] The contour results obtained by segmentation are converted into a binary image, and the white area in the binary image is the photovoltaic panel area predicted by the model.

[0027] In a second aspect, an embodiment of the present application provides a photovoltaic panel target segmentation system in a complex scene image, the system comprising:

[0028] The acquisition module is used to obtain the photovoltaic panel dataset; the photovoltaic panel image is taken by a drone, and the outline of the photovoltaic panel is marked using the labelme tool, and the outline information of the photovoltaic panel is saved in a JSON file;

[0029] The segmentation module is used to train the photovoltaic panel segmentation model. The labeled photovoltaic panel dataset is trained through the yolov7-tiny architecture to achieve the segmentation of photovoltaic panels.

[0030] The cutting module is used to cut the puzzle of photovoltaic panels that need to be numbered into multiple 640x640 images:

[0031] The input module is used to input the cut image into the trained yolov7-tiny segmentation model to obtain the segmentation results of each photovoltaic panel;

[0032] The processing module is used to process the segmentation results to obtain the coordinates of the four corner points of the photovoltaic panel; by knowing the battery size and image coordinates, the image coordinates are converted into longitude and latitude coordinates.

[0033] In one embodiment, obtaining a photovoltaic panel dataset includes:

[0034] S11: Turn on the camera on the drone;

[0035] S12: Recording a video of the photovoltaic area by flying a drone, and cutting the obtained video every 0.5 seconds to obtain an image;

[0036] S13: label the obtained image using labelme to obtain the outline of each photovoltaic panel;

[0037] S14: Save all contour coordinates in a JSON file to form a dataset.

[0038] In one embodiment, the training by the yolov7-tiny architecture includes:

[0039] Divide the data set in the JSON file into a training set and a validation set in a ratio of 0.9:0.1;

[0040] The images and labels in the training set are fed into the yolov7-tiny network for training, and the network autonomously learns the segmentation of the photovoltaic panel area;

[0041] After the training is completed, the images in the validation set are input into the yolov7-tiny network to judge the performance of the yolov7-tiny network in photovoltaic panel area segmentation;

[0042] When the segmentation performance reaches the threshold, step S3 is performed; when the segmentation performance does not reach the threshold, the process returns to step S1 and increases the number of samples in the training set.

[0043] The beneficial effects of this application include:

[0044] The present application provides a method for target segmentation of photovoltaic panels in complex scene images, which obtains high-resolution photovoltaic panel images by using drones for aerial photography. Then, with the help of the labelme annotation tool, the outline of the photovoltaic panel is accurately annotated, and the outline information is saved in a JSON file. Subsequently, the yolov7-tiny architecture is used to train the annotated photovoltaic panel data set, and the automatic segmentation of photovoltaic panels is realized through deep learning technology. At the same time, the accuracy and robustness of the model are guaranteed by combining the training set and the validation set.

[0045] In actual inspections, by performing jigsaw cutting on the images obtained by aerial photography, multiple images of 640x640 size are obtained. This cutting method not only maintains the clarity of the image, but also makes the segmented images easy to process. Subsequently, these cut images are input into the trained yolov7-tiny segmentation model to obtain the segmentation results of each photovoltaic panel. In order to ensure the accuracy of the segmentation results, this application refines the segmentation results and obtains the coordinate information of the four corner points of the photovoltaic panel. At this point, the conversion of image coordinates to longitude and latitude coordinates is successfully achieved by knowing the size and image coordinates of the battery. Finally, the precise numbering of each photovoltaic panel can be obtained, thereby providing reliable data support for subsequent maintenance and management.

[0046] The method provided in the present application not only greatly improves the efficiency of photovoltaic panel inspection and reduces the cost of manual inspection, but also ensures the safe operation of photovoltaic power stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0048] Figure 1 A schematic diagram of the process of a photovoltaic panel target segmentation method under a complex scene image provided by an embodiment of the present application is shown;

[0049] Figure 2An exemplary structural block diagram of a photovoltaic panel target segmentation system 200 under a complex scene image according to an embodiment of the present application is shown;

[0050] Figure 3 A schematic diagram of the structure of the yolov7-tiny target segmentation model provided in an embodiment of the present application is shown;

[0051] Figure 4 A schematic diagram of an input image provided by an embodiment of the present application is shown;

[0052] Figure 5 A schematic diagram of a binary image of a yolov7-tiny direct segmentation result provided in an embodiment of the present application is shown;

[0053] Figure 6 A schematic diagram of the fitting quadrilateral result of the yolov7-tiny direct segmentation result provided in an embodiment of the present application is shown;

[0054] Figure 7 A schematic diagram of a binary image after the post-processing E2 stage is completed provided in an embodiment of the present application is shown;

[0055] Figure 8 A schematic diagram of a binary image obtained through post-processing provided in an embodiment of the present application is shown;

[0056] Fig. 9 A schematic diagram of a fitting quadrilateral result obtained after post-processing provided in an embodiment of the present application is shown;

[0057] Fig.10 A schematic diagram is shown of numbering photovoltaic panel cells using the segmentation results provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant application, rather than to limit the application. It is also necessary to explain that, for ease of description, only the parts related to the application are shown in the accompanying drawings.

[0059] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] Please refer to Figure 1 Combined with Figure 3-Figure 10 As shown, Figure 1 A photovoltaic panel target segmentation method under a complex scene image provided by an embodiment of the present application is shown, and the method includes:

[0061] Step 110: Obtain a photovoltaic panel dataset; use a drone to take a photo of a photovoltaic panel, use the labelme tool to mark the outline of the photovoltaic panel, and save the outline information of the photovoltaic panel into a JSON file;

[0062] Step 120: training a photovoltaic panel segmentation model; training the labeled photovoltaic panel dataset through the yolov7-tiny architecture to achieve photovoltaic panel segmentation;

[0063] Step 130: Cut the jigsaw puzzle of the photovoltaic panels to be numbered into multiple images of 640x640 size;

[0064] Step 140: Input the cut image into the trained yolov7-tiny segmentation model to obtain the segmentation result of each photovoltaic panel;

[0065] Step 150: Process the segmentation results to obtain the coordinates of the four corner points of the photovoltaic panel; by knowing the battery size and image coordinates, convert the image coordinates into longitude and latitude coordinates.

[0066] The present application provides a method for target segmentation of photovoltaic panels in complex scene images, which obtains high-resolution photovoltaic panel images by using drones for aerial photography. Then, with the help of the labelme annotation tool, the outline of the photovoltaic panel is accurately annotated, and the outline information is saved in a JSON file. Subsequently, the yolov7-tiny architecture is used to train the annotated photovoltaic panel data set, and the automatic segmentation of photovoltaic panels is realized through deep learning technology. At the same time, the accuracy and robustness of the model are guaranteed by combining the training set and the validation set.

[0067] Specifically, refer to Figure 3-Figure 10 , the method comprises the following steps:

[0068] A: Get the photovoltaic panel segmentation dataset. First, it involves installing a camera on a drone. Then, the drone flies over the photovoltaic station and records a video of the photovoltaic panel area. The video captured by the camera captures an image every three seconds at the set frame rate, and about 5,000 images are captured from the entire video. Next, use the labelme tool in Python to annotate the photovoltaic panel area in each image, mark the precise location of the photovoltaic panel, and the result of the annotation is the contour points of the photovoltaic panel area. These contour point information is recorded and saved in a JSON file, providing detailed data for the location of each photovoltaic panel. With these original images and the corresponding label files (JSON data), you can proceed to the next step of segmentation model training. The implementation of this series of steps has laid a solid foundation for the subsequent training of the photovoltaic panel segmentation model.

[0069] B: Training the photovoltaic panel segmentation model. Using the yolov7-tiny architecture to train the photovoltaic panel segmentation model brings significant advantages. Compared with the traditional semantic segmentation model, yolov7-tiny has the characteristics of fast reasoning and small computational complexity. In the subsequent post-processing stage, compared with the traditional semantic segmentation model, yolov7-tiny performs better when the puzzle effect is damaged. The traditional semantic segmentation model is easily affected by problems such as incomplete splicing of the edges of the photovoltaic panels in the puzzle or distortion of the shape of the photovoltaic panels, which may cause holes in the binary image of the photovoltaic panel area generated by the model, thus affecting the subsequent processing flow. The use of yolov7-tiny for segmentation can effectively solve these problems. The process of training yolov7-tiny is as follows: First, convert the JSON label data in step A into txt format. Then, divide the dataset into a training set and a validation set, where 90% of the images are used for training and 10% of the images are used for validation. Next, create a data.yaml file containing information such as the number of categories of the dataset, category names, and paths to the training and validation sets. In the hyp.scratch.tiny.yaml file, set the value of mosaic to 0 and turn off mosaic data enhancement. The reason for this is that many photovoltaic panels are long in the image and almost occupy the entire width of the image. Enabling mosaic data enhancement may affect the training effect of the model. Finally, when starting the train.py script for model training, set the number of training rounds to 50. In each round of training, the performance of the model will be verified on the validation set. If the accuracy of the model is better than the previous version, a new model file will be saved. After the training is completed, run the val.py script to evaluate the accuracy of the model using the validation set data. After the training model is completed, proceed to the subsequent steps.

[0070] C: Jigsaw cutting. For aerial images of large photovoltaic scenes, since their resolution is usually very high (may reach tens of thousands of pixels × tens of thousands of pixels), it is not possible to directly apply the pre-trained YOLOv7-tiny model for target segmentation when resources are limited. To solve this problem, the aerial images need to be cut into smaller image blocks, each of which is 640x640 pixels in size, and these image blocks are saved in a folder. Next, these cut images can be input into the yolov7-tiny model for target segmentation to obtain the segmentation results of each image block.

[0071] D: Photovoltaic panel segmentation. First, use the trained yolov7-tiny model to detect and segment these cut images. The model predicts each pixel and outputs information such as confidence, segmentation contour or bounding box for each pixel. Then, extract the segmentation contour from the model's prediction results. Next, create a pure black image of the same size for each cut image. Place the extracted segmentation contour on the pure black image and fill the internal area of ​​the contour with white pixel values. This results in the final binary image segmentation result for each cut image, where white pixels represent the area where the photovoltaic panel is located and black pixels represent the background in the cut image.

[0072] E: Post-processing. The results of the photovoltaic panel segmentation performed by yolov7-tiny are not always perfect. The segmented area may not completely cover the photovoltaic panel, resulting in a difference of several pixels between the boundary and the actual photovoltaic panel area. If such a result is used directly, it may lead to incorrect matching of the number of small batteries. Therefore, the segmentation results of the photovoltaic panel need to be post-processed. :

[0073] Step E1: Extract the outline of the white area from the binary image in D and calculate the inscribed rectangle of each outline. Next, convert the corresponding original image into a grayscale image, and then calculate the pixel average of all inscribed rectangles in the grayscale image area. This average is the pixel average of the grayscale image corresponding to the photovoltaic panel area, called pix _ ave.

[0074] Among them, the method for calculating the inscribed rectangle of the outline is as follows:

[0075] Step 1: Calculate the minimum enclosing rectangle:

[0076] Assume {C1, C2, ..., C n} is the detected contour set. For each contour, calculate its minimum enclosing rectangle (MinAreaRect), which is a rotated rectangle that minimizes the area of ​​the rectangle covering the contour. Contour minimum enclosing rectangle

[0077] MinAreaRect(C i )={(x0, y0), (x1, y1), (x2, y2), (x3, y3)}

[0078] Where (x0, y0), (x1, y1), (x2, y2), (x3, y3) are the four vertices of the minimum enclosing rectangle.

[0079] Step 2: Traverse each angle and rotate the minimum enclosing rectangle to align it with the image coordinate system:

[0080] For each angle θ, calculate the inscribed rectangle:

[0081] 2.1. Rotation profile:

[0082] (x′, y′)=R(θ)·(x, y)

[0083] in is the rotation matrix.

[0084] 2.2. Calculate the minimum enclosing rectangle after rotation:

[0085] MinAreaRect(R(θ)·C i )={(x′0, y′0), (x′1, y′1), (x′2, y′2), (x′3, y′3)}

[0086] 2.3. Calculate the area of ​​the inscribed rectangle:

[0087] A(θ)=w(θ)·h(θ)

[0088] where w(θ) and h(θ) are the width and height of the inscribed rectangle respectively.

[0089] Step 3: Traverse all angles and find the inscribed rectangle with the largest area:

[0090]

[0091] Step 4: Finally, the inscribed rectangle can be represented by the coordinates of its upper left corner and its width and height:

[0092] R=(x best ,y best , W best ,h best )

[0093] Among them, (x best ,y best ) is the coordinate of the upper left corner of the inscribed rectangle, (w best ,h best ) are the width and height of the inscribed rectangle respectively.

[0094] At the same time, the formula for calculating the average value of all inscribed rectangles on the grayscale image is as follows:

[0095] Step 1: Assume there is a rotated rectangle with an angle:

[0096] Suppose there is a rotated rectangle with vertex coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3).

[0097] Step 2: Determine the pixel points within the rotated rectangle:

[0098] To determine which pixels (i, j) are inside the rotated rectangle, you can use a point-in-polygon test algorithm, such as the ray method or the angle sum method:

[0099] For each pixel (i, j), check whether it is inside the rotated rectangle.

[0100] This can be achieved using the cross product method: for a point (i, j), check whether it is to the left of each side of the rotated rectangle.

[0101] If (x1-x0)·(j-y0)-(y1-y0)·(i-x0)>0, then the point is to the left of the edge defined by (x0, y0) and (x1, y1). Perform similar tests on all edges of the rotated rectangle to determine if the point (i, j) is inside the rectangle.

[0102] Step 3: Calculate the average:

[0103] Assumptions It is the combination of all pixels in the rotated rectangle, and the pixel value is I(i k , j k ). The average value of pixels within the rotated rectangle is calculated as follows:

[0104]

[0105] Where: N is the number of pixels in the rotated rectangle. k , j k ) is the point (i k , j k ) is the pixel value at .

[0106] Step E2: Use the set range pix _ ave-20 <pix<pix _ ave+20 sets the pixel values ​​in this range to white pixels (255) in the grayscale image, and all other pixels to black pixels (0). Then the contour is expanded outward by 2 pixels, and all pixels in the corresponding area outside the contour on the binary image are set to black. After this processing, a binary image that accurately represents the photovoltaic panel cell area can be obtained.

[0107] Step E3: Expand all white pixel values ​​in the binary image by two pixels and connect the white areas at the same photovoltaic panel position to obtain the final accurate photovoltaic panel area.

[0108] Step E4: For each photovoltaic panel area, the coordinate values ​​of the four corner points are obtained by fitting quadrilaterals. Then, according to the position of the cut image in the original large image, the coordinate values ​​of the corner points of each quadrilateral are converted to obtain its position on the large image. Then, all quadrilaterals are traversed and the overlapping quadrilaterals are merged into a single quadrilateral. In this way, the correct coordinates of each photovoltaic panel area in the large image are obtained.

[0109] F: Convert the coordinates of the four corner points of each quadrilateral in the large image into GPS longitude and latitude coordinates. The specific implementation process is:

[0110] Step F1: For each given pixel, firstly calculate its offset relative to a certain reference point according to the image folder and image number where it is located.

[0111] Given a pixel point (x, y), the image and folder number (i, j), its offset (Δx, Δy) relative to a reference point can be calculated.

[0112] Assume the coordinates of the reference point are (x0, y0), then:

[0113] Δx=x-x0

[0114] Δy=y-y0

[0115] Step F2: Calculate the scale n of the image range according to the image level. Then use the x coordinate of the given pixel point and the image level to calculate its offset relative to the reference point. Finally, convert the offset into a longitude coordinate value.

[0116] The range scaling ratio n of the image is calculated according to the image level.

[0117] Assuming the image level is L, the scaling ratio n can be expressed as:

[0118] n=2 L

[0119] Using the given pixel x coordinate and image level L, calculate its offset relative to the reference point and convert it to latitude and longitude coordinate value λ.

[0120] Assume that the maximum longitude range of the earth is [-180°, 180°] and the width of the image is W, then:

[0121]

[0122] Step F3: Similarly, the scaling ratio n of the image range is calculated based on the image level. The offset relative to the reference point is calculated using the y coordinate of the given pixel and the image level. The latitude coordinate value is then calculated based on the calculated offset using the inverse hyperbolic sine function and the mathematical constant π.

[0123] Similarly, the scaling ratio n of the image range is calculated according to the image level.

[0124] Using the y-coordinate image level L of a given pixel, calculate its offset Δy relative to the reference point.

[0125] Then, based on the calculated offset, the inverse hyperbolic sine function and the mathematical constant π are used to calculate the longitude and latitude coordinates.

[0126] Assume that the maximum latitude and longitude range of the earth is [-85.05113°, 85.05113°] and the image height is H, then:

[0127]

[0128] Among them, sinh is the hyperbolic sine function and atan is the inverse tangent function.

[0129] Step F4: Combine F2 and F3 to get the real GPS value corresponding to the pixel coordinates of each corner point. According to the size of the panels in the photovoltaic station, each photovoltaic panel cell can be accurately numbered to facilitate the maintenance of the photovoltaic station in the subsequent process.

[0130] The present application provides a method for target segmentation of photovoltaic panels in complex scene images, which obtains high-resolution photovoltaic panel images by using drones for aerial photography. Then, with the help of the labelme annotation tool, the outline of the photovoltaic panel is accurately annotated, and the outline information is saved in a JSON file. Subsequently, the yolov7-tiny architecture is used to train the annotated photovoltaic panel data set, and the automatic segmentation of photovoltaic panels is realized through deep learning technology. At the same time, the accuracy and robustness of the model are guaranteed by combining the training set and the validation set.

[0131] In actual inspections, by performing jigsaw cutting on the images obtained by aerial photography, multiple images of 640x640 size are obtained. This cutting method not only maintains the clarity of the image, but also makes the segmented images easy to process. Subsequently, these cut images are input into the trained yolov7-tiny segmentation model to obtain the segmentation results of each photovoltaic panel. In order to ensure the accuracy of the segmentation results, this application refines the segmentation results and obtains the coordinate information of the four corner points of the photovoltaic panel. At this point, the conversion of image coordinates to longitude and latitude coordinates is successfully achieved by knowing the size and image coordinates of the battery. Finally, the precise numbering of each photovoltaic panel can be obtained, thereby providing reliable data support for subsequent maintenance and management.

[0132] The method provided in the present application not only greatly improves the efficiency of photovoltaic panel inspection and reduces the cost of manual inspection, but also ensures the safe operation of photovoltaic power stations.

[0133] In some embodiments, obtaining a photovoltaic panel data set includes: turning on a camera on a drone; recording a video of the photovoltaic area by flying the drone, and cutting the obtained video every 0.5 seconds to obtain an image; annotating the obtained image using labelme to obtain the outline of each photovoltaic panel; and saving all outline coordinates in a JSON file to form a data set.

[0134] In some embodiments, training is performed through the yolov7-tiny architecture, including: dividing the data set in the JSON file into a training set and a validation set in a ratio of 0.9:0.1; sending the images and labels in the training set to the yolov7-tiny network for training, and the network autonomously learns the segmentation of the photovoltaic panel area; after the training is completed, the images in the validation set are input into the yolov7-tiny network to judge the performance of the yolov7-tiny network for the segmentation of the photovoltaic panel area; when the segmentation performance reaches a threshold, proceed to the next step; when the segmentation performance does not reach the threshold, return to the starting step, and at the same time, increase the number of samples in the training set.

[0135] In some embodiments, jigsaw cutting includes: using a drone to fly and photograph the photovoltaic station and perform real-time jigsaw puzzle to obtain an aerial photo of the entire area of ​​the photovoltaic station; cutting the photographed jigsaw puzzle into multiple 640x640 images, and the intersecting pixel area of ​​adjacent images after cutting is 32 pixels.

[0136] In some embodiments, the segmentation result of each photovoltaic panel is obtained, including: inputting each cut image into a trained yolov7-tiny network to obtain the segmentation result predicted by the model; converting the contour result obtained by the segmentation into a binary image, and the white area in the binary image is the photovoltaic panel area predicted by the model.

[0137] It should be noted that although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0138] For further information, please refer to Figure 2 , Figure 2 An exemplary structural block diagram of a photovoltaic panel target segmentation system 200 under a complex scene image according to an embodiment of the present application is shown. The system includes:

[0139] The acquisition module 210 is used to acquire a photovoltaic panel data set; the photovoltaic panel image is photographed by a drone, and the outline of the photovoltaic panel is marked by a labelme tool, and the outline information of the photovoltaic panel is saved in a JSON file;

[0140] The segmentation module 220 is used to train the photovoltaic panel segmentation model; the labeled photovoltaic panel data set is trained through the yolov7-tiny architecture to achieve the segmentation of the photovoltaic panel;

[0141] A cutting module 230 is used to cut the jigsaw puzzle of the photovoltaic panels to be numbered into multiple images of 640x640 size;

[0142] An input module 240 is used to input the cut image into a trained yolov7-tiny segmentation model to obtain a segmentation result for each photovoltaic panel;

[0143] The processing module 250 is used to process the segmentation results to obtain the coordinates of the four corner points of the photovoltaic panel; by knowing the battery size and image coordinates, the image coordinates are converted into longitude and latitude coordinates.

[0144] It should be understood that the units or modules described in the system 200 are similar to those described in the reference Figure 1 The various steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the system 200 and the units contained therein, and will not be repeated here. The system 200 can be pre-implemented in the browser or other security applications of the electronic device, and can also be loaded into the browser or its security application of the electronic device by downloading or the like. The corresponding units in the system 200 can cooperate with the units in the electronic device to implement the solution of the embodiment of the present application.

[0145] The present application also shows a schematic diagram of the structure of a computer system of a terminal device or server suitable for implementing an embodiment of the present application.

[0146] The computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part into the random access memory (RAM). In the RAM, various programs and data required for system operation are also stored. The CPU, ROM and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0147] The following components are connected to the I / O interface: an input section including a keyboard, a mouse, etc.; an output section including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a LAN card, a modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that a computer program read therefrom is installed into the storage section as needed.

[0148] In particular, according to the embodiments of the present disclosure, the above reference Figure 1 The described process may be implemented as a computer software program or provide related processing services in the form of an HTTP interface. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly embodied on a machine-readable medium, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program may be downloaded and installed from a network through the communication section, and / or installed from a removable medium.

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units or modules involved in the embodiments described in the present application may be implemented by software or by hardware. The described units or modules may also be provided in a processor, for example, may be described as: a processor includes a first sub-region generation unit, a second sub-region generation unit, and a display area generation unit. Among them, the names of these units or modules do not constitute a limitation on the unit or module itself in certain circumstances, for example, the display area generation unit may also be described as "a unit for generating a display area for text based on a first sub-region and a second sub-region".

[0151] As another aspect, the present application further provides a computer-readable storage medium, which may be the computer-readable storage medium included in the aforementioned device in the above embodiment; or may be a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the aforementioned programs are used by one or more processors to execute the text generation method for transparent window envelopes described in the present application.

[0152] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A photovoltaic panel target segmentation method in complex scene images, characterized in that: The method includes: S1: Obtain the photovoltaic panel dataset; use a drone to take images of photovoltaic panels, use the labelme tool to mark the outlines of the photovoltaic panels, and save the outline information of the photovoltaic panels in a JSON file; S2: Training the photovoltaic panel segmentation model; training the labeled photovoltaic panel dataset through the yolov7-tiny architecture to achieve photovoltaic panel segmentation; S3: Cut the jigsaw puzzle of the photovoltaic panels to be numbered into multiple images of 640x640 size; S4: Input the cut image into the trained yolov7-tiny segmentation model to obtain the segmentation result of each photovoltaic panel; S5: Process the segmentation results to obtain the coordinates of the four corner points of the photovoltaic panel; by knowing the battery size and image coordinates, convert the image coordinates into longitude and latitude coordinates.

2. The photovoltaic panel target segmentation method in complex scene images according to claim 1 is characterized in that: In step S1, obtaining a photovoltaic panel data set includes: S11: Turn on the camera on the drone; S12: Recording a video of the photovoltaic area by flying a drone, and cutting the obtained video every 0.5 seconds to obtain an image; S13: label the obtained image using labelme to obtain the outline of each photovoltaic panel; S14: Save all contour coordinates in a JSON file to form a dataset.

3. The photovoltaic panel target segmentation method in complex scene images according to claim 2, characterized in that: In step S2, the training is performed using the yolov7-tiny architecture, including: The data set in the JSON file is divided into a training set and a validation set in a ratio of 0.9:0.1; The images and labels in the training set are fed into the yolov7-tiny network for training, and the network autonomously learns the segmentation of the photovoltaic panel area; After the training is completed, the images in the validation set are input into the yolov7-tiny network to judge the performance of the yolov7-tiny network in photovoltaic panel area segmentation; When the segmentation performance reaches the threshold, step S3 is performed; when the segmentation performance does not reach the threshold, the process returns to step S1 and increases the number of samples in the training set.

4. The photovoltaic panel target segmentation method in complex scene images according to claim 3 is characterized in that: In step S3, the jigsaw cutting includes: Use drones to fly and photograph photovoltaic stations and perform real-time mosaics to obtain aerial photos of the entire photovoltaic station area; The captured puzzle is cut into multiple images of 640x640 size, and the intersecting pixel area of ​​adjacent images after cutting is 32 pixels.

5. The photovoltaic panel target segmentation method in complex scene images according to claim 4, characterized in that: In step S4, the segmentation result of each photovoltaic panel is obtained, including: Input each cut image into the trained yolov7-tiny network to obtain the segmentation result predicted by the model; The contour results obtained by segmentation are converted into a binary image, and the white area in the binary image is the photovoltaic panel area predicted by the model.

6. A photovoltaic panel target segmentation system under complex scene images, characterized in that: The system includes: The acquisition module is used to obtain the photovoltaic panel dataset; the photovoltaic panel image is taken by a drone, and the outline of the photovoltaic panel is marked using the labelme tool, and the outline information of the photovoltaic panel is saved in a JSON file; The segmentation module is used to train the photovoltaic panel segmentation model. The labeled photovoltaic panel dataset is trained through the yolov7-tiny architecture to achieve the segmentation of photovoltaic panels. A cutting module is used to cut the jigsaw puzzle of the photovoltaic panels that need to be numbered into multiple images of 640x640 size; The input module is used to input the cut image into the trained yolov7-tiny segmentation model to obtain the segmentation results of each photovoltaic panel; The processing module is used to process the segmentation results to obtain the coordinates of the four corner points of the photovoltaic panel; by knowing the battery size and image coordinates, the image coordinates are converted into longitude and latitude coordinates.

7. The photovoltaic panel target segmentation system under complex scene images according to claim 6, characterized in that: The step of obtaining a photovoltaic panel data set comprises: S11: Turn on the camera on the drone; S12: Recording a video of the photovoltaic area by flying a drone, and cutting the obtained video every 0.5 seconds to obtain an image; S13: label the obtained image using labelme to obtain the outline of each photovoltaic panel; S14: Save all contour coordinates in a JSON file to form a dataset.

8. The photovoltaic panel target segmentation system under complex scene images according to claim 7, characterized in that: The training is performed through the yolov7-tiny architecture, including: The data set in the JSON file is divided into a training set and a validation set in a ratio of 0.9:0.1; The images and labels in the training set are fed into the yolov7-tiny network for training, and the network autonomously learns the segmentation of the photovoltaic panel area; After the training is completed, the images in the validation set are input into the yolov7-tiny network to judge the performance of the yolov7-tiny network in photovoltaic panel area segmentation; When the segmentation performance reaches the threshold, step S3 is performed; when the segmentation performance does not reach the threshold, the process returns to step S1 and increases the number of samples in the training set.

Citation Information

Patent Citations

  • Photovoltaic panel infrared image segmentation method, storage medium and electronic equipment

    CN116188502A

  • Digital method for unmanned aerial vehicle to automatically inspect and position photovoltaic panel defects

    CN116773598A

  • Visible light photovoltaic support panorama automatic identification method based on improved global self-attention mechanism

    CN117671536A

  • Layout identification method and device of photovoltaic optimizer, electronic equipment and storage medium

    CN118097705A

  • KR20210099450A

Cited By

  • Photovoltaic module defect diagnosis method, device and equipment

    CN119338820A