Real-time panoramic stitching method and system for photovoltaic power station based on drone video stream

Abstract

The present invention discloses a real-time panoramic stitching method and system of a photovoltaic power station based on a drone video stream, and relates to the technical field of panoramic stitching. The method comprises: obtaining a video of a photovoltaic component shot by a drone, inputting it into a target detection model, and identifying a component target frame of the photovoltaic component. Based on a tracking algorithm, the component identification of each photovoltaic component is determined. According to the component information and the photovoltaic component video, a component positioning method is used to determine the sorting and positioning of the photovoltaic component. An image generation method is used to generate a component image of the photovoltaic component. Finally, based on the component image and the sorting and positioning, the panoramic stitching of the photovoltaic power station is completed. The present invention is based on component detection, tracking, sorting and positioning, image generation and panoramic stitching. Real-time reconstruction of the panoramic view of the photovoltaic power station can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of panoramic stitching, and in particular to a real-time panoramic stitching method and system for a photovoltaic power station based on unmanned aerial vehicle video streams. Background Art

[0002] Existing methods for reconstructing panoramic maps of photovoltaic power plants include: using CAD images for plane matching; using three-dimensional reconstruction methods such as aerial triangulation; and creating orthophotos of the entire photovoltaic power plant at a higher altitude. It is difficult to observe component-level image information, and most of them are offline panoramic stitching, which usually requires a lot of computing resources. The real-time online reconstruction method is usually achieved through image stitching and feature matching. Due to the limited scale of processing data, it has limitations in actual power plant applications. Summary of the invention

[0003] The purpose of the present invention is to provide a real-time panoramic stitching method and system for a photovoltaic power station based on a drone video stream, which can ensure real-time reconstruction of a panoramic map of a large photovoltaic power station.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] In a first aspect, the present invention provides a real-time panoramic stitching method for a photovoltaic power station based on a drone video stream, comprising:

[0006] Get a video of the PV panels taken by a drone.

[0007] The photovoltaic module video is input into a trained target detection model to identify the module target frame of each photovoltaic module in the photovoltaic module video.

[0008] According to the photovoltaic component video and the component target frame of the photovoltaic component, based on a tracking algorithm, a component identifier of each photovoltaic component is determined.

[0009] For each of the photovoltaic components, the component sorting and positioning of the photovoltaic components is determined based on the component positioning method according to the component information of the photovoltaic component and the photovoltaic component video; the component information includes a component target frame and a component identification.

[0010] For each of the photovoltaic components, a component image of the photovoltaic component is determined based on the component information of the photovoltaic component and the photovoltaic component video and based on an image generation method.

[0011] Based on the component images and component sorting and positioning of each photovoltaic component, each photovoltaic component is arranged in rows and columns to complete the panoramic stitching of the photovoltaic power station.

[0012] Optionally, before inputting the photovoltaic module video into the trained target detection model, the method further includes:

[0013] The photovoltaic module video is processed to obtain a processed photovoltaic module video; the video processing includes segmentation processing and setting a pixel threshold.

[0014] Optionally, the training process of the target detection model is:

[0015] Sample infrared image data of photovoltaic modules in a photovoltaic power station is obtained; the sample infrared image data is obtained through historical photovoltaic module videos.

[0016] The sample infrared image data is annotated with component frames to obtain annotated sample infrared image data.

[0017] Based on the labeled sample infrared image data, the YOLOv8 target detection algorithm is trained to obtain a trained target detection model; wherein the YOLOv8 target detection algorithm is an algorithm obtained by using the improved CSPDarket53 as the backbone network, replacing the CSP module in the Backbone with the C2f module, using the PAN+FPN structure in the Neck part to construct a feature pyramid, and using a decoupling head structure as the detection part on the basis of the YOLOv5 target detection algorithm.

[0018] Optionally, for each of the photovoltaic components, according to the component information of the photovoltaic component and the photovoltaic component video, based on a component positioning method, determining the component sorting and positioning of the photovoltaic component specifically includes:

[0019] According to the component information of the photovoltaic components and the photovoltaic component video, component filtering is performed on photovoltaic components that appear repeatedly in different frames in the photovoltaic component video to obtain filtered photovoltaic components.

[0020] According to the component target frame of the filtered photovoltaic components, the photovoltaic components in the same frame of the photovoltaic component video are sorted in rows and columns to obtain sorted component position data.

[0021] According to the sorted component position data and the component identification of the filtered photovoltaic component, the component identification of the photovoltaic component in each frame in the photovoltaic component video is compared in chronological order, and the component position information is updated. When the comparison is to the last frame in the photovoltaic component video, the component sorting and positioning of the photovoltaic component is obtained.

[0022] Optionally, for each of the photovoltaic components, determining the component image of the photovoltaic component based on the component information of the photovoltaic component and the photovoltaic component video and based on an image generation method specifically includes:

[0023] According to the component information of the photovoltaic component and the photovoltaic component video, image processing is performed on the photovoltaic component video based on a distortion image correction method, a component image extraction method, an image fusion method, a defect filtering method and a reflection filtering method to obtain a component image of the photovoltaic component.

[0024] Optionally, the tracking algorithm adopts a SORT target tracking algorithm.

[0025] Optionally, the photovoltaic module video is a real-time infrared video stream.

[0026] In a second aspect, the present invention provides a real-time panoramic stitching system for a photovoltaic power station based on drone video streams, comprising:

[0027] The video acquisition module is used to acquire the photovoltaic module video taken by the drone.

[0028] The detection module is used to input the photovoltaic component video into a trained target detection model to identify the component target frame of each photovoltaic component in the photovoltaic component video.

[0029] The tracking module is used to determine the component identification of each photovoltaic component based on the photovoltaic component video and the component target frame of the photovoltaic component based on a tracking algorithm.

[0030] The sorting and positioning module is used to determine the component sorting and positioning of each photovoltaic component according to the component information of the photovoltaic component and the photovoltaic component video based on the component positioning method; the component information includes a component target frame and a component identification.

[0031] The imaging module is used to determine, for each photovoltaic component, a component image of the photovoltaic component according to the component information of the photovoltaic component and the photovoltaic component video based on an image generation method.

[0032] The splicing module is used to arrange each photovoltaic component in rows and columns based on the component images and component sorting and positioning of each photovoltaic component, so as to complete the panoramic splicing of the photovoltaic power station.

[0033] Optionally, also include:

[0034] The video processing module is used to perform video processing on the photovoltaic module video to obtain a processed photovoltaic module video; the video processing includes segment processing and setting a pixel threshold.

[0035] Optionally, it also includes: a training module for training the target detection model, specifically including:

[0036] The image acquisition submodule is used to acquire sample infrared image data of photovoltaic modules in a photovoltaic power station; the sample infrared image data is obtained through historical photovoltaic module videos.

[0037] The labeling submodule is used to label the sample infrared image data with component frames to obtain the labeled sample infrared image data.

[0038] The training submodule is used to train the YOLOv8 target detection algorithm based on the labeled sample infrared image data to obtain a trained target detection model; wherein the YOLOv8 target detection algorithm is an algorithm obtained by using the improved CSPDarket53 as the backbone network, replacing the CSP module in the Backbone with the C2f module, using the PAN+FPN structure in the Neck part to construct a feature pyramid, and using the decoupling head structure as the detection part on the basis of the YOLOv5 target detection algorithm.

[0039] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The present invention provides a real-time panoramic stitching method and system for a photovoltaic power station based on a drone video stream, the method comprising: obtaining a photovoltaic module video taken by a drone; inputting the photovoltaic module video into a trained target detection model, identifying a module target frame of each photovoltaic module in the photovoltaic module video; determining a module identification of each photovoltaic module based on a tracking algorithm according to the photovoltaic module video and the module target frame of the photovoltaic module; for each photovoltaic module, determining a module sorting and positioning of the photovoltaic module based on a module positioning method according to the module information of the photovoltaic module and the photovoltaic module video; the module information comprises a module target frame and a module identification; for each photovoltaic module, determining a module image of the photovoltaic module based on an image generation method according to the module information of the photovoltaic module and the photovoltaic module video; and arranging each photovoltaic module in rows and columns based on the module image and the module sorting and positioning of each photovoltaic module to complete the panoramic stitching of the photovoltaic power station. The present invention obtains photovoltaic module videos collected by drones; uses a target detection model to perform component recognition on the photovoltaic module videos and detects the boundary boxes of the photovoltaic modules; transmits the component detection results to a target tracking algorithm to uniquely track each component; and accurately locates the global coordinates of each component through a sorting positioning and image generation strategy, thereby realizing real-time panoramic stitching of photovoltaic power stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A schematic flow chart of a method for real-time panoramic stitching of a photovoltaic power station based on drone video streams provided in the first embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the LabelImg annotation interface provided in the first embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the YOLOv8 network structure provided in Example 1 of the present invention;

[0045] Figure 4 This is a graph of adjacent frame component tracking results provided by the first embodiment of the present invention;

[0046] Figure 5 A flowchart of the SORT tracking algorithm provided in the first embodiment of the present invention;

[0047] Figure 6 A local sorting diagram of components provided in Embodiment 1 of the present invention;

[0048] Figure 7 A schematic diagram of the global positioning of components provided in Embodiment 1 of the present invention;

[0049] FIG8 is a schematic diagram of distortion correction of a power station 2 image provided by Embodiment 1 of the present invention; FIG8(a) is a distorted image; FIG8(b) is a corrected image;

[0050] Fig. 9 A diagram of a distortion correction procedure for power station 2 provided in Embodiment 1 of the present invention;

[0051] Fig.10 A schematic diagram of an infrared image of component ID27 in a specific frame provided in the first embodiment of the present invention;

[0052] Fig.11 Schematic diagram of infrared images of different ID components provided in the first embodiment of the present invention;

[0053] Fig.12 A schematic diagram of the movement process of a component in a video provided in Embodiment 1 of the present invention;

[0054] Fig.13 A schematic diagram of the K-means clustering algorithm provided in Example 1 of the present invention;

[0055] Fig.14 Schematic diagram of abnormal thermal reflection of components in different frames provided in Embodiment 1 of the present invention;

[0056] Fig.15 A panoramic stitching flow chart provided in the first embodiment of the present invention;

[0057] Fig.16A panoramic stitching effect diagram of four partial areas of a power station provided in the first embodiment of the present invention;

[0058] Fig.17 This is a three-dimensional reconstruction rendering of a partial area of ​​power station 4 provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The purpose of the present invention is to provide a real-time panoramic stitching method and system for a photovoltaic power station based on a drone video stream, which can ensure real-time reconstruction of a panoramic map of a large photovoltaic power station.

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Embodiment 1

[0063] like Figure 1 As shown, this embodiment provides a real-time panoramic stitching method of a photovoltaic power station based on a drone video stream, comprising:

[0064] Step 101: Obtain a video of a photovoltaic module taken by a drone.

[0065] Step 102: input the photovoltaic module video into a trained target detection model to identify the module target frame of each photovoltaic module in the photovoltaic module video.

[0066] Step 103: Determine the component identification of each photovoltaic component based on the photovoltaic component video and the component target frame of the photovoltaic component and a tracking algorithm.

[0067] Step 104: for each of the photovoltaic components, according to the component information of the photovoltaic component and the photovoltaic component video, based on a component positioning method, determine the component sorting and positioning of the photovoltaic component; the component information includes a component target frame and a component identifier.

[0068] Step 105: for each of the photovoltaic components, determine a component image of the photovoltaic component according to the component information of the photovoltaic component and the photovoltaic component video based on an image generation method.

[0069] Step 106: Based on the component images and component sorting and positioning of each photovoltaic component, each photovoltaic component is arranged in rows and columns to complete the panoramic stitching of the photovoltaic power station.

[0070] When executing step 101, the specific steps may be as follows:

[0071] Equipped with edge computers and thermal infrared cameras on drones. For secondary development of drones, the video stream from the thermal infrared camera can be obtained in real time through the SDK interface of the drone, and the infrared video of the component can be transmitted to the ground server in real time through the communication protocol.

[0072] Before executing step 102, the following steps may also be included:

[0073] The photovoltaic module video is processed to obtain a processed photovoltaic module video; the video processing includes segmentation processing and setting a pixel threshold.

[0074] Specifically, after acquiring the video stream, first manually segment the video stream to ensure that there are no turns in each video segment. This not only improves the accuracy of subsequent target tracking, but also facilitates the positioning of strings in different columns during panoramic stitching. Secondly, manually set the pixel threshold for detection in the video to ensure that adjacent video segments do not repeatedly detect and track the same row / column string. This is because there will be overlapping areas when the drone flies along the row / column of strings, so setting the pixel threshold ensures the accuracy of subsequent component positioning. The processed video segment can then be sent to the YOLOv8 target detection and SORT tracking algorithm to output the target box and unique ID of the component. Figure 4 Tracking results for adjacent frame components.

[0075] Before executing step 102, the target detection model is further trained, which may be specifically as follows:

[0076] The training process of the target detection model is:

[0077] Step 201: Acquire sample infrared image data of photovoltaic modules in a photovoltaic power station; the sample infrared image data is obtained through historical photovoltaic module videos.

[0078] Step 202: labeling the sample infrared image data with component frames to obtain labeled sample infrared image data.

[0079] Step 203: Based on the labeled sample infrared image data, the YOLOv8 target detection algorithm is trained to obtain a trained target detection model; wherein the YOLOv8 target detection algorithm is an algorithm obtained by using the improved CSPDarket53 as the backbone network, replacing the CSP module in the Backbone with the C2f module, using the PAN+FPN structure in the Neck part to construct a feature pyramid, and using the decoupling head structure as the detection part on the basis of the YOLOv5 target detection algorithm.

[0080] Specifically, the infrared image data of photovoltaic module samples can be prepared as follows:

[0081] like Figure 2 As shown in the figure, the infrared image dataset of photovoltaic power station components is obtained in advance through drone inspection, 300 images are selected, and the component frames are annotated with the LabelImg annotation tool, and the label name is PV.

[0082] Specifically, the training target detection model is as follows:

[0083] Since the component tracking algorithm adopted in this embodiment is a two-stage paradigm of target detection and tracking, the specific tracking process is: first pass through the YOLOv8 target detection network to detect the components in the video frame, and then input the SORT tracking algorithm to predict and match the same component of different video frames, and assign a unique identifier (ID). Therefore, it is necessary to train the YOLOv8 target detection network in advance to provide model weights for the component tracking algorithm of this embodiment. The specific training process is: train 300 rounds of 300 labeled infrared images, learn component features, continuously reduce the loss function, and reach the optimal value.

[0084] Among them, the YOLOv8 target detection network structure is as follows Figure 3 As shown in the figure, the YOLOv8 network structure is mainly composed of backbone, neck and head structures.

[0085] (1) Backbone structure. YOLOv8 uses the improved CSPDarket53 as the backbone network, downsamples the input features five times, and obtains features of five different scales in turn. The structure of the backbone network is shown in Figure \. The CSP (Cross Stage Partial) module in Backbone is replaced by the C2f module (n represents the number of Bottlenecks). The C2f module uses gradient shunt connection to enrich the information flow of the feature extraction network while maintaining lightweight. The CBS module performs convolution operations on the input information, then performs batch normalization, and finally uses the SiLU activation function to obtain the output result. BackBone finally uses the SPPF (Spatial Pyramid Pooling Fast) module to pool the input feature map into a fixed-size map for adaptive size output. Compared with the structure of SPP (Spatial Pyramid Pooling), SPPF connects three maximum pooling layers in gradient order, reduces the computational workload, and has lower latency.

[0086] (2) Neck structure. Inspired by PANet, YOLOv8 designed a PAN+FPN structure in the Neck part to construct a feature pyramid. PAN+FPN constructs a top-down and bottom-up network structure, realizes the complementarity of shallow position information, and fully integrates multi-scale information. Except that the C3 module in FPN-PAN is replaced by the C2f module, the rest is basically the same as the PAN+FPN structure of YOLOv5.

[0087] (3) Head structure. The detection part of YOLOv8 adopts a decoupled head structure. This decoupled head structure sets up two independent branches for object classification and predicted bounding box regression, and uses different loss functions for these two tasks. For classification tasks, Binary Cross-Entropy Loss (BCE Loss) is used. For the predicted bounding box regression task, Distribution Focal Loss (DFL) and CIoU are used. This detection structure helps to improve detection accuracy and accelerate model convergence. YOLOv8 is an anchor-free detection model that can concisely specify positive and negative samples. In addition, it also uses a task-aligned assigner to dynamically assign samples, which helps to improve the detection accuracy and robustness of the model.

[0088] When executing step 103, the specific steps may be as follows:

[0089] The tracking algorithm uses the SORT target tracking algorithm, which is a target tracking algorithm based on the Kalman filter and the Hungarian algorithm. The core idea is to first detect the target in the video through the target detection algorithm and represent it as a detection frame. SORT can be used with various target detectors (such as YOLO, FasterR-CNN, etc.) to extract the target position information in each frame. Then the Kalman filter algorithm is used to estimate and predict the state of the target, and finally the Hungarian algorithm is used to match the target detection frame of the current frame with the target tracked in the previous frame to achieve target tracking. Figure 5 The process of the SORT tracking algorithm is demonstrated. The basic steps are summarized as: component detection, Kalman filter component prediction, Hungarian IoU target matching, and target ID creation and destruction.

[0090] When executing step 104, the specific steps may be as follows:

[0091] Step 301: Filter photovoltaic components that appear repeatedly in different frames of the photovoltaic component video according to the component information of the photovoltaic components and the photovoltaic component video to obtain filtered photovoltaic components.

[0092] Step 302: sorting the photovoltaic components in the same frame of the photovoltaic component video in rows and columns according to the component target frames of the filtered photovoltaic components to obtain sorted component position data.

[0093] Step 303: According to the sorted component position data and the component identification of the filtered photovoltaic component, the component identification of the photovoltaic component in each frame in the photovoltaic component video is compared in chronological order, and the component position information is updated. When the comparison is to the last frame in the photovoltaic component video, the component sorting and positioning of the photovoltaic component is obtained.

[0094] Specifically, the component filtering method is as follows:

[0095] Although there are no turning scenes in the components in the current video, there are overlapping scenes between adjacent video segments of the same power station, which will cause the same component to be detected and tracked repeatedly, affecting the efficiency and accuracy of the final panoramic stitching. Here, by setting the threshold range, the components detected in the previous video segment are filtered out to ensure that there is no repeated detection.

[0096] Specifically, the method for local sorting of components is:

[0097] Local sorting of components refers to correctly sorting the components by rows and columns in the same frame based on component target detection, see Figure 6The target detection algorithm obtains the center point coordinates (x, y) and width and height (w, h) of each component in the image coordinate system, and the initial default is sorted by confidence. Figure 6 In the figure (a), the physical meaning of the first row is that the (x, y) of the component ID 3 in the first frame is (359.999, 433), (w, h) is (125.003, 72.4745), and the initial ID is: 3, 6, 2, 1, 4, 8, 7, 5. First, sort the components according to the y value of the ordinate of the center point of the component to sort out the components belonging to the same row. Figure 6 The IDs in the graph (b) are sorted by y value as follows: 6, 5, 7, 8, 2, 4, 1, 3. By comparing the y values ​​between each other, different rows can be classified, such as Figure 6 (b) The y values ​​of ID: 8 and ID: 2 are quite different, so they are considered to be in different rows. Then, the components in each row are sorted by x coordinates. Figure 6 The final order in (c) is ID: 5, 6, 7, 8, 1, 2, 3, 4, and Figure 6 The component positions correspond in (c).

[0098] Specifically, the global positioning of components is as follows:

[0099] Component global positioning specifically refers to finding the location information of the component and its global position in the power station coordinate system in different frames based on component tracking. The component coordinates sorted in the frame in the previous step are input into the target tracking algorithm, and the tracking algorithm assigns a unique ID to each component. Taking the i-th frame video and the i+1-th frame as an example, find the ID corresponding to the last row of the i-th frame, and compare it with each row of the ID of the i+1-th frame. There are two situations: one is that when comparing to the last row of the i+1-th frame, it is still the same as the last row of the i-th frame. This means that no new target appears in the i+1 frame. At this time, it is not updated, and the i-th frame continues to be compared with the i+n-th (i=2, 3, ...) frame until a new target appears. See Figure 7 (b) As shown. The second is to correspond to the non-last row ID of the i+nth (i=2, 3, ...)th frame, then add the IDs corresponding to the subsequent rows in the i+nth frame to the i-th frame ID, and continue to find the last row of the i+nth frame and compare it with the i+n+1th frame, and so on. Figure 7 (c) as shown.

[0100] At this point, the position of the next row has been found, and the position is updated to obtain the global positioning. Figure 7 (d) Add the new component row IDs: 25, 26, 27, 28 to the end of the original component row, and increase the row coordinates of the new row by 1 based on the original component row. Figure 7In (d), the components with IDs 25, 26, 27, and 28 are updated to row 7. In this way, the unique location of the component in the power station coordinate system is obtained. For example, the component with ID 25 is located in the first column of the seventh row of the third string area of ​​the photovoltaic power station, marked as 3-7-1.

[0101] When executing step 105, the specific steps may be as follows:

[0102] According to the component information of the photovoltaic component and the photovoltaic component video, image processing is performed on the photovoltaic component video based on a distortion image correction method, a component image extraction method, an image fusion method, a defect filtering method and a reflection filtering method to obtain a component image of the photovoltaic component.

[0103] Specifically, distortion correction is as follows:

[0104] When the camera internal parameters and distortion parameters are known, the distortion correction of photovoltaic infrared video is performed using the existing OpenCV tools. The sub-images (a) and (b) of Figure 8 are the comparison of the video frames of power station 2 before and after correction. In this embodiment, only power station 2 is subjected to distortion correction. Power station 2 has obvious distortion due to improper handling of the camera and sea surface position in the offshore shooting scene. The other three power stations are basically free of distortion.

[0105] Specifically, the component image is extracted as follows:

[0106] The extraction of component images is performed synchronously during the component tracking process. By counting the video tracking data of power stations 1, 2, 3, and 4, it is found that each component appears in about 100 frames in the video on average, and the average movement distance of the same component in adjacent frames is only 0.5 pixels. In order to reduce the waste of resources and improve the efficiency of subsequent image fusion, this embodiment adopts frame extraction for component images, and only extracts videos with a frame number that is a multiple of 5.

[0107] In order to correctly correspond to the position of the components after sorting, the extracted component images need to be written with specific names. Fig. 9 The component storage directory extracted in this experiment is shown. The meaning of each level of naming is explained in the annotations in the figure. Among them, id_0 and id_1 are the storage locations of components with IDs 0 and 1 after the component graph is extracted. The sub-directory of id_0 saves the extraction results of the ID: 0 component in all specific frames (frame values ​​are multiples of 5). Fig.10 The infrared image of component ID: 27 in a specific frame is shown. This naming method not only facilitates the next step of component image fusion, but also ensures the correct matching of component image information and position information.

[0108] Among them, component image fusion is specifically as follows:

[0109] In the photovoltaic power station panoramic stitching link of this embodiment, each component is required to have only one corresponding image, so the multiple component images extracted under the same component in the previous step need to be fused. Since the image features of the same component are basically the same, and the infrared image will carry temperature information, this scene cannot use a complex image fusion algorithm, so the embodiment uses the average fusion method.

[0110] The specific steps are as follows: resize the component image collection to a uniform size to ensure the same resolution. Stack the pixel data of all images into an array. If there are N images, you get an array of shape (N, height, width, channels), where height and width are the dimensions of the image, and channels is usually 3 (for RGB images). For each pixel position in the array, divide N to calculate the average value of each RGB channel. The calculated average pixel value creates a new image matrix, which is the fused image. Fig.11 Shows the effect of merging different ID components, saved in Fig. 9 In the displayed all_track_id folder.

[0111] Specifically, the abnormality handling process includes defect filtering and reflection filtering, which can be as follows:

[0112] The specific details of incomplete filtering are:

[0113] The purpose of incomplete filtering is to ensure that the component images involved in the fusion are complete components. During the unmanned flight process, the components in the video screen will move. Fig.12 The three sub-figures in the figure show the position change process of the component in the picture. When the component just enters the video screen and is about to disappear in the screen, the extracted component is incomplete.

[0114] The specific steps are as follows: During the component tracking process, record the width and height (w, h) information of all components detected in the first two frames of the power station video. Use the K-means clustering algorithm to classify the width and height of all components. Divide the width into two categories: full width and half width, and divide the height into two categories: full height and half height. Then calculate the average pixel length (wavg, havg) of the full height and full width respectively as the filtering threshold. After extracting the component image, filter out the components with image size smaller than (wavg, havg). Fig.13 The width, height and length clustering results of component images extracted from 17 components in different frames are shown. It can be judged that 10 component images are incomplete and located at the edge of the video screen.

[0115] Specifically, the reflection filtering is as follows:

[0116] When using a thermal imaging camera to monitor PV modules, different camera angles may capture solar reflections, which will affect the measurement of the infrared temperature of the module and be misjudged as hot spots in subsequent PV module defect detection. Abnormal bright spots will appear in the infrared image of the module in the video, affecting the effect of module image fusion. Because the sun's reflection is not always fixed in the same position, it will only affect part of the video frame. Fig.14 This is the extraction effect of the same component in different frames. It can be observed that abnormal thermal reflection appears in some frames in the middle.

[0117] The specific steps are as follows: the color information of the infrared image corresponds to the temperature information. On the basis of the uniform size of the component images, find the connected area of ​​the high temperature points on each component and the position coordinates of this area. The connected area of ​​the high temperature points is set as follows: after obtaining the temperature value of the component, find the highest temperature point, read the pixels within 20 pixels squared of the highest temperature point and within a difference of 5 degrees Celsius. The closed area formed by these pixels is the connected area of ​​the high temperature points. Compare the coordinates of the high temperature point area between different frame images of each component. If there are component images with a difference of more than 10 pixels in the coordinates of the high temperature point area, it means that there are solar reflections on these images instead of hot spot anomalies, and these component images are filtered out.

[0118] When executing step 106, the specific steps may be as follows:

[0119] Based on the component images and component sorting and positioning of the photovoltaic modules, a panoramic stitching system of the entire photovoltaic power station is completed.

[0120] Specifically, the panoramic stitching effect is as follows: Fig.16 As shown, Fig.16 The splicing effect of the four parts of the power station is shown. Each component after splicing has a unique position code. The specific coding meaning of the components 2-5-7 in the figure is: the second sub-area ( Fig.16 The panoramic view of power station 4 in the middle shows the fifth row and seventh column of components. All panoramic splicing effects of power stations 1, 2, 3, and 4 are in Fig.16 It can be observed that the components numbered 2-5-7 and 2-5-8 have hot spot defects.

[0121] Fig.17 yes Fig.15 The effect of 3D reconstruction of the display area can be compared with the effect of panoramic stitching. The comparison shows that panoramic stitching can better preserve the texture characteristics of the components, help to find defects in the components, and provide assistance for the refined inspection and operation and maintenance of photovoltaic power stations.

[0122] In summary, the splicing process of this embodiment is as follows: Fig.15As shown in the figure, firstly, the video of photovoltaic components taken by the drone is obtained, and the infrared video is processed in segments to ensure that each video segment is a straight line of the drone; the target detection model is used to identify the target frame of the photovoltaic component and detect the photovoltaic component; the detected component position information and video stream are sent to the tracking algorithm to track each component uniquely; one channel is responsible for sorting and positioning the components, and the specific process includes: component filtering, local sorting of components and global positioning of components. Another channel is responsible for the image generation of the components, and the main process includes: distortion correction, component image extraction and image fusion. When the component image is generated, the extracted incomplete components need to be filtered, and the reflective components caused by sunlight need to be filtered out. Finally, the unique position and image information of each photovoltaic component is obtained, and the relative coordinates in the position information are used to arrange each component in rows and columns to complete the panoramic stitching of the photovoltaic power station.

[0123] This embodiment applies the panoramic stitching method to four large photovoltaic power stations. Taking a 5MWp hydroelectric power station (power station 1) as an example, the processing of 9064 frames of infrared video can be completed within 45.5 seconds, including the automatic complete process of photovoltaic module detection, image extraction, positioning and stitching restoration.

[0124] Table 1 Panoramic stitching time distribution table

[0125]

[0126] Embodiment 2

[0127] This embodiment provides a real-time panoramic stitching system for a photovoltaic power station based on drone video streams, including:

[0128] The video acquisition module is used to acquire the photovoltaic module video taken by the drone.

[0129] The detection module is used to input the photovoltaic component video into a trained target detection model to identify the component target frame of each photovoltaic component in the photovoltaic component video.

[0130] The tracking module is used to determine the component identification of each photovoltaic component based on the photovoltaic component video and the component target frame of the photovoltaic component based on a tracking algorithm.

[0131] The sorting and positioning module is used to determine the component sorting and positioning of each photovoltaic component according to the component information of the photovoltaic component and the photovoltaic component video based on the component positioning method; the component information includes a component target frame and a component identification.

[0132] The imaging module is used to determine, for each photovoltaic component, a component image of the photovoltaic component according to the component information of the photovoltaic component and the photovoltaic component video based on an image generation method.

[0133] The splicing module is used to arrange each photovoltaic component in rows and columns based on the component images and component sorting and positioning of each photovoltaic component, so as to complete the panoramic splicing of the photovoltaic power station.

[0134] The system also includes:

[0135] The video processing module is used to perform video processing on the photovoltaic module video to obtain a processed photovoltaic module video; the video processing includes segment processing and setting a pixel threshold.

[0136] The system further includes: a training module for training the target detection model, specifically including:

[0137] The image acquisition submodule is used to acquire sample infrared image data of photovoltaic modules in a photovoltaic power station; the sample infrared image data is obtained through historical photovoltaic module videos.

[0138] The labeling submodule is used to label the sample infrared image data with component frames to obtain the labeled sample infrared image data.

[0139] The training submodule is used to train the YOLOv8 target detection algorithm based on the labeled sample infrared image data to obtain a trained target detection model; wherein the YOLOv8 target detection algorithm is an algorithm obtained by using the improved CSPDarket53 as the backbone network, replacing the CSP module in the Backbone with the C2f module, using the PAN+FPN structure in the Neck part to construct a feature pyramid, and using the decoupling head structure as the detection part on the basis of the YOLOv5 target detection algorithm.

[0140] In summary, the present invention has the following beneficial effects:

[0141] The real-time panoramic stitching method of a photovoltaic power station based on drone video stream provided by the present invention obtains the real-time video stream of photovoltaic components; uses the target detection model to identify the components of the infrared video of the photovoltaic components and detect the boundary box of the photovoltaic components; transmits the component detection results to the target tracking algorithm to track each component uniquely; and accurately locates the global coordinates of each component through the self-written sorting positioning and image generation strategy. The panoramic stitching of the photovoltaic power station is realized, providing a global perspective for the intelligent operation and maintenance of photovoltaics.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A real-time panoramic stitching method for a photovoltaic power station based on drone video stream, characterized in that: include: Get the video of PV panels taken by drone; Inputting the photovoltaic module video into a trained target detection model to identify the module target frame of each photovoltaic module in the photovoltaic module video; The training process of the target detection model is: Acquire sample infrared image data of photovoltaic modules in a photovoltaic power station; the sample infrared image data is obtained through historical photovoltaic module videos; Performing component box annotation on the sample infrared image data to obtain annotated sample infrared image data; Based on the labeled sample infrared image data, the YOLOv8 target detection algorithm is trained to obtain a trained target detection model; wherein the YOLOv8 target detection algorithm is an algorithm obtained by using the improved CSPDarket53 as the backbone network, replacing the CSP module in the Backbone with the C2f module, using the PAN+FPN structure in the Neck part to construct a feature pyramid, and using the decoupling head structure as the detection part on the basis of the YOLOv5 target detection algorithm; Determining a component identification of each photovoltaic component based on the photovoltaic component video and the component target frame of the photovoltaic component and based on a tracking algorithm; For each of the photovoltaic components, according to the component information of the photovoltaic component and the photovoltaic component video, based on the component positioning method, the component sorting and positioning of the photovoltaic component is determined; the component information includes a component target frame and a component identification; Wherein, for each of the photovoltaic components, according to the component information of the photovoltaic component and the photovoltaic component video, based on the component positioning method, the component sorting and positioning of the photovoltaic component is determined, specifically including: According to the component information of the photovoltaic components and the photovoltaic component video, the photovoltaic components that appear repeatedly in different frames in the photovoltaic component video are filtered to obtain filtered photovoltaic components; According to the component target frame of the filtered photovoltaic components, the photovoltaic components in the same frame of the photovoltaic component video are sorted in rows and columns to obtain sorted component position data; According to the sorted component position data and the component identification of the filtered photovoltaic component, the component identification of the photovoltaic component in each frame in the photovoltaic component video is compared in time sequence, and the position information of the component is updated, and when the comparison is to the last frame in the photovoltaic component video, the component sorting and positioning of the photovoltaic component is obtained; For each of the photovoltaic components, determining a component image of the photovoltaic component based on the component information of the photovoltaic component and the photovoltaic component video and based on an image generation method; Based on the component images and component sorting and positioning of each photovoltaic component, each photovoltaic component is arranged in rows and columns to complete the panoramic stitching of the photovoltaic power station.

2. A real-time panoramic stitching method for a photovoltaic power station based on drone video stream according to claim 1, characterized in that: Before inputting the photovoltaic module video into the trained target detection model, the method further includes: The photovoltaic module video is processed to obtain a processed photovoltaic module video; the video processing includes segmentation processing and setting a pixel threshold.

3. The method for real-time panoramic stitching of a photovoltaic power station based on drone video stream according to claim 1 is characterized in that: For each of the photovoltaic components, determining a component image of the photovoltaic component based on the component information of the photovoltaic component and the photovoltaic component video and based on an image generation method specifically includes: According to the component information of the photovoltaic component and the photovoltaic component video, image processing is performed on the photovoltaic component video based on a distortion image correction method, a component image extraction method, an image fusion method, a defect filtering method and a reflection filtering method to obtain a component image of the photovoltaic component.

4. The method for real-time panoramic stitching of a photovoltaic power station based on drone video stream according to claim 1 is characterized in that: The tracking algorithm adopts the SORT target tracking algorithm.

5. The method for real-time panoramic stitching of a photovoltaic power station based on drone video stream according to claim 1, characterized in that: The photovoltaic module video is a real-time infrared video stream.

6. A real-time panoramic stitching system for photovoltaic power stations based on drone video streams, characterized in that: include: Video acquisition module, used to acquire the video of photovoltaic modules taken by drone; A detection module, used to input the photovoltaic module video into a trained target detection model to identify a component target frame of each photovoltaic module in the photovoltaic module video; The training module is used to train the target detection model, specifically including: An image acquisition submodule is used to acquire sample infrared image data of photovoltaic modules in a photovoltaic power station; the sample infrared image data is obtained through historical photovoltaic module videos; A labeling submodule, used for labeling the sample infrared image data with component frames to obtain labeled sample infrared image data; A training submodule is used to train the YOLOv8 target detection algorithm based on the labeled sample infrared image data to obtain a trained target detection model; wherein the YOLOv8 target detection algorithm is an algorithm obtained by using the improved CSPDarket53 as the backbone network, replacing the CSP module in the Backbone with the C2f module, using the PAN+FPN structure in the Neck part to construct a feature pyramid, and using the decoupling head structure as the detection part on the basis of the YOLOv5 target detection algorithm; A tracking module, used for determining the component identification of each photovoltaic component based on the photovoltaic component video and the component target frame of the photovoltaic component and based on a tracking algorithm; A sorting and positioning module is used to determine the component sorting and positioning of each photovoltaic component according to the component information of the photovoltaic component and the photovoltaic component video based on the component positioning method; the component information includes a component target frame and a component identification; Wherein, for each of the photovoltaic components, according to the component information of the photovoltaic component and the photovoltaic component video, based on the component positioning method, the component sorting and positioning of the photovoltaic component is determined, specifically including: According to the component information of the photovoltaic components and the photovoltaic component video, the photovoltaic components that appear repeatedly in different frames in the photovoltaic component video are filtered to obtain filtered photovoltaic components; According to the component target frame of the filtered photovoltaic components, the photovoltaic components in the same frame of the photovoltaic component video are sorted in rows and columns to obtain sorted component position data; According to the sorted component position data and the component identification of the filtered photovoltaic component, the component identification of the photovoltaic component in each frame in the photovoltaic component video is compared in time sequence, and the position information of the component is updated, and when the comparison is to the last frame in the photovoltaic component video, the component sorting and positioning of the photovoltaic component is obtained; An imaging module, configured to determine, for each photovoltaic component, a component image of the photovoltaic component based on the component information of the photovoltaic component and the photovoltaic component video and based on an image generation method; The splicing module is used to arrange each photovoltaic component in rows and columns based on the component images and component sorting and positioning of each photovoltaic component, so as to complete the panoramic splicing of the photovoltaic power station.

7. A real-time panoramic stitching system for a photovoltaic power station based on drone video stream according to claim 6, characterized in that: Also includes: The video processing module is used to perform video processing on the photovoltaic module video to obtain a processed photovoltaic module video; the video processing includes segment processing and setting a pixel threshold.

Images