An unmanned aerial vehicle intelligent inspection method and system applied to a photovoltaic industry

Abstract

The application provides an unmanned aerial vehicle intelligent inspection method and system applied to a photovoltaic industry, relates to the technical field of image processing, and comprises the following steps: using an unmanned device to detect a region to be measured from multiple angles, and obtaining initial images; performing three-dimensional modeling based on the initial images, obtaining a three-dimensional photovoltaic model of a photovoltaic target to be measured and a corresponding three-dimensional environment model, performing edge recognition and surface recognition on the three-dimensional photovoltaic model, obtaining multi-dimensional features of the photovoltaic target to be measured, combining operation data monitored by a centralized control center in real time, determining a first type of abnormal photovoltaic target, and determining a second type of abnormal photovoltaic target according to a relative position relationship between the three-dimensional photovoltaic model and the three-dimensional environment model; using a deep learning model to extract corner pixel coordinates and center pixel coordinates of an abnormal photovoltaic target in adjacent image frames before and after three-dimensional modeling images, performing overlap recognition and triangulation measurement, and determining unique geographic position coordinate information of the abnormal photovoltaic target.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an intelligent inspection method and system for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry. Background Technology

[0002] As a crucial component of clean energy, the photovoltaic (PV) industry has experienced rapid development and widespread application globally, with centralized PV power plants becoming a key development focus for provinces, cities, and large power generation groups. However, with the rapid growth of PV installations, manual operations can no longer keep pace with industry demands, and routine inspections and panel cleaning at PV power plants have gradually become industry pain points. Drones, with their advantages of being unaffected by rugged terrain, capable of aerial photography over large areas, and offering high scanning efficiency, have been applied to some extent in PV inspection, achieving good inspection results.

[0003] However, in the actual process of inspecting photovoltaic power station areas by drone, a single drone is usually used to scan row by row to collect image data of all photovoltaic power station strings in the area. The inspection efficiency is low, and relying solely on the detection data of drones to directly judge anomalies results in a large number of false detections and missed detections. This makes it difficult to find and locate anomalies, causing key equipment to operate with faults for a long time, reducing equipment life and utilization efficiency. Summary of the Invention

[0004] In view of this, the present invention proposes an intelligent inspection method and system for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry.

[0005] The technical solution of this invention is implemented as follows: The first aspect of this invention provides an intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry, comprising: Unmanned equipment is used to perform multi-angle detection on the area to be tested to obtain initial images; the area to be tested includes the photovoltaic target to be tested and the surrounding environment of the photovoltaic target to be tested. Based on the initial image, a 3D model is created to obtain a 3D photovoltaic model of the photovoltaic target under test and a corresponding 3D environment model. Edge recognition and surface recognition are performed on the 3D photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with the real-time monitoring data from the central control center, a first type of abnormal photovoltaic target is identified. A second type of abnormal photovoltaic target is identified based on the relative positional relationship between the 3D photovoltaic model and the 3D environment model. The multi-dimensional features include texture, color, shape, and spectral characteristics. The operating data includes power output, current fluctuation, and equipment temperature. The corner pixel coordinates and center pixel coordinates of the first type and the second type of anomalous photovoltaic targets in the adjacent image frames of the three-dimensional modeling image are extracted using a deep learning model. Based on the corner pixel coordinates and center pixel coordinates, overlap recognition and triangulation measurement are performed to determine the unique geographical location coordinate information of the first type and the second type of anomalous photovoltaic targets. Based on the unique geographic location coordinate information, the first type of abnormal photovoltaic targets and the second type of abnormal photovoltaic targets are excluded from the anomaly list.

[0006] Based on the above technical solutions, preferably, the step of using unmanned equipment to perform multi-angle detection of the area to be measured and obtain initial images includes: The multispectral image acquisition module on the unmanned equipment is used to detect the area under test and obtain the reflection images of the photovoltaic target under test in different bands. Adjusting the shooting position of the unmanned device and the shooting angle of the multispectral image acquisition module yields reflected images at different shooting distances and angles, forming an initial image.

[0007] Based on the above technical solutions, preferably, the step of performing three-dimensional modeling based on the initial image to obtain a three-dimensional photovoltaic model and a corresponding three-dimensional environment model of the photovoltaic target to be tested includes: Multiple image frames are extracted from the initial image in chronological order, and the image frames are segmented to obtain multiple image grid units; each image grid unit corresponds to a potential patch; For each potential patch, feature recognition and feature matching are performed. Combined with projection relationship constraints, the geometric correspondence between different potential patches is determined to obtain multiple potential patch pairs. The potential patch pairs are filtered based on their visibility in adjacent images to obtain target patch pairs. The three-dimensional point coordinates of the target patch pairs are calculated using the camera parameters of the unmanned device to generate a corresponding dense point cloud. Based on the dense point cloud, a three-dimensional reconstruction is performed to generate a three-dimensional photovoltaic model of the photovoltaic target to be tested.

[0008] Based on the above technical solutions, preferably, the step of performing edge recognition and surface recognition on the three-dimensional photovoltaic model to obtain multi-dimensional features of the photovoltaic target to be tested, combined with the real-time monitoring data of the central control center, determines the first type of abnormal photovoltaic target, including: Edge detection algorithms are used to identify and extract features from the edges of the three-dimensional photovoltaic model to obtain edge feature vectors. Image segmentation algorithms are used to identify and extract features from the surface of the three-dimensional photovoltaic model to obtain surface feature vectors. The edge feature vector and the surface feature vector are integrated to construct the multi-dimensional feature vector of the three-dimensional photovoltaic model; By analyzing the correlation between the operating data of the photovoltaic target under test and the multi-dimensional feature vector, and combining it with a preset anomaly threshold for screening, the first type of abnormal photovoltaic targets is determined.

[0009] Based on the above technical solutions, preferably, the three-dimensional environment model includes terrain, buildings, and vegetation; the step of determining the second type of abnormal photovoltaic target based on the relative positional relationship between the three-dimensional photovoltaic model and the three-dimensional environment model includes: Based on the relative positional relationship between the three-dimensional photovoltaic model and the three-dimensional environment model, photovoltaic targets that are blocked by buildings and affected by vegetation and terrain are determined; The photovoltaic target was identified as a second type of anomalous photovoltaic target.

[0010] Based on the above technical solution, preferably, the deep learning model is a Faster R-CNN model; before using the deep learning model to extract the corner pixel coordinates and center pixel coordinates of the first type of anomalous photovoltaic targets and the second type of anomalous photovoltaic targets in the adjacent image frames of the 3D modeling image, the method further includes: The preprocessed sample images are input into the initial Faster R-CNN model. The smooth L1 loss function is used as the main loss, and the IoU loss function is added for iterative optimization until the smooth L1 loss function and the IoU loss function reach a preset threshold, thus obtaining the trained Faster R-CNN model.

[0011] Based on the above technical solutions, preferably, the step of determining the unique geographic location coordinate information of the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target by performing overlap recognition and triangulation measurement based on corner pixel coordinates and center pixel coordinates includes: The average Euclidean distance and cross-union ratio are calculated based on the corner pixel coordinates and the center pixel coordinates. The first type of abnormal photovoltaic targets and the second type of abnormal photovoltaic targets overlapping in the previous and next frames are screened based on the average Euclidean distance and the cross-union ratio to determine the real abnormal photovoltaic targets. Based on the corner pixel coordinates and center pixel coordinates of the real abnormal photovoltaic target, triangulation measurement is performed in conjunction with the intrinsic and extrinsic parameters of the unmanned equipment to determine the unique geographical location coordinate information of the real abnormal photovoltaic target.

[0012] Furthermore, a second aspect of the present invention provides an intelligent inspection system for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry, comprising: an image acquisition module, an anomaly recognition module, a coordinate determination module, and an anomaly elimination module; wherein, The image acquisition module is configured to use unmanned equipment to perform multi-angle detection on the area to be tested and acquire an initial image; the area to be tested includes the photovoltaic target to be tested and the surrounding environment of the photovoltaic target to be tested; The anomaly identification module is configured to perform 3D modeling based on the initial image to obtain a 3D photovoltaic model of the photovoltaic target under test and a corresponding 3D environment model. It then performs edge and surface recognition on the 3D photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with real-time monitoring data from the central control center, it identifies a first type of abnormal photovoltaic target and a second type of abnormal photovoltaic target based on the relative positional relationship between the 3D photovoltaic model and the 3D environment model. The multi-dimensional features include texture, color, shape, and spectral characteristics; the operating data includes power output, current fluctuations, and equipment temperature. The coordinate determination module is configured to use a deep learning model to extract the corner pixel coordinates and center pixel coordinates of the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target in the adjacent image frames of the three-dimensional modeling image, and perform overlap recognition and triangulation measurement based on the corner pixel coordinates and center pixel coordinates to determine the unique geographical location coordinate information of the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target. The anomaly removal module is configured to remove anomalies from the first type of abnormal photovoltaic targets and the second type of abnormal photovoltaic targets based on the unique geographic location coordinate information.

[0013] More preferably, a third aspect of the present invention provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the intelligent inspection method for unmanned aerial vehicles applied to the photovoltaic industry as described in the first aspect.

[0014] More preferably, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the intelligent inspection method for unmanned aerial vehicles applied to the photovoltaic industry as described in the first aspect.

[0015] The intelligent inspection method and system for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry of the present invention has the following advantages over the prior art: 1. Data is collected from multiple angles by drones, and combined with 3D modeling technology to generate high-precision 3D photovoltaic and environmental models, covering blind spots in traditional inspections. On one hand, multi-feature fusion is performed by combining texture, color, shape, spectral characteristics, and operational data to complete anomaly identification and reduce the false detection rate. On the other hand, a comprehensive judgment is made by analyzing the relative positions of the 3D photovoltaic model and the 3D environmental model to avoid missed detections due to temporary occlusion. Based on this, a deep learning model is used for feature tracking to achieve accurate positioning of the photovoltaic target under test, solving the false detection problem caused by overlapping recognition, and greatly improving the efficiency and accuracy of photovoltaic inspections.

[0016] 2. Utilizing unmanned equipment equipped with a multispectral image acquisition module, reflectance images from different bands and multiple perspectives are acquired as initial data by adjusting the shooting position and angle. Multispectral band detection expands the fault differentiation range and improves fault identification sensitivity. By acquiring images from multiple angles, blind spots are eliminated, and both local and global detection are considered, thereby constructing high-precision 3D photovoltaic models and 3D environmental models, providing accurate data for subsequent anomaly identification.

[0017] 3. By identifying, matching, and constraining the projection relationships of potential facets, geometric correspondences are determined and target facet pairs are screened, eliminating invalid and interfering facets and reducing false judgments caused by spurious features. Based on the target facet pairs, a dense point cloud is generated and 3D reconstruction is completed, constructing a 3D photovoltaic model of the photovoltaic target under test. This overcomes the limitations of traditional 2D image detection, enabling more comprehensive capture of the spatial morphological features of photovoltaic modules. It combines operational data of the photovoltaic target under test for correlation analysis, rather than solely relying on image visual data, significantly reducing the probability of false detection and missed detection, and greatly improving the accuracy of anomaly detection. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating an intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry, provided as an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an unmanned aerial vehicle (UAV) intelligent inspection system applied to the photovoltaic industry, provided by an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] In some embodiments, such as Figure 1 As shown, Figure 1 This invention provides a flowchart illustrating an intelligent drone inspection method for the photovoltaic industry, as provided in an embodiment of the present invention. The intelligent drone inspection method for the photovoltaic industry provided by the present invention includes: S110 uses unmanned equipment to perform multi-angle detection of the area to be tested and obtain initial images; the area to be tested includes the photovoltaic target to be tested and the surrounding environment of the photovoltaic target to be tested.

[0022] Here, unmanned equipment can include drones and robotic dogs. Drones conduct inspections at a low altitude of about 1 meter, covering the top of the photovoltaic target under test and open areas, focusing on checking the external structure and heat dissipation system of the target photovoltaic device. Robotic dogs inspect narrow areas and the bottom of the equipment that drones cannot reach on the ground, paying particular attention to cable connections and grounding conditions. The surrounding environment may also include terrain undulations, obstructions, etc.

[0023] In some embodiments, an unmanned device is used to perform multi-angle detection on the area to be measured to obtain an initial image, including: The multispectral image acquisition module on the unmanned equipment is used to detect the area under test and obtain the reflection images of the photovoltaic target under test in different bands. Adjusting the shooting position of the unmanned equipment and the shooting angle of the multispectral image acquisition module yields reflected images at different shooting distances and angles, forming the initial image.

[0024] For example, using the initial flight altitude as a baseline, the flight altitude of the unmanned equipment is gradually adjusted to achieve data acquisition at different shooting distances (e.g., an initial flight altitude of 1 meter, adjusted sequentially to 2 meters, 5 meters, 10 meters, and 20 meters). After each altitude adjustment, once the flight attitude is stabilized, the multi-band acquisition operation is repeated to acquire images of the photovoltaic target under test across the entire area, ensuring that images of all preset bands completely cover the target at each shooting distance. While keeping the unmanned equipment's flight altitude constant, the shooting angle of the multispectral image acquisition module is adjusted, and the flight position of the unmanned equipment can be fine-tuned to ensure clear imaging of the photovoltaic target from different angles. The shooting angle adjustment range can be from -30° to 30°, with horizontal shooting as 0°, downward as negative, and upward as positive. Each adjustment of the shooting angle completes full-band, full-area image acquisition, ensuring unobstructed and clear images from different angles. For photovoltaic targets in the test area with severe shading, densely packed components, or suspected anomalies, the shooting distance and angle are further adjusted to supplement the acquisition of multi-band images, ensuring sufficient image information for that area.

[0025] S120 performs 3D modeling based on the initial image to obtain a 3D photovoltaic model of the photovoltaic target under test and a corresponding 3D environment model. It then performs edge recognition and surface recognition on the 3D photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with the real-time monitoring data from the central control center, it identifies the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target based on the relative positional relationship between the 3D photovoltaic model and the 3D environment model. The multi-dimensional features include texture, color, shape, and spectral characteristics, while the operating data includes power output, current fluctuations, and equipment temperature.

[0026] In some embodiments, three-dimensional modeling is performed based on the initial image to obtain a three-dimensional photovoltaic model of the photovoltaic target to be tested and a corresponding three-dimensional environment model, including: Multiple image frames are extracted from the initial image in a temporal sequence, and the image frames are segmented to obtain multiple image grid units; each image grid unit corresponds to a potential patch. For each potential patch, feature recognition and feature matching are performed. Combined with projection relationship constraints, the geometric correspondence between different potential patches is determined to obtain multiple potential patch pairs. Potential facet pairs are filtered based on their visibility in adjacent images to obtain target facet pairs. The 3D point coordinates of the target facet pairs are then calculated using the camera parameters of the unmanned equipment to generate corresponding dense point clouds. Based on the dense point clouds, 3D reconstruction is performed to generate a 3D photovoltaic model of the photovoltaic target to be tested.

[0027] In this embodiment, multiple image frames are extracted from the initial image in temporal order to ensure uniform time intervals between adjacent frames. For each extracted temporal image frame, a semantic segmentation algorithm, such as U-Net or Mask R-CNN, is used to segment the image, dividing each frame into multiple regular image grid units. During segmentation, the rectangular shape features and multi-band spectral features of the photovoltaic target under test are combined to optimize segmentation accuracy and avoid grid units crossing the boundaries of photovoltaic modules. Each image grid unit corresponds to a potential patch, and each potential patch represents a local region in the image. The pixel range, frame number, and band information of each potential patch are recorded. Multi-dimensional feature extraction is performed on each potential patch, including texture, color, shape, and spectral characteristics, forming a feature vector for each potential patch. Using adjacent time frames as matching units, for each potential patch in the previous frame, similar potential patches are searched in the adjacent next frame for preliminary feature matching, resulting in potential patch pairs. A projection geometric constraint model is established using the camera intrinsic and extrinsic parameters of the unmanned aerial vehicle (UAV). The preliminarily matched potential patch pairs are verified and filtered. The 3D projection coordinates corresponding to the pixel coordinates of the potential patch in the previous frame are calculated and compared with the projection coordinates of the corresponding potential patch in the next frame to verify the geometric correspondence. False patch pairs with excessive projection deviation or mismatched geometric shapes are eliminated, retaining potential patch pairs that conform to the projection constraints to ensure that patch pairs correspond to the same local region. Using the camera parameters of the UAV, triangulation measurements are performed on each target patch pair to solve for the 3D world coordinates of multiple pixels within the patch. The 3D coordinates of all target patch pairs are summarized, detail points are added, and a dense point cloud covering all photovoltaic targets under test is generated, with a focus on improving the point cloud density in the photovoltaic module area to ensure clear module details. The generated dense point cloud is optimized, and a surface reconstruction algorithm is used to convert the optimized dense point cloud into a three-dimensional mesh model.

[0028] In some embodiments, edge recognition and surface recognition are performed on the three-dimensional photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with the real-time monitoring data from the central control center, a first type of abnormal photovoltaic target is identified, including: Edge detection algorithms are used to identify and extract features from the edges of the 3D photovoltaic model to obtain edge feature vectors. Image segmentation algorithms are used to identify and extract features from the surface of the 3D photovoltaic model to obtain surface feature vectors. By integrating edge feature vectors and surface feature vectors, a multi-dimensional feature vector for the three-dimensional photovoltaic model is constructed. By analyzing the correlation between the operating data of the photovoltaic target under test and the multi-dimensional feature vector, and combining it with the preset anomaly threshold for screening, the first type of abnormal photovoltaic targets are identified.

[0029] In this embodiment, Canny edge detection can be used to comprehensively identify the edges of the 3D photovoltaic model, capturing the outline edges and local edges of the component, such as broken gaps and splicing seams. Feature extraction is performed on the identified edges, extracting core feature parameters including edge length, edge curvature, edge direction, edge continuity, and the number and size of edge gaps. These parameters are quantized to form fixed-dimensional edge feature vectors, with one edge feature vector corresponding to each photovoltaic target. A 3D image segmentation algorithm, such as a region growing algorithm, is used to divide the surface of the 3D photovoltaic model into regions, defining the effective area of ​​the component. Feature extraction is then performed on each surface region, extracting core surface features including texture features, color features, spectral features, and flatness features. These feature parameters are quantized to form fixed-dimensional surface feature vectors. The edge feature vectors and surface feature vectors of the same photovoltaic target are normalized and weighted to obtain corresponding multi-dimensional feature vectors. The correlation between the multi-dimensional feature vectors and the running data is analyzed to establish a correlation model. Two types of abnormal thresholds are preset: one is the multi-dimensional feature vector abnormal threshold, and the other is the operational data abnormal threshold. Based on the correlation analysis results, photovoltaic targets with abnormal multi-dimensional feature vectors and corresponding abnormal operational data are selected and identified as the first type of abnormal photovoltaic targets.

[0030] In some embodiments, the three-dimensional environment model includes terrain, buildings, and vegetation; determining the second type of anomalous photovoltaic targets based on the relative positional relationship between the three-dimensional photovoltaic model and the three-dimensional environment model includes: Based on the relative positional relationship between the 3D photovoltaic model and the 3D environmental model, photovoltaic targets that are blocked by buildings and affected by vegetation and terrain are determined. The photovoltaic target was identified as a second type of abnormal photovoltaic target.

[0031] If the horizontal distance between a building and the photovoltaic (PV) target is too close or the vertical height is too high, it will create shadows during periods of sunshine, obstructing the PV modules and reducing their power generation. Improper relative positioning of vegetation to the PV target will also cause shading, and vegetation growth may come into contact with the PV modules, causing damage or exacerbating the shading. Excessive terrain slope or undulation can also lead to abnormal installation angles for PV modules, preventing them from maximizing sunlight reception, or causing instability due to terrain tilt, indirectly affecting operational stability. By calculating the relative positional parameters between the 3D PV model and the 3D environmental model, such as distance, angle, and shading area, and combining this with preset environmental impact thresholds, it is possible to determine whether the PV target is affected by environmental factors, whether the degree of impact reaches an abnormal standard, and thus identify the second type of abnormal PV target.

[0032] S130: Using a deep learning model, the corner pixel coordinates and center pixel coordinates of the first type of anomalous photovoltaic targets and the second type of anomalous photovoltaic targets in the adjacent image frames of the three-dimensional modeling image are extracted. Based on the corner pixel coordinates and center pixel coordinates, overlap recognition and triangulation measurement are performed to determine the unique geographical location coordinate information of the first type of anomalous photovoltaic targets and the second type of anomalous photovoltaic targets.

[0033] Deep learning models, through the hierarchical structure of ResNet or HRNet (High-Resolution Network), can automatically learn multi-level features from local edges to global semantics. For example, in photovoltaic panel detection, low-level convolutional kernels capture the texture details of the panel edges, while high-level convolutional kernels identify the overall shape and layout of the panel. This multi-scale feature fusion capability significantly improves the robustness of detection in complex scenes. For adjacent frames, ConvLSTM or 3D convolutional networks can be used to jointly model spatiotemporal information. ConvLSTM replaces the fully connected layers of traditional LSTM with convolutional operations, enhancing the transmission of spatial features while retaining the ability to remember temporal series data.

[0034] The target's corner and center coordinates in a single frame are predicted using bounding box regression, and triangulation of coordinates in adjacent frames is performed using multi-view geometry principles. Specifically, by matching the correspondence of feature points from different viewpoints and combining the camera's intrinsic and extrinsic parameter matrices, the unique coordinates of the target in 3D space are calculated.

[0035] In some embodiments, the deep learning model is a Faster R-CNN model; before using the deep learning model to extract the corner pixel coordinates and center pixel coordinates of the first type of anomalous photovoltaic targets and the second type of anomalous photovoltaic targets in the adjacent image frames before and after the 3D modeling image, the method further includes: The preprocessed sample images are input into the initial Faster R-CNN model. The smooth L1 loss function is used as the main loss, and the IoU loss function is added for iterative optimization until the smooth L1 loss function and the IoU loss function reach a preset threshold, thus obtaining the trained Faster R-CNN model.

[0036] The Faster R-CNN model is a two-stage object detection model based on region proposal. Its core consists of four parts: the feature extraction network Backbone, the region proposal network RPN, the region of interest pooling (RoI Pooling), and the classification and regression head.

[0037] Let the input image be H represents the height, W represents the width, and 3 represents the RGB channels. The backbone contains L convolutional layers and pooling layers. The weights of the l-th convolutional layer are... , The kernel size is the convolution kernel size. The number of output channels, with an offset of... . No. Convolutional layers of the first layer Layer output feature map Perform convolution operations to output feature maps. :

[0038] ; in, These are the pixel coordinates on the feature map. For channel indexing, This is the activation function.

[0039] After multiple convolutions and pooling, the final output is a shared feature map F: .

[0040] Let the convolutional layer weights of the Region Proposal Network (RPN) be... , bias is , The value is 512 or 256. A 3×3 convolution is performed on the shared feature map F to generate an intermediate feature map:

[0041] ; Using regression branch pairs Perform a 1×1 convolution to output the bounding box regression offset for each anchor box: Determine the candidate region. Let the regression branch weights be... , bias is The regression output is:

[0042] ; in, ,correspond , This is the output for regression.

[0043] By pooling regions of interest, candidate regions of different scales are uniformly mapped to feature vectors of fixed size. A classification head is used to classify the target category using the fixed-size features, and a regression head is used to perform fine regression on the bounding boxes. Finally, the target category and accurate bounding boxes are output.

[0044] In some embodiments, overlap identification and triangulation measurements are performed based on corner pixel coordinates and center pixel coordinates to determine the unique geographic location coordinate information of the first type of anomalous photovoltaic targets and the second type of anomalous photovoltaic targets, including: The average Euclidean distance and cross-union ratio are calculated based on the corner pixel coordinates and the center pixel coordinates. Based on the average Euclidean distance and cross-union ratio, the first and second types of abnormal photovoltaic targets overlapping in the previous and next frames are screened to determine the real abnormal photovoltaic targets. Based on the corner and center pixel coordinates of the real abnormal photovoltaic target, triangulation measurement is performed using the intrinsic and extrinsic parameters of the unmanned equipment to determine the unique geographical location coordinate information of the real abnormal photovoltaic target.

[0045] In this embodiment, for the same anomalous target initially matched in adjacent frames, the Euclidean distance and Intersection over Union (IoU) of its corresponding feature points are calculated. The Euclidean distance characterizes the spatial deviation of feature points between adjacent frames; the smaller the deviation, the higher the probability that the targets in the adjacent frames are the same target. The closer the IoU value is to 1, the higher the overlap of the anomalous target regions in the adjacent frames, and the more likely they are the same target. Here, targets that simultaneously meet the preset Euclidean distance threshold and the preset IoU threshold are determined to be genuine anomalous photovoltaic targets. Using the camera intrinsic parameters of the unmanned equipment and the camera extrinsic parameters of adjacent frames, combined with the corner and center pixel coordinates of the genuine anomalous photovoltaic target, triangulation measurements are performed to determine the unique geographical location coordinates of the genuine anomalous photovoltaic target.

[0046] For Euclidean distance, for example, With the center point, If the point is a corner, then: ; ; The arithmetic mean of the Euclidean distances between the five feature points is used to obtain the mean Euclidean distance. : ; in, The distance in the x-direction between the corner point with index 2 and the center point is represented. The distance in the y-direction between the corner point with index 2 and the corner point with index 1 is represented by the same parameter and subscript.

[0047] S140, based on unique geographic location coordinate information, anomalies are eliminated for the first and second types of abnormal photovoltaic targets.

[0048] Here, the unique geographical coordinates and anomaly type information of the real abnormal photovoltaic target are synchronized to the terminal or mobile device. Combined with the on-site electronic map, the geographical coordinates are accurately marked on the map, which facilitates accurate manual processing.

[0049] In some embodiments, please refer to Figure 2 , Figure 2This is a schematic diagram of the structure of an intelligent drone inspection system applied to the photovoltaic industry, provided by an embodiment of the present invention. The present invention provides an intelligent drone inspection system 200 applied to the photovoltaic industry, comprising: an image acquisition module 210, an anomaly recognition module 220, a coordinate determination module 230, and an anomaly removal module 240; wherein,

[0050] The image acquisition module 210 is configured to use unmanned equipment to perform multi-angle detection of the area to be tested and acquire initial images; the area to be tested includes the photovoltaic target to be tested and the surrounding environment of the photovoltaic target to be tested; The anomaly identification module 220 is configured to perform 3D modeling based on the initial image to obtain a 3D photovoltaic model of the photovoltaic target under test and a corresponding 3D environment model. It then performs edge and surface recognition on the 3D photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with the real-time monitoring data from the central control center, it identifies the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target based on the relative positional relationship between the 3D photovoltaic model and the 3D environment model. The multi-dimensional features include texture, color, shape, and spectral characteristics, while the operating data includes power output, current fluctuations, and equipment temperature. The coordinate determination module 230 is configured to use a deep learning model to extract the corner pixel coordinates and center pixel coordinates of the first type of anomalous photovoltaic targets and the second type of anomalous photovoltaic targets in the adjacent image frames of the three-dimensional modeling image, and to perform overlap recognition and triangulation measurement based on the corner pixel coordinates and center pixel coordinates to determine the unique geographical location coordinate information of the first type of anomalous photovoltaic targets and the second type of anomalous photovoltaic targets. The anomaly removal module 240 is configured to remove anomalies for the first type of abnormal photovoltaic targets and the second type of abnormal photovoltaic targets based on unique geographic location coordinate information.

[0051] In some embodiments, the image acquisition module 210 is specifically configured as follows: The multispectral image acquisition module on the unmanned equipment is used to detect the area under test and obtain the reflection images of the photovoltaic target under test in different bands. Adjusting the shooting position of the unmanned equipment and the shooting angle of the multispectral image acquisition module yields reflected images at different shooting distances and angles, forming the initial image.

[0052] In some embodiments, the anomaly detection module 220 is specifically configured as follows: Multiple image frames are extracted from the initial image in a temporal sequence, and the image frames are segmented to obtain multiple image grid units; each image grid unit corresponds to a potential patch. For each potential patch, feature recognition and feature matching are performed. Combined with projection relationship constraints, the geometric correspondence between different potential patches is determined to obtain multiple potential patch pairs. Potential facet pairs are filtered based on their visibility in adjacent images to obtain target facet pairs. The 3D point coordinates of the target facet pairs are then calculated using the camera parameters of the unmanned equipment to generate corresponding dense point clouds. Based on the dense point clouds, 3D reconstruction is performed to generate a 3D photovoltaic model of the photovoltaic target to be tested.

[0053] In some embodiments, the anomaly detection module 220 is specifically configured as follows: Edge detection algorithms are used to identify and extract features from the edges of the 3D photovoltaic model to obtain edge feature vectors. Image segmentation algorithms are used to identify and extract features from the surface of the 3D photovoltaic model to obtain surface feature vectors. By integrating edge feature vectors and surface feature vectors, a multi-dimensional feature vector for the three-dimensional photovoltaic model is constructed. By analyzing the correlation between the operating data of the photovoltaic target under test and the multi-dimensional feature vector, and combining it with the preset anomaly threshold for screening, the first type of abnormal photovoltaic targets are identified.

[0054] In some embodiments, the three-dimensional environment model includes terrain, buildings, and vegetation; the anomaly detection module 220 is specifically configured as follows: Based on the relative positional relationship between the 3D photovoltaic model and the 3D environmental model, photovoltaic targets that are blocked by buildings and affected by vegetation and terrain are determined. The photovoltaic target was identified as a second type of abnormal photovoltaic target.

[0055] In some embodiments, the deep learning model is a Faster R-CNN model; the UAV intelligent inspection system 200 applied to the photovoltaic industry also includes a training module; the training module is specifically configured as follows: The preprocessed sample images are input into the initial Faster R-CNN model. The smooth L1 loss function is used as the main loss, and the IoU loss function is added for iterative optimization until the smooth L1 loss function and the IoU loss function reach a preset threshold, thus obtaining the trained Faster R-CNN model.

[0056] In some embodiments, the coordinate determination module 230 is specifically configured as follows: The average Euclidean distance and cross-union ratio are calculated based on the corner pixel coordinates and the center pixel coordinates. Based on the average Euclidean distance and cross-union ratio, the first and second types of abnormal photovoltaic targets overlapping in the previous and next frames are screened to determine the real abnormal photovoltaic targets. Based on the corner and center pixel coordinates of the real abnormal photovoltaic target, triangulation measurement is performed using the intrinsic and extrinsic parameters of the unmanned equipment to determine the unique geographical location coordinate information of the real abnormal photovoltaic target.

[0057] It should be noted that the UAV intelligent inspection system for the photovoltaic industry provided in this application embodiment and the UAV intelligent inspection method for the photovoltaic industry provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned UAV intelligent inspection method for the photovoltaic industry, and the repeated parts will not be described again.

[0058] In some embodiments, please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 300 provided in this application includes a processor 310 and a memory 320; the memory 320 stores a computer program, wherein the computer program, when executed by the processor, implements the aforementioned intelligent inspection method for drones applied to the photovoltaic industry.

[0059] Specifically, processor 310 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 310 may also include onboard memory for caching purposes. Processor 310 may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0060] The memory 320 may be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, the memory 320 may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. Specific examples of the memory 320 include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and may also be random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0061] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this program implements the aforementioned intelligent drone inspection method applied to the photovoltaic industry. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0062] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0063] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A method for intelligent inspection of photovoltaic products using unmanned aerial vehicles (UAVs), characterized in that, include: Unmanned equipment is used to perform multi-angle detection on the area to be tested to obtain initial images; the area to be tested includes the photovoltaic target to be tested and the surrounding environment of the photovoltaic target to be tested. Based on the initial image, a 3D model is created to obtain a 3D photovoltaic model of the photovoltaic target under test and a corresponding 3D environment model. Edge recognition and surface recognition are performed on the 3D photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with the real-time monitoring data from the central control center, a first type of abnormal photovoltaic target is identified. A second type of abnormal photovoltaic target is identified based on the relative positional relationship between the 3D photovoltaic model and the 3D environment model. The multi-dimensional features include texture, color, shape, and spectral characteristics. The operating data includes power output, current fluctuation, and equipment temperature. The corner pixel coordinates and center pixel coordinates of the first type and the second type of anomalous photovoltaic targets in the adjacent image frames of the three-dimensional modeling image are extracted using a deep learning model. Based on the corner pixel coordinates and center pixel coordinates, overlap recognition and triangulation measurement are performed to determine the unique geographical location coordinate information of the first type and the second type of anomalous photovoltaic targets. Based on the unique geographic location coordinate information, the first type of abnormal photovoltaic targets and the second type of abnormal photovoltaic targets are excluded from the anomaly list.

2. The intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry as described in claim 1, characterized in that, The process of using unmanned equipment to perform multi-angle detection of the area to be tested and acquire initial images includes: The multispectral image acquisition module on the unmanned equipment is used to detect the area under test and obtain the reflection images of the photovoltaic target under test in different bands. Adjusting the shooting position of the unmanned device and the shooting angle of the multispectral image acquisition module yields reflected images at different shooting distances and angles, forming an initial image.

3. The intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry as described in claim 2, characterized in that, The step of performing 3D modeling based on the initial image to obtain a 3D photovoltaic model and a corresponding 3D environment model of the photovoltaic target to be tested includes: Multiple image frames are extracted from the initial image in chronological order, and the image frames are segmented to obtain multiple image grid units; each image grid unit corresponds to a potential patch; For each potential patch, feature recognition and feature matching are performed. Combined with projection relationship constraints, the geometric correspondence between different potential patches is determined to obtain multiple potential patch pairs. The potential patch pairs are filtered based on their visibility in adjacent images to obtain target patch pairs. The three-dimensional point coordinates of the target patch pairs are calculated using the camera parameters of the unmanned device to generate a corresponding dense point cloud. Based on the dense point cloud, a three-dimensional reconstruction is performed to generate a three-dimensional photovoltaic model of the photovoltaic target to be tested.

4. The intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry as described in claim 1, characterized in that, The process involves edge and surface recognition of the three-dimensional photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with real-time monitoring data from the central control center, a first category of abnormal photovoltaic targets is identified, including: Edge detection algorithms are used to identify and extract features from the edges of the three-dimensional photovoltaic model to obtain edge feature vectors. Image segmentation algorithms are used to identify and extract features from the surface of the three-dimensional photovoltaic model to obtain surface feature vectors. The edge feature vector and the surface feature vector are integrated to construct the multi-dimensional feature vector of the three-dimensional photovoltaic model; By analyzing the correlation between the operating data of the photovoltaic target under test and the multi-dimensional feature vector, and combining it with a preset anomaly threshold for screening, the first type of abnormal photovoltaic targets is determined.

5. The intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry as described in claim 1, characterized in that, The three-dimensional environment model includes terrain, buildings, and vegetation; determining the second type of anomalous photovoltaic targets based on the relative positional relationship between the three-dimensional photovoltaic model and the three-dimensional environment model includes: Based on the relative positional relationship between the three-dimensional photovoltaic model and the three-dimensional environment model, photovoltaic targets that are blocked by buildings and affected by vegetation and terrain are determined; The photovoltaic target was identified as a second type of anomalous photovoltaic target.

6. The intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry as described in claim 1, characterized in that, The deep learning model is the Faster R-CNN model; before using the deep learning model to extract the corner pixel coordinates and center pixel coordinates of the first type and the second type of anomalous photovoltaic targets in the adjacent image frames of the 3D modeled image, the method further includes: The preprocessed sample images are input into the initial Faster R-CNN model. The smooth L1 loss function is used as the main loss, and the IoU loss function is added for iterative optimization until the smooth L1 loss function and the IoU loss function reach a preset threshold, thus obtaining the trained Faster R-CNN model.

7. The intelligent inspection method for unmanned aerial vehicles (UAVs) applied to the photovoltaic industry as described in claim 1, characterized in that, The method of determining the unique geographic location coordinates of the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target by performing overlap recognition and triangulation measurement based on corner pixel coordinates and center pixel coordinates includes: The average Euclidean distance and cross-union ratio are calculated based on the corner pixel coordinates and the center pixel coordinates. The first type of abnormal photovoltaic targets and the second type of abnormal photovoltaic targets overlapping in the previous and next frames are screened based on the average Euclidean distance and the cross-union ratio to determine the real abnormal photovoltaic targets. Based on the corner pixel coordinates and center pixel coordinates of the real abnormal photovoltaic target, triangulation measurement is performed in conjunction with the intrinsic and extrinsic parameters of the unmanned equipment to determine the unique geographical location coordinate information of the real abnormal photovoltaic target.

8. A drone-based intelligent inspection system for the photovoltaic industry, characterized in that, include: The module comprises an image acquisition module, an anomaly detection module, a coordinate determination module, and an anomaly elimination module; among which, The image acquisition module is configured to use unmanned equipment to perform multi-angle detection on the area to be tested and acquire an initial image; the area to be tested includes the photovoltaic target to be tested and the surrounding environment of the photovoltaic target to be tested; The anomaly identification module is configured to perform 3D modeling based on the initial image to obtain a 3D photovoltaic model of the photovoltaic target under test and a corresponding 3D environment model. It then performs edge and surface recognition on the 3D photovoltaic model to obtain multi-dimensional features of the photovoltaic target under test. Combined with real-time monitoring data from the central control center, it identifies a first type of abnormal photovoltaic target and a second type of abnormal photovoltaic target based on the relative positional relationship between the 3D photovoltaic model and the 3D environment model. The multi-dimensional features include texture, color, shape, and spectral characteristics; the operating data includes power output, current fluctuations, and equipment temperature. The coordinate determination module is configured to use a deep learning model to extract the corner pixel coordinates and center pixel coordinates of the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target in the adjacent image frames of the three-dimensional modeling image, and perform overlap recognition and triangulation measurement based on the corner pixel coordinates and center pixel coordinates to determine the unique geographical location coordinate information of the first type of abnormal photovoltaic target and the second type of abnormal photovoltaic target. The anomaly removal module is configured to remove anomalies from the first type of abnormal photovoltaic targets and the second type of abnormal photovoltaic targets based on the unique geographic location coordinate information.

9. An electronic device comprising a processor and a memory; said memory storing a computer program, wherein, When the computer program is executed by the processor, it implements the unmanned aerial vehicle (UAV) intelligent inspection method for the photovoltaic industry as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program, wherein when the computer program is executed by a processor, it implements the intelligent inspection method for drones applied to the photovoltaic industry as described in any one of claims 1 to 7.

Images