Unmanned aerial vehicle image automatic splicing method for fault photovoltaic module positioning

Abstract

The invention relates to an unmanned aerial vehicle image automatic splicing method for fault photovoltaic module positioning, and the method comprises the steps: extracting a feature point of a photovoltaic slate point edge in an image through employing a SuperPoint deep learning model, and precisely positioning a significant position which can be used for matching in the image; constructing a key point association graph through a SuperGlue algorithm, calculating a feature similarity matrix in combination with an attention mechanism, and establishing a corresponding relation of feature points among different images; solving an optimal feature matching pair by applying a Sinkhorn algorithm to obtain an optimal feature point corresponding relation between adjacent images, and providing accurate matching data for geometric transformation; according to the method, a SuperPoint self-supervision framework is utilized, through a VGG encoder and a double-branch decoder, the photovoltaic panel regular texture region feature point extraction capability is improved, uniform distribution is realized through non-maximum suppression, and the problems of sparse feature points and non-uniform distribution in a traditional algorithm are solved. And a SuperGlue graph neural network is introduced, and a similarity matrix with a direction weight is constructed in combination with a self-crossover attention mechanism, so that the accuracy of the edge feature points of the photovoltaic panel is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle intelligent inspection and computer vision, in particular to an unmanned aerial vehicle image automatic splicing method for positioning of a fault photovoltaic module. BACKGROUND

[0002] In the unmanned aerial vehicle inspection of a photovoltaic power station, image splicing is a core link for realizing analysis of a module panoramic image. Registration and splicing of adjacent images are the key to panoramic image generation technology, and a splicing method based on feature template matching feature points is a method that is currently used more. The method mainly includes pre-splicing of images, extraction of feature points, image matrix transformation and splicing, and smoothing processing of images.

[0003] The SIFT and ORB algorithms in the traditional method have weak feature point extraction capability in a low-texture area of a photovoltaic panel, resulting in obvious splicing seams. Due to lens distortion of an unmanned aerial vehicle and a curved surface of a photovoltaic panel, the error of a homography matrix estimated by the RANSAC algorithm is large, and the error rate of splicing is large; the coverage range of a single unmanned aerial vehicle operation is limited, and there is a lack of a dynamic task allocation mechanism in cluster operation, and the efficiency of multi-machine cooperation is low. SUMMARY

[0004] The technical problem of the present application is to provide an unmanned aerial vehicle image automatic splicing method for positioning of a fault photovoltaic module, to solve the problems of large matching error and low splicing efficiency of a traditional algorithm in photovoltaic module inspection, and to realize seamless panoramic image splicing and accurate fault positioning.

[0005] The purpose of the present application is to solve the above problems, and the unmanned aerial vehicle image automatic splicing method for positioning of a fault photovoltaic module comprises the following steps: S1: dividing the area of a photovoltaic power station into multiple sub-areas, and dynamically allocating an inspection task according to the real-time power of each unmanned aerial vehicle, the sensor state and environmental factors; S2: presetting an inspection route of each unmanned aerial vehicle, and collecting photovoltaic area images through a high-definition camera and an infrared sensor carried by the unmanned aerial vehicle; S3: using a SuperPoint deep learning model to extract feature points of a convex variable area edge of a photovoltaic panel from the collected photovoltaic area images, and positioning a salient position in the image for matching; S4: based on the feature points and their descriptors extracted in S3, using a SuperGlue algorithm to construct a correlation graph between the feature points, and combining an attention mechanism to calculate a similarity matrix between the feature points, selecting an optimal matching strategy according to the similarity matrix, and establishing a preliminary correspondence relationship between the feature points of different images; S5: Solve the optimal feature matching by Sinkhorn algorithm, solve the double random form of the matching score matrix by row and column normalization iteration, obtain the optimal feature point correspondence between images based on the point-to-plane distance theory, and obtain the matching data between images with the highest confidence; S6: Estimate the homography matrix by MAGSAC++ algorithm, and calculate the mathematical model describing the perspective transformation relationship between the two images; S7: Perform image stitching based on the GraphCut optimization algorithm to eliminate the seam effect and output a high-resolution panoramic image, and transform the panoramic image to the target plane by stitching method until a complete photovoltaic area map is obtained.

[0006] Further, in step S1, the photovoltaic power station area is divided into multiple sub-regions, and the divided sub-regions include convex regions and non-convex regions. The division standard is mainly based on geometric characteristics. Any two points in the convex region are completely contained in the region. There are at least two points in the non-convex region, and the line connecting the two points is not completely contained in the region.

[0007] Preferably, the convex region is divided by a variable step line scanning strategy. The division line position is dynamically adjusted according to the number of unmanned aerial vehicles n. The unmanned aerial vehicle plans a Z-shaped scanning path along the inertia principal axis direction of the sub-region. The scanning interval is dynamically adjusted according to the flight height and camera focal length, and the turning path uses Dubins curve constraint.

[0008] Preferably, the camera focal length is dynamically adjusted by an adaptive step. The nonlinear mapping characteristics of the logarithmic function are used. When the error exceeds the threshold , the step is dynamically enlarged. The adaptive step is used for adaptive part of large error large step convergence and small error small step fine adjustment. The calculation formula of the adaptive step is: ; In the formula, represents the area error, S represents the reference step, K s represents the adjustment coefficient, β represents the threshold coefficient, represents the accuracy threshold.

[0009] Preferably, the non-convex region is divided by introducing a Delaunay subdivision with a boundary rectangle and a Monte Carlo optimized path, and combining a multi-objective function, a resource allocation theory and an unmanned aerial vehicle energy consumption model to construct a multi-objective optimization function to generate a sub-region task allocated to each unmanned aerial vehicle.

[0010] Preferably, the objective function of the nonlinear programming for solving the sub-region merging is: ; In the formula,E d represents the normalized value of the electric quantity, T p represents the path time, C e represents the environment complexity, represents the area balance weight, represents the task allocation matrix generated by the Hungarian algorithm.

[0011] Preferably, the inspection tasks are dynamically allocated according to the real-time electric quantity, sensor state and environmental factors of each unmanned aerial vehicle, and the task allocation matrix is generated by a multi-objective function and the Hungarian algorithm, so as to reduce the area difference rate of the sub-regions allocated to each unmanned aerial vehicle.

[0012] Preferably, the resource allocation theory in operational research and the unmanned aerial vehicle energy consumption model are fused to construct a multi-objective function, and the calculation formula is as follows: ; In the formula, E d represents the normalized value of the electric quantity, T p represents the path time, C e represents the environment complexity, represents the area balance weight, represents the task allocation matrix generated by the Hungarian algorithm.

[0013] Further, the unmanned aerial vehicle flight in step S2 is based on the principal axis of inertia of the rigid body, and the principal axis of inertia of the polygon is in the direction in which the moment of inertia is minimum. When scanning in this direction, the change of the angular momentum of the unmanned aerial vehicle is minimum. The principal axis direction is selected based on the rigid body inertia theory, so as to ensure that the change of the angular momentum of the unmanned aerial vehicle is minimum, reduce the flight energy consumption, and ensure the overlap consistency of the image acquisition timing, and the Dubins path is used to process the turning transition. The calculation formula of the scanning interval of the image acquisition timing is as follows: ; In the formula, represents the scanning interval, R min represents the minimum turning radius of the unmanned aerial vehicle, and ω represents the camera field of view width.

[0014] Further, the minimum turning radius constraint uses the Dubins path to process the turning transition, and the Dubins path uses a right-left-right turning path. The calculation formula of the center coordinates of the circle is as follows: ; In the formula, (x s ,y s) Indicates the starting point. (x f ,y f () indicates the endpoint. θ Indicates the initial heading angle. (x 1 ,y 1) (x 2 ,y 2) and (x f ,y f The first circle turns right, the second circle turns left, and the third circle turns right, respectively.

[0015] Preferably, step S2 also includes defining the drone's altitude and focus using similar triangle relationships, and planning the sampling density of each drone using Shannon's sampling theorem.

[0016] The formula for calculating the relationship between flight altitude and focal length, which satisfies the similarity of triangles, is: ; In the formula, H Indicates flight altitude. f Indicates the focal length of the captured image. Indicates flight speed.

[0017] Furthermore, in step S3, sub-region boundary constraints are introduced into the SuperPoint model, giving higher weights to feature points near the vertices of the sub-regions after Delaunay triangulation, and enhancing the detection probability of photovoltaic panel edge features through an attention mechanism.

[0018] Preferably, the SuperPoint model includes a decoder and a feature point decoder. The feature point decoder contains branch 1 and branch 2. Branch 2 outputs a 64-dimensional descriptor DϵRH×W×64, processed by bicubic interpolation and L2 normalization. Image features are extracted by a VGG encoder. The input image is converted into a feature map through multi-layer convolution and pooling operations. The SE attention module is used to enhance the feature representation, strengthening important channel information and suppressing unimportant channel information. The weighted feature map is decoded by the feature point decoder to generate a heatmap. The final descriptor is generated by combining sub-region processing and the descriptor decoder. The entire process covers multiple steps, including feature extraction, attention mechanism, decoding, and descriptor generation, and is used for tasks such as image feature extraction and matching.

[0019] Furthermore, the formula for calculating the optimal allocation matrix using the Sinkhorn algorithm in step S5 is as follows: ; ; In the formula, C is a feature point matching distance, a and b are distribution probability vectors of feature points in two images, and P T 1=b, the Sinkhorn algorithm ensures the matching coverage of the low-texture area of the photovoltaic panel, avoids the missing matching caused by the sparsity of features in the traditional algorithm, and at the same time, the matching probability weight b of the key component area is constrained j , and the matching accuracy of the fault area is improved.

[0020] Further, step S6 includes calculating a homography matrix hypothesis from a randomly sampled minimum point set in the optimal matching pair, using an adaptive threshold strategy based on maximum likelihood estimation to evaluate the inliers and outliers of all matching points under the hypothesis, refining the homography matrix parameters corresponding to the inliers by weighted least squares, and iteratively executing the above process until the homography matrix with the largest inlier set support and the smallest fitting error is selected. The expression of the MAGSAC++ algorithm for estimating the homography matrix is: ; In the formula, p is an adaptive loss function based on maximum likelihood, and further, in step S7, the image pair to be spliced is subjected to geometric transformation and resampling, and the optimal seam path is determined by minimizing the energy function of graph cuts, and then the panoramic image is transformed onto the target plane for splicing by the splicing method. Steps S3 to S7 are repeatedly executed until all sub-area images are processed, and a complete photovoltaic area image is obtained.

[0021] Preferably, the calculation formula for constructing the energy function is: ; In the formula, D p (L p ) represents an image pixel, and p represents an assigned label. Preferably, the panoramic image is transformed onto the target plane for splicing by the splicing method, which includes measuring the geometric error of the projection of the graphic pixel to the three-dimensional photovoltaic panel plane based on the point-to-plane distance theory, and the calculation formula of the smoothing term is: ; In the formula, V p,q (L p ,L q ) is the energy value of the smoothing term between pixel p and adjacent pixel q, and lambda is the smoothing term weight coefficient. The greater the value, the smoother the seam path.

[0022] The calculation formula of the double-weight fusion strategy is: ; In the formula, I fused(x,y) is the fused pixel value, I1 and I2 are the pixel values of the two images to be spliced in the overlapping area, and w1, w2 are adaptive weights.

[0023] Compared with the prior art, the beneficial effects of the present application include: 1) The unmanned aerial vehicle image automatic stitching method for fault photovoltaic module positioning proposed by the present application divides the area of the photovoltaic power station into convex regions and non-convex regions, sub-divides the convex regions, adopts a variable step line scanning method, and introduces an adaptive step size, so that the area variance error of the sub-regions is reduced by real-time calculation of the area error and dynamic adjustment of the step size, and the task allocation uniformity is improved; for the non-convex regions, a boundary rectangle wrapping combined with Delaunay triangulation is introduced, and a multi-objective function is solved by the Monte Carlo method, so that the sub-region width is reduced, the turning path of the unmanned aerial vehicle is reduced, and the problems of uneven region division and low efficiency of task allocation are solved.

[0024] 2) The present application uses the SuperPoint self-supervised framework, increases the feature point extraction ability of the regular texture area of the photovoltaic panel through the VGG encoder and the double-branch decoder, and realizes uniform distribution through non-maximum suppression, solving the problems of sparse feature points and uneven distribution of traditional algorithms.

[0025] 3) The present application introduces a SuperGlue graph neural network, constructs a similarity matrix with direction weight combined with a self-cross attention mechanism, and improves the accuracy of the edge feature points of the photovoltaic panel.

[0026] 4) The present application improves the robustness of extracting photovoltaic panels in complex texture scenes by solving the optimal transport through the Sinkhorn algorithm.

[0027] 5) The present application adopts the MAGSAC++ algorithm, optimizes the geometric consistency score and adjacency weight, enhances the robustness to nonlinear distortion, constructs an energy function containing a time sequence weight based on GraphCut, and combines a double-weight gradual in-out fusion strategy to improve the stitching quality of the panoramic image, reduce the distortion rate, and eliminate the seam effect. BRIEF DESCRIPTION OF DRAWINGS

[0028] The present application will be further described below in conjunction with the drawings and examples.

[0029] Figure 1 The flowchart of the unmanned aerial vehicle image automatic stitching method for fault photovoltaic module positioning of the embodiment of the present application Figure 2 The SuperPoint model structure diagram of the embodiment of the present application Figure 3 The feature point extraction and feature matching flowchart of the embodiment of the present application Figure 4 The photovoltaic module image stitching diagram collected by the unmanned aerial vehicle of the embodiment of the present application Figure 5An effect diagram of automatic image splicing of the unmanned aerial vehicle of the embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the following description of embodiments is merely illustrative in nature, and the present application is not intended to limit its application or its use, and the present application is not limited to the following embodiments.

[0031] As shown in Figure 1 , the unmanned aerial vehicle image automatic splicing method for fault photovoltaic module positioning comprises the following steps: S1: Divide the area of the photovoltaic power station into multiple sub-areas, and dynamically allocate the inspection tasks according to the real-time power of each unmanned aerial vehicle, the sensor state and the environmental factors.

[0032] In step S1, the area of the photovoltaic power station is divided into multiple sub-areas, and the divided sub-areas include convex regions and non-convex regions. The division standard is mainly based on geometric characteristics. Any two points in the convex region are completely contained in the region. There are at least two points in the non-convex region, and the line connecting the two points is not completely contained in the region.

[0033] The unmanned aerial vehicle adopts cluster control, carries high-definition camera, infrared sensor and RTK positioning; the surface control base station is responsible for scheduling and path planning, the scheduling unit is responsible for dynamic allocation of sub-area inspection tasks, and the path planning unit generates the optimal flight path under obstacle avoidance constraints.

[0034] The convex region adopts a variable step line scanning strategy for dissection, and the dissection line position is dynamically adjusted according to the number of unmanned aerial vehicles n. The unmanned aerial vehicle plans a Z-shaped scanning path along the inertia principal axis direction of the sub-area. The scanning interval is dynamically adjusted according to the flight height and camera focal length, the adjacent image overlap rate is constrained to be not less than 70%, and the turning path adopts Dubins curve constraint.

[0035] The adaptive step of the camera focal length dynamic adjustment utilizes the nonlinear mapping characteristics of the logarithmic function. When the error exceeds the threshold , the step is dynamically enlarged for adaptive part of large error large step convergence and small error small step fine adjustment. The calculation formula of the adaptive step is: ; In the formula, represents the area error, S represents the reference step, S =0.1km; K s represents the adjustment coefficient, K s =0.5; β represents the threshold coefficient, β =0.8; denotes the precision threshold, .

[0036] The non-convex region introduces a Delaunay subdivision with a boundary rectangle and divides the path with a Monte Carlo optimization, and combines a multi-objective function, a resource allocation theory, and a UAV energy consumption model to construct a multi-objective optimization function to generate a sub-region task allocated to each UAV.

[0037] Preferably, the objective function of the nonlinear programming for solving the sub-region merging is: ; In the formula, E d denotes the power normalization value, T p denotes the path time, α 1 and α 2 respectively denote the weight coefficient, balancing the sub-region width and the area balance, α 1=50、 α 2=1; C e denotes the environmental complexity, denotes the area balance weight, denotes the task allocation matrix generated by the Hungarian algorithm.

[0038] According to the real-time power, sensor state and environmental factors of each UAV, the dynamic allocation of the inspection task is performed, including generating a task allocation matrix by a multi-objective function and a Hungarian algorithm, which is used to reduce the area difference rate of the sub-region allocated to each UAV.

[0039] The dynamic allocation of the task includes: 1) When the wind speed is detected to be less than 5 m / s or the UAV is less than 20%, the UAV re-plans the flight path in real time.

[0040] 2) An optimal flight trajectory is generated by using real-time navigation combined with real-time obstacle avoidance sensor data.

[0041] The resource allocation theory in operational research and the UAV energy consumption model are fused to construct a multi-objective function, and the calculation formula is: ; In the formula, E d denotes the power normalization value, T p denotes the path time, C e denotes the environmental complexity, denotes the area balance weight, , denotes the task allocation matrix generated by the Hungarian algorithm, , ensure that the area of each sub-region allocated by the UAV is less than 5% different.

[0042] S2: preset the inspection route of each UAV, and collect photovoltaic area images through the high-definition camera and infrared sensor carried by the UAV.

[0043] In step S2, the UAV flight is based on the principal axis of inertia of the rigid body, and the principal axis of inertia of the polygon is in the direction in which the moment of inertia is minimized. When scanning in this direction, the angular momentum change of the UAV turning is minimized. The principal axis direction is selected based on the rigid body inertia theory, which ensures that the angular momentum change of the UAV turning is minimized, reduces the flight energy consumption, and ensures the overlap consistency of the image acquisition timing. The Dubins path is used to process the turning transition; the calculation formula of the scanning interval of the image acquisition timing is: ; In the formula, , represents the scanning interval, R min , represents the minimum turning radius of the UAV, and ω represents the field of view width of the camera.

[0044] The minimum turning radius constraint uses the Dubins path to process the turning transition. The Dubins path uses a right-left-right turning path. The calculation formula of the center coordinates is: ; In the formula, (x s ,y s ) , represents the starting point, (x f ,y f , represents the end point, θ , represents the initial heading angle. When scanning in the principal axis direction, the total length of the path is reduced by 26% compared to the non-principal axis direction, and the turning path accounts for 36% less, which ensures the overlap consistency of the image acquisition timing, (x 1 ,y 1)、 (x 2 ,y 2) and (x f ,y f , respectively represent the center of the first arc right turn, the second arc left turn, and the third arc right turn.

[0045] In step S2, the UAV height and focus are defined by similar triangle relationship, and the sampling theorem of Shannon is used to plan the acquisition density of each UAV.

[0046] The calculation formula for the flight height and focal length to satisfy the similar triangle relationship is: ; wherein, H represents the flight height, f represents the focal length of the photographed image, represents the flight speed.

[0047] S3: using a SuperPoint deep learning model to extract feature points of the convex region edge of the photovoltaic panel from the collected photovoltaic region image, and locating the salient positions in the image for matching.

[0048] In step S3, a sub-region boundary constraint is introduced in the SuperPoint model, higher weights are given to the feature points near the vertices of the sub-region after Delaunay triangulation, and the detection probability of the edge features of the photovoltaic panel is enhanced through the attention mechanism.

[0049] In this embodiment, SuperPoint adopts a VGG-style encoder to reduce the image size through convolution layers, max-pooling layers and nonlinear activation layers. Through 3 max-pooling layers, the input image size is reduced to 1 / 8 of the original size, and the image is reduced from I ∈ R H×W converted into a tensor B ∈ R HG×WG×F wherein, F is the number of feature channels. Local features such as corner points, edge inflection points, etc. of the photovoltaic panel rock point edge are extracted.

[0050] As shown in Figure 2 , the SuperPoint model includes a decoder and a feature point decoder, the feature point decoder includes branch 1 and branch 2, and branch 2 outputs a 64-dimensional description sub D∈RH×W×64, which is processed through bicubic interpolation and L2 normalization. Through the VGG encoder, the image features are extracted, the input image is converted into a feature map through multi-layer convolution and pooling operations, the SE attention module is used to enhance the feature representation, important channel information in the feature map is enhanced, and unimportant channel information is suppressed, the weighted feature map is decoded through the feature point decoder to generate a heat map, and the final description sub is generated by combining the sub-region processing and the description sub decoder. The whole process covers multiple steps such as feature extraction, attention mechanism, decoding and description sub generation, which is used for image feature extraction and matching tasks.

[0051] The feature point decoding end outputs feature point probabilities for each pixel through the decoder, the input tensor dimension is RHG×WG×65, and the output dimension is RH×W, wherein each pixel corresponds to the feature point probability of the 8×8 local region of the original image. Through the channel dimension softmax operation, the non-feature point class is deleted and the tensor shape is transformed into RN×N, so as to determine the potential feature point position of the photovoltaic panel rock point edge.

[0052] S4: Based on the feature points and their descriptors extracted in S3, a SuperGlue algorithm is used to construct a correlation graph between the feature points, and an attention mechanism is used to calculate a similarity matrix between the feature points. An optimal matching strategy is selected based on the similarity matrix, and a preliminary correspondence relationship between the feature points of different images is established.

[0053] S5: An optimal feature matching is solved by a Sinkhorn algorithm, a double random form of the matching score matrix is solved by row and column normalization iteration, and an optimal feature point correspondence relationship between images is obtained based on the point-to-plane distance theory, and matching data with the highest confidence between images is obtained.

[0054] The calculation formula for solving the optimal assignment matrix in step S5 by the Sinkhorn algorithm is: ; ; In the formula, C is the feature point matching distance, a and b are the distribution probability vectors of the feature points in the two images, and P1=a and P T 1=b are constrained, the Sinkhorn algorithm ensures the matching coverage of the low-texture area of the photovoltaic panel, avoids the missing matching caused by the sparsity of the feature points in the traditional algorithm, and at the same time, the matching probability weight b j of the key component area is constrained to improve the matching accuracy of the fault area.

[0055] S6: Estimate the homography matrix by the MAGSAC++ algorithm, and calculate the mathematical model describing the perspective transformation relationship between the two images; Step S6 includes calculating the homography matrix hypothesis from the randomly sampled minimum point set in the optimal matching pair, using an adaptive threshold strategy based on maximum likelihood estimation to evaluate the inliers and outliers of all matching points under the hypothesis, refining the homography matrix parameters corresponding to the inliers by weighted least squares method, and iteratively executing the above process until the homography matrix with the largest inlier set support and the smallest fitting error is selected. The expression of the MAGSAC++ algorithm for estimating the homography matrix is: ; In the formula, p is an adaptive loss function based on maximum likelihood. Further, step S7 also includes performing geometric transformation and resampling on the image pair to be spliced, and determining the optimal seam path by minimizing the energy function of graph cuts, and then splicing the panoramic image onto the target plane by splicing method. Steps S3 to S7 are repeatedly executed until all sub-area images are processed, and a complete photovoltaic area image is obtained.

[0056] The calculation formula for constructing the energy function is: ; In the formula, D p (L p ) represents an image pixel, and p represents an assigned label. The panoramic image is transformed onto the target plane for splicing by a splicing method, including measuring geometric errors of graphic pixel projection to a three-dimensional photovoltaic panel plane based on a point-to-plane distance theory, wherein a calculation formula of a smoothing term is: ; In the formula, V p,q (L p ,L q ) is an energy value of a smoothing term between the pixel p and an adjacent pixel q, and λ is a smoothing term weight coefficient, and the greater the value, the smoother the seam path A calculation formula of a double weight fusion strategy is: ; In the formula, I fused(x,y) is a fused pixel value, I1 and I2 are pixel values of two images to be spliced in an overlapping area, and w1 and w2 are adaptive weights.

[0057] S7: image splicing is performed based on a GraphCut optimization algorithm, seam effects are eliminated, a high-resolution panoramic image is output, and the panoramic image is transformed onto the target plane for splicing by a splicing method until a complete photovoltaic area image is obtained.

[0058] To verify the splicing capability of the application, comparative experiments are performed on the application and commonly used SIFT algorithm, ORB algorithm and SIFT-FREAK algorithm, and four evaluation indexes of AG average gradient, RMSE root mean square error, PSNR peak signal-to-noise ratio and SSIM structural similarity are used to measure the automatic splicing effect of the unmanned aerial vehicle picture.

[0059] As shown in Table 1, in the evaluation index comparison of indoor image splicing, the AG of the application is 5.2, the RMSE is 0.45, the PSNR is 24.81, and the SSIM is 0.85. Table 1

[0060] As shown in Table 2, in the evaluation index comparison of outdoor image splicing, the AG of the application is 5.3, the RMSE is 0.57, the PSNR is 24.24, and the SSIM is 0.79.

[0061] Table 2

[0062] Compared with the SIFT algorithm, the ORB algorithm and the SIFT-FREAK algorithm, the application is better, which indicates that the fused image has higher definition, smaller error and better structural similarity, effectively eliminates the joint effect, and outputs a high-resolution panoramic image.

[0063] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical range disclosed by the present application can be easily thought by any person skilled in the art, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for automatic stitching of drone images oriented to the localization of failed photovoltaic modules, characterized by, The method comprises the following steps: S1: dividing the area of the photovoltaic power station into multiple sub-regions, and dynamically assigning the inspection task according to the real-time power of each unmanned aerial vehicle, the sensor state and the environmental factors; S2: presetting the inspection route of each unmanned aerial vehicle, and collecting photovoltaic region images through the high-definition camera and infrared sensor carried thereon; S3: using a SuperPoint deep learning model to extract feature points of the edge of the convex region of the photovoltaic panel from the collected photovoltaic region images, and positioning the salient positions in the images for matching; S4: based on the feature points and their descriptors extracted in S3, using a SuperGlue algorithm to construct a correlation graph between the feature points, and combining an attention mechanism to calculate a similarity matrix between the feature points, selecting an optimal matching strategy according to the similarity matrix, and establishing a preliminary correspondence between the feature points of different images; S5: solving the optimal feature matching through a Sinkhorn algorithm, solving the double random form of the matching score matrix through row and column normalization iteration, obtaining the optimal correspondence between the feature points of the images, and obtaining the matching data between the images with the highest confidence; S6: estimating a homography matrix through a MAGSAC++ algorithm, and calculating a mathematical model describing the perspective transformation relationship between two images; S7: performing image stitching based on a GraphCut optimization algorithm, eliminating the joint effect and outputting a high-resolution panoramic image, and transforming the panoramic image to a target plane for stitching until a complete photovoltaic region image is obtained.

2. The method of claim 1, wherein, In step S1, the area of the photovoltaic power station is divided into multiple sub-regions, and the divided sub-regions include convex regions and non-convex regions. The division standard is mainly based on geometric characteristics. In the convex region, the line connecting any two points is completely contained in the region. In the non-convex region, there are at least two points whose connecting line is not completely contained in the region.

3. The method of claim 2, wherein, The convex region is divided using a variable step line scanning strategy, and the turning path uses a Dubins curve constraint.

4. The method of claim 2, wherein, The non-convex region is divided by combining Delaunay subdivision and Monte Carlo optimized path, and combining multi-objective function, resource allocation theory and unmanned aerial vehicle energy consumption model to construct a multi-objective optimization function to generate the sub-region task allocated to each unmanned aerial vehicle.

5. The method of claim 1, wherein, The dynamic allocation of the inspection task according to the real-time power of each unmanned aerial vehicle, the sensor state and the environmental factors includes generating a task allocation matrix through a multi-objective function and a Hungarian algorithm, which is used to reduce the area difference rate of the sub-regions allocated to each unmanned aerial vehicle.

6. The method of claim 1, wherein, In step S2, the inertia principal axis of the rigid body, the direction of the inertia principal axis of the polygon that minimizes the moment of inertia, is used to ensure that the moment of inertia change of the unmanned aerial vehicle is minimized, the flight energy consumption is reduced, and the overlap consistency of the image acquisition timing is ensured. The Dubins path is used to process the turning transition.

7. The method of claim 1, wherein, In step S2, the height of the unmanned aerial vehicle and the focus are defined by the similar triangle relationship, and the sampling density of each unmanned aerial vehicle is planned by the Shannon sampling theorem.

8. The method of claim 1, wherein, In step S3, a sub-region boundary constraint is introduced in the SuperPoint model, higher weights are given to feature points near the vertices of the Delaunay triangulation sub-region, and the detection probability of the edge features of the photovoltaic panel is strengthened through the attention mechanism.

9. The method of claim 1, wherein, Step S6 includes calculating the homography matrix hypothesis from the minimum point set randomly sampled from the optimal matching pair, evaluating the inliers and outliers of all matching points under the hypothesis using an adaptive threshold strategy based on maximum likelihood estimation, refining the homography matrix parameters corresponding to the inliers by weighted least squares, and iteratively performing the above process until the homography matrix with the largest inlier set support and the smallest fitting error is selected.

10. The method of claim 1, wherein, In step S7, the image pair to be spliced is also subjected to geometric transformation and resampling, and the optimal seam path is determined by minimizing the energy function through graph cut, and then the panoramic image is transformed to the target plane for splicing by splicing method, and steps S3 to S7 are repeatedly executed until all sub-region images are processed, and a complete photovoltaic region image is obtained.

Images