A method and system for batch analysis of unmanned aerial vehicle inspection images

Abstract

The application discloses a kind of unmanned aerial vehicle inspection image batch analysis method and system, belong to image recognition technical field.Extract each frame feature point to generate descriptor, eliminate false matching in bidirectional matching, establish interframe correspondence between discrete images;Initial pose is solved by inlier coordinates, and distortion is corrected frame by frame according to the tilt angle of three consecutive frames optical axis, to avoid distortion accumulation;According to the error determined by re-projection residual, if necessary, construct nonlinear optimization problem containing exterior and point cloud depth to solve iteratively, globally constrain frame-by-frame error, balanced absorption, precision and efficiency are considered;Eliminate the seam of splicing by resampling alignment and weighted fusion.Thereby, under the condition of view angle change and distortion accumulation, interframe correspondence is stably established and exterior parameter drift is constrained, and a panoramic monitoring image with consistent trend and credible scale is obtained, which is used for hidden danger continuous interpretation and position backtracking, solve the technical problems that large batch of inspection images are difficult to stably establish interframe correspondence and overcome the influence of distortion accumulation.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method and system for batch analysis of UAV inspection images. Background Technology

[0002] With the increasing demand for operation and maintenance of wide-area infrastructure such as power transmission lines, oil and gas pipelines, and transportation facilities, the use of drones to continuously collect on-site images along predetermined routes has gradually replaced traditional manual inspections. This method has a wide operating range and high data collection efficiency, and can accumulate a large number of images covering long-distance lines in a short period of time, providing a data foundation for hazard identification and condition assessment.

[0003] However, the images generated by a single inspection are usually presented as independent frames. If each frame is viewed as an isolated sample or only partially stitched together, the original spatial correspondence between the images is broken, and the trajectory of the inspected object in real space is difficult to fully reconstruct. When the same target spans multiple frames, it is difficult to determine whether the anomaly is an isolated point or a continuous defect, and it is also difficult to trace the location of the problem along the entire inspection path.

[0004] The difficulty in re-registering frame-by-frame images into a coherent view stems primarily from the imaging method of drones: during flight, the aircraft's attitude and gimbal pointing continuously change, resulting in differences in the angle and position of the same object across different frames; simultaneously, the images from the airborne wide-angle camera exhibit inherent distortion at the edges. These deviations are not significant in a single frame, but when a large number of images are stitched together sequentially, they accumulate frame by frame along the flight path, leading to misaligned seams, broken or ghosted targets, and even drift.

[0005] Therefore, how to stably establish inter-frame correspondences among a large number of inspection images, and how to unify discrete images into the same coordinate system to form a panoramic visual model that is coherent and covers the entire inspection area, while overcoming the cumulative effects of perspective changes and lens distortion, has become a technical problem that urgently needs to be solved to improve the batch analysis capabilities of UAV wide-area inspections. Summary of the Invention

[0006] This application provides a method and system for batch analysis of UAV inspection images, which can provide panoramic visual images with coherent directions and reliable proportions.

[0007] This application provides a method for batch analysis of UAV inspection images, including:

[0008] Based on the discrete image sequence captured by the drone, feature points of each frame are extracted and descriptors are generated to obtain the first feature point set;

[0009] Bidirectional matching is performed on the descriptors of adjacent frames in the first feature point set, and mismatched points are removed to obtain a second feature point set including in-point markers;

[0010] Based on the interior point markers in the second feature point set, extract the interior point coordinates of the corresponding interior points, obtain the rotation matrix, translation vector and three-dimensional depth value of the feature points between adjacent images, and obtain the initial pose data;

[0011] Based on the optical axis tilt angle of three consecutive frames in the initial pose data, distortion correction is performed on the edge region of the image to obtain the first corrected image;

[0012] The presence of cumulative error caused by camera extrinsic drift is determined based on the reprojection residual of the first corrected image.

[0013] For the first corrected image sequence with accumulated errors, a nonlinear optimization problem involving the depth values ​​of the 3D point cloud outside the camera is constructed and solved iteratively to determine globally consistent target pose data.

[0014] Based on the target pose data, the first corrected image is resampled onto a uniform projection plane to obtain a geometrically aligned second corrected image;

[0015] The overlapping region pixels are weighted and fused based on the width of the overlapping region of adjacent frames in the second corrected image sequence to obtain a panoramic monitoring image with stitching seams eliminated.

[0016] This application also provides a batch analysis system for UAV inspection images, including:

[0017] The feature extraction module is used to extract feature points from each frame of the discrete image sequence captured by the UAV and generate descriptors to obtain the first feature point set.

[0018] The matching and elimination module is used to perform bidirectional matching on the descriptors of adjacent frames in the first feature point set and eliminate mismatched points to obtain a second feature point set including in-point markers.

[0019] The initial pose module is used to extract the interior coordinates of the corresponding interior points based on the interior point markers in the second feature point set, obtain the rotation matrix, translation vector and three-dimensional depth value of the feature points between adjacent images, and obtain the initial pose data.

[0020] The distortion correction module is used to correct the distortion of the image edge region based on the optical axis tilt angle of three consecutive frames in the initial pose data, and obtain the first corrected image.

[0021] The error determination module is used to determine whether there is an accumulated error caused by the camera extrinsic drift based on the reprojection residual of the first corrected image.

[0022] The global optimization module is used to construct a nonlinear optimization problem for the first corrected image sequence with accumulated errors, including the depth values ​​of the 3D point cloud with external camera participants, and solve iteratively to determine globally consistent target pose data.

[0023] The image resampling module is used to resample the first corrected image onto a uniform projection plane based on the target pose data to obtain a geometrically aligned second corrected image.

[0024] The image fusion module is used to perform weighted fusion of pixels in the overlapping area based on the width of the overlapping area of ​​adjacent frames in the second corrected image sequence, so as to obtain a panoramic monitoring image with stitching seams eliminated.

[0025] This application provides a method and system for batch analysis of UAV inspection images. By extracting feature points from each frame of images and generating descriptors, bidirectionally matching descriptors of adjacent frames and eliminating mismatched points, a stable inter-frame correspondence is established between discrete images that are discontinuous in both time and space. Then, the initial pose is calculated based on the interior point coordinates, and edge distortion is corrected frame by frame according to the optical axis tilt angle of three consecutive frames, avoiding the accumulation of distortion along the flight path. Furthermore, the accumulated error is determined based on the reprojection residual, and a nonlinear optimization problem including the depth value of the three-dimensional point cloud outside the camera is constructed and iteratively solved only when necessary. The error accumulated frame by frame is uniformly constrained and evenly absorbed in the global scope, taking into account both accuracy and processing efficiency of large batches of images. Finally, visible stitching seams are eliminated through resampling alignment and weighted fusion of overlapping areas. This provides a visual intelligent analysis capability for drone inspection of large batches of discrete images. It can stably establish inter-frame correspondence and uniformly constrain extrinsic parameter drift under the conditions of continuous changes in viewpoint and frame-by-frame accumulation of lens distortion. It can obtain panoramic monitoring images with coherent direction, reliable proportions, and covering the entire inspection area. It can be used for continuous interpretation of hidden dangers and location tracing, thus solving the technical problem that it is difficult to stably establish inter-frame correspondence and overcome the effects of viewpoint changes and lens distortion accumulation on a large batch of inspection images.

[0026] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the implementation of a batch analysis method for UAV inspection images in an embodiment of this application.

[0029] Figure 2 This is a flowchart illustrating the implementation of another method for batch analysis of UAV inspection images in this application.

[0030] Figure 3 This is a flowchart illustrating the implementation of another method for batch analysis of UAV inspection images in this application.

[0031] Figure 4 This is a flowchart illustrating the implementation of another method for batch analysis of UAV inspection images in this application.

[0032] Figure 5 This is a flowchart illustrating the implementation of another method for batch analysis of UAV inspection images in this application.

[0033] Figure 6 This is a flowchart illustrating the implementation of another method for batch analysis of UAV inspection images in this application.

[0034] Figure 7 This is a flowchart illustrating the implementation of another method for batch analysis of UAV inspection images in this application.

[0035] Figure 8 This is a flowchart illustrating another method for batch analysis of UAV inspection images in this application.

[0036] Figure 9 This is a flowchart illustrating another method for batch analysis of UAV inspection images in this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0038] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0039] The design concept of the embodiments of this application is briefly introduced below:

[0040] Existing UAV inspection image processing solutions often interpret each frame of image captured along the flight path individually, or perform limited local stitching only at the same hovering point. This approach is limited by whether manual review is comprehensive and whether the spatial relationship between frames is explicitly established. It may miss the complete depiction of the continuous movement of the inspected object in real space. Once the target spans multiple frames, it becomes difficult to distinguish between isolated anomalies and continuous defects. To address this, some solutions introduce image registration to expand the field of view. However, current registration and stitching schemes often process only two adjacent frames independently, discretizing the global alignment problem. This only guarantees an approximate match of local seams and does not take into account the perspective differences introduced by the continuous changes in the UAV's attitude and gimbal pointing during flight, as well as the edge distortion of wide-angle cameras. This results in small deviations being transmitted frame by frame along the flight path. Furthermore, due to the lack of a unified constraint on the cumulative error caused by camera extrinsic drift, the error is continuously amplified after hundreds or thousands of images are stitched together in sequence. Adjacent images may exhibit misaligned seams, broken and ghosted targets, or even drifting of the movement, making it difficult to obtain a panoramic visual model with a coherent movement and reliable proportions.

[0041] In view of this, embodiments of this application provide a method and system for batch analysis of UAV inspection images. The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features described herein can be combined with each other without conflict. References Figure 1 The following is a flowchart illustrating the implementation of a batch analysis method for UAV inspection images provided in this application embodiment. The specific implementation process of this method is as follows:

[0042] S101: Based on the discrete image sequence captured by the UAV, extract the feature points of each frame and generate descriptors to obtain the first feature point set.

[0043] When a drone flies along a flight path, such as along a power line, its onboard camera captures images frame by frame at fixed time intervals, forming a discrete image sequence. For example, if a drone flies at a cruising speed of 8 m / s along a single-loop power transmission line of approximately 2 km, and the camera captures images at a frequency of 5 frames per second, a single mission yields approximately 1250 frames of images with a resolution of 5472×3648 pixels. For each frame, locally salient feature points (such as tower corners, insulator edges, conductor intersections, and other locations of abrupt changes in grayscale) are extracted, and a descriptor describing the appearance of its local neighborhood is generated for each feature point. The feature points and descriptors from all frames are then combined to obtain the first feature point set. For example, an average of 3000 to 8000 feature points are extracted per frame, and the size of the first feature point set after the entire sequence is aggregated is approximately in the millions.

[0044] S102: Perform bidirectional matching on the descriptors of adjacent frames in the first feature point set and remove mismatched points to obtain a second feature point set including in-point markers.

[0045] The k-th and (k+1)-th frames are grouped into adjacent frames. The descriptors of these two frames are compared to identify the most similar feature point pairs. Mismatches caused by repetitive textures (such as rows of similar insulators or parallel conductors) are eliminated through bidirectional consistency checks and geometric constraints. Correctly matched points are marked with an inner point label. After processing all adjacent frames in the complete sequence, a second set of feature points, including the inner point labels, is obtained. For example, if an initial matching of adjacent frames yields 2100 pairs of matching points, after elimination, approximately 1500 pairs of inner points are retained, resulting in a mismatch elimination rate of approximately 29%.

[0046] S103: Extract the interior coordinates of the corresponding interior points based on the interior point markers in the second feature point set, obtain the rotation matrix, translation vector and three-dimensional depth value of the feature points between adjacent images, and obtain the initial pose data.

[0047] The image coordinates (i.e., interior coordinates) of the feature points that are determined to be correctly matched are extracted from the second set of feature points according to their interior point markings. Based on this, the relative attitude change (rotation matrix) and relative translation direction (translation vector) between adjacent frames are estimated, and the three-dimensional depth values ​​of these feature points relative to the camera are recovered. After recursively applying this method frame by frame, the initial pose data covering the entire sequence is obtained. For example, the relative rotation calculated in a certain adjacent frame is approximately 3.2 degrees around the heading axis and 1.7 degrees around the pitch axis, and the three-dimensional depth values ​​of the feature points are distributed in the range of 35m to 120m, which is consistent with the relative flight altitude of approximately 80m for this sortie.

[0048] S104: Based on the optical axis tilt angle of three consecutive frames in the initial pose data, the distortion of the image edge region is corrected to obtain the first corrected image.

[0049] The camera optical axis is taken from three consecutive frames, and its relative tilt is evaluated. Based on this, the bending distortion caused by the wide-angle lens at the image edges is corrected, restoring the stretched or compressed edge areas to near-true geometric proportions, resulting in the first corrected image. For example, the area outside the normalized radius of 0.85 of the image width is defined as the edge area. After correction, the bending deflection of straight lines in this area is reduced from about 6 pixels before correction to less than 1 pixel.

[0050] S105: Determine whether there is a cumulative error caused by the camera extrinsic drift based on the reprojection residual of the first corrected image.

[0051] The 3D feature points are projected back onto the image plane according to their current pose, and the deviation between the projected position and the actual observed position is compared (reprojection residual). When the deviation systematically increases along the sequence, it indicates that the camera extrinsic parameters (position and attitude) have slowly drifted and accumulated in the frame-by-frame iteration. Based on this, it is determined that there is a cumulative error caused by the drift of the camera extrinsic parameters in the first corrected image sequence. For example, if the root mean square of the reprojection residual in the first segment of the sequence is about 0.6 pixels and rises to 4.8 pixels in the last segment, showing a monotonically increasing trend, it is determined that there is a cumulative error.

[0052] S106: For the first corrected image sequence with accumulated errors, construct a nonlinear optimization problem that includes the depth values ​​of the 3D point cloud outside the camera and solve iteratively to determine globally consistent target pose data.

[0053] By simultaneously considering all 3D point cloud depth values ​​from camera extrapolations across all frames of the entire sequence as variables to be optimized, a nonlinear optimization problem is constructed with the objective of minimizing the sum of all reprojection residuals. This problem is then solved iteratively, redistributing and unifying the errors accumulated frame by frame globally to obtain globally consistent target pose data. For example, when constructing an optimization problem for 1250 frames and approximately 300,000 3D points, the root mean square of the reprojection residuals across the entire sequence decreased from 4.8 pixels to 0.7 pixels after iterative convergence.

[0054] S107: Resample the first corrected image onto the unified projection plane based on the target pose data to obtain a geometrically aligned second corrected image.

[0055] Based on the optimized target pose data, each frame of the first corrected image is mapped from its respective imaging viewpoint onto the same projection plane and resampled to achieve geometric alignment of each frame in a unified coordinate system, resulting in the second corrected image. For example, using the imaging plane of a frame in the middle of the sequence as the reference projection plane, the alignment residual of corresponding points in adjacent frames is controlled within 1 pixel after resampling.

[0056] S108: The pixels in the overlapping area are weighted and fused according to the width of the overlapping area of ​​adjacent frames in the second corrected image sequence to obtain a panoramic monitoring image with the stitching seams eliminated.

[0057] Within the overlapping area of ​​adjacent frames, fusion weights are assigned based on the distance of pixels from the seam. Weighted fusion is then performed pixel-by-pixel in the overlapping area, resulting in a smooth transition in brightness and color between the two frames at the seam, eliminating visible seams, and ultimately obtaining a coherent panoramic monitoring image. For example, if the width of the overlapping area between adjacent frames is approximately 35% of the image width (approximately 1900 pixels), the brightness step on both sides of the seam after fusion is reduced from approximately 18 gray levels before correction to less than 2 gray levels.

[0058] As can be seen, the batch analysis method for UAV inspection images provided in this embodiment of the invention establishes a stable inter-frame correspondence between discrete images that are discontinuous in both time and space by extracting feature points of each frame image and generating descriptors, performing bidirectional matching of descriptors of adjacent frames and eliminating mismatched points; then, it calculates the initial pose based on the interior point coordinates and corrects edge distortion frame by frame according to the optical axis tilt angle of three consecutive frames, avoiding the accumulation of distortion along the flight path; furthermore, it determines the accumulated error based on the reprojection residual, and only when necessary constructs a nonlinear optimization problem including the depth value of the three-dimensional point cloud outside the camera and solves iteratively, uniformly constraining and absorbing the error accumulated frame by frame in a global range, taking into account both accuracy and processing efficiency of a large number of images; finally, visible stitching seams are eliminated by resampling alignment and weighted fusion of overlapping areas. This provides a visual intelligent analysis capability for drone inspection of large batches of discrete images. It can stably establish inter-frame correspondence and uniformly constrain extrinsic parameter drift under the conditions of continuous changes in viewpoint and frame-by-frame accumulation of lens distortion. It can obtain panoramic monitoring images with coherent direction, reliable proportions, and covering the entire inspection area. It can be used for continuous interpretation of hidden dangers and location tracing, thus solving the technical problem that it is difficult to stably establish inter-frame correspondence and overcome the effects of viewpoint changes and lens distortion accumulation on a large batch of inspection images.

[0059] refer to Figure 2 This embodiment of the invention further elaborates on the process of extracting feature points from each frame of an image based on a discrete image sequence captured by a drone and generating descriptors to obtain a first set of feature points, as follows: Figure 2 As shown, it includes S1011 to S1013.

[0060] S1011: For each frame of a discrete image sequence, the scale-invariant feature transform algorithm is used to extract candidate feature points whose corner response values ​​are higher than the adaptive threshold.

[0061] The scale-invariant feature transform algorithm detects candidate feature points by performing Gaussian differences on the image at multiple scales, identifying pixels that are extreme values ​​in both scale space and spatial location. To stabilize feature point detection in unevenly lit images, an adaptive threshold is used instead of a fixed threshold. The adaptive threshold is determined based on the statistical distribution of corner response values ​​in the current frame. For example, the threshold is the mean of all candidate corner response values ​​in the frame plus 1.5 times the standard deviation, retaining only candidate feature points with response values ​​higher than this threshold. This approach reduces the detection of overexposed sky areas with strong light and adaptively retains feature points in low-texture shadow areas of towers according to the contrast of the current frame, avoiding the inconsistent detection counts caused by a fixed threshold across frames with varying lighting conditions. For example, after adding 1.5 times the standard deviation to the mean of a certain frame, the threshold corresponds to approximately 0.04 (normalized corner response), with approximately 6200 candidate feature points exceeding this value.

[0062] The construction of the Gaussian difference scale space can be represented as:

[0063] ;

[0064] In the formula, For scale Gaussian difference response under the given conditions Standard deviation The Gaussian kernel is given, and φ is the scaling factor between adjacent scales. is the grayscale value of the input image, and * represents a two-dimensional convolution.

[0065] The adaptive threshold can be expressed as:

[0066] ;

[0067] In the formula, 𝑇 is the adaptive threshold for corner response. This is the average of the corner response values ​​in the current frame. The standard deviation of the corner response values ​​in the current frame. This is a multiplier, for example, 1.5.

[0068] S1012: Generate descriptor vectors based on the principal direction gradient histogram of candidate feature points.

[0069] Gradient direction histograms are plotted within the neighborhood of each candidate feature point. The peak direction of the histogram is taken as the principal direction of the feature point, and the neighborhood is rotated and aligned to the principal direction to obtain rotation invariance. Subsequently, the gradient direction distribution is plotted in blocks within the aligned neighborhood and concatenated to form a descriptor vector. For example, a 16×16 pixel neighborhood around a feature point is divided into 16 sub-regions of 4×4 pixels. Gradients in 8 directions are plotted in each sub-region, resulting in a 16×8=128-dimensional descriptor vector.

[0070] The gradient accumulation in a certain directional interval can be expressed as:

[0071] ;

[0072] In the formula, Let be the histogram value of the ⁏th direction interval. Let be the set of pixels whose orientation falls within the 𝑏-th interval. Let be the gradient magnitude of pixel 𝑝. The Gaussian weighting coefficients are set according to the distance from the center of the feature point.

[0073] S1013: Combine the spatial coordinates of the candidate feature points with the descriptor vector to obtain the first set of feature points.

[0074] Each candidate feature point's spatial coordinates in the image are combined with its 128-dimensional descriptor vector to form a feature record. The collection of feature records from all frames constitutes the first feature point set. For example, each record is in the form of frame number, x coordinates, y coordinates, and a 128-dimensional descriptor. The entire sequence contains approximately millions of records.

[0075] refer to Figure 3 This embodiment of the invention further elaborates on the process in S102 of performing bidirectional matching of descriptors of adjacent frames in the first feature point set and removing mismatched points to obtain a second feature point set including interior point markers, as follows: Figure 3 As shown, it includes S1021 to S1023.

[0076] S1021: For the feature descriptors of adjacent frames in the first feature point set, the nearest neighbor distance ratio method is used for bidirectional matching and filtering to obtain matching point pairs.

[0077] For two adjacent frames, the descriptor of a feature point in one frame is taken. In the other frame, the two points with the closest and second-closest distances to the descriptor are found, and the ratio of the closest distance to the second-closest distance is calculated. The smaller the ratio, the more unique and reliable the match. At the same time, a search is performed in both directions, and only point pairs that are both nearest neighbors to each other are retained, resulting in matching point pairs. For example, if the nearest neighbor distance ratio threshold is set to 0.7, and only point pairs with a ratio less than 0.7 are retained, approximately 2100 matching point pairs are obtained after bidirectional filtering for a certain adjacent frame.

[0078] The nearest neighbor distance ratio can be expressed as:

[0079] ;

[0080] In the formula, 𝑟 is the distance ratio. The Euclidean distance from the descriptor to be matched to its nearest neighbor. The distance to the next nearest neighbor is the Euclidean distance; when 𝑟 is less than the ratio threshold, it is considered a valid matching point pair.

[0081] S1022: Calculate the dynamic distance threshold based on the overlap rate of adjacent frame images.

[0082] When the overlap rate between adjacent frames is high, the displacement of corresponding points is small, and the allowable matching distance should be stricter. When the overlap rate is low, it should be appropriately relaxed. Therefore, a fixed threshold is not used, but a dynamic distance threshold is calculated based on the overlap rate. The overlap rate is estimated by the time interval between adjacent frames, the flight speed, and the ground sampling distance. For example, when the flight speed is 8 m / s, the frame rate is 5 frames per second, and the ground coverage width of a single frame is about 60 m, the displacement of adjacent frames along the flight direction is about 1.6 m, and the overlap rate is about (60−1.6) / 60≈97%.

[0083] The dynamic distance threshold can be expressed as:

[0084] ;

[0085] In the formula, For dynamic distance threshold, As the baseline distance threshold, The overlap rate between adjacent frames. For example, the overlap rate influence coefficient. Take 0.5, Take 0.3 and substitute it into the equation. have to .

[0086] S1023: If the Euclidean distance between the matching point pairs is greater than the dynamic distance threshold, the random sampling consensus algorithm is used to fit the fundamental matrix, eliminate mismatched points with inconsistent geometric constraints, and obtain the second feature point set including interior point labels.

[0087] For suspicious matching point pairs whose Euclidean spatial distance exceeds the dynamic distance threshold, a random sampling consensus algorithm is used to repeatedly randomly sample several point pairs to fit the fundamental matrix. The number of points satisfying the epipolar geometric constraints is counted, and the fitting result with the most inliers is taken as the optimal model. Point pairs that satisfy the constraints are marked as inliers, and those that do not are judged as mismatches and removed, resulting in a second feature point set including inlier markings. For example, after 1000 iterations and with an epipolar constraint error threshold of 1 pixel, approximately 1500 pairs of inliers are ultimately retained in a certain adjacent frame, with an inlier rate of approximately 71%.

[0088] The epipolar geometric constraint can be expressed as:

[0089] ;

[0090] In the formula, and These are the homogeneous coordinates of a pair of matching points in the two frames. The basic matrix; when If the error is less than the constraint error threshold, the point pair is determined to be an interior point.

[0091] refer to Figure 4This embodiment of the invention further elaborates on the process of S103, which involves extracting the interior coordinates of corresponding interior points based on the interior point markers in the second feature point set, obtaining the rotation matrix, translation vector, and three-dimensional depth value of the feature points between adjacent images, and obtaining the initial pose data. Figure 4 As shown, it includes S1031 to S1033.

[0092] S1031: Extract the coordinates of the interior points corresponding to the interior points based on the interior point markers in the second feature point set.

[0093] The second set of feature points is traversed, and only feature points with interior point labels are extracted. Their image coordinates in the two adjacent frames are read as interior point coordinates for subsequent pose calculation. For example, about 1500 pairs of interior point coordinates are extracted from an adjacent frame, all in pixels with the top left corner of the image as the origin.

[0094] S1032: Based on the coordinates of the interior points, the essential matrix is ​​decomposed using epipolar geometric constraints to obtain the rotation matrix and translation vector between adjacent images.

[0095] The intrinsic matrix is ​​estimated by normalizing the coordinates of the interior points using the camera's intrinsic parameters. Then, singular value decomposition is performed on the intrinsic matrix, and a unique physically feasible solution is selected by combining it with the constraint that the feature points should be located in front of the camera. This yields the rotation matrix and translation vector between adjacent images. For example, the rotation matrix corresponds to 3.2 degrees around the yaw axis, 1.7 degrees around the pitch axis, and 0.5 degrees around the roll axis, while the translation vector points in the direction of the flight path.

[0096] The relationship between the essential matrix and intrinsic / extrinsic parameters can be expressed as:

[0097] ;

[0098] ;

[0099] In the formula, For the essential matrix, For the camera intrinsic parameter matrix, Let be a rotation matrix. It is a translation vector. It is the antisymmetric matrix of the translation vector.

[0100] S1033: Based on the rotation matrix and translation vector, the three-dimensional depth values ​​of the feature points are recovered using triangulation to obtain initial pose data including camera position and attitude angle.

[0101] Given the relative rotation matrix and translation vector of two adjacent frames, the line of sight to the same feature point in both frames is extended in reverse to find the intersection point, i.e., triangulation, to recover the three-dimensional depth value of the feature point relative to the camera. Combined with the camera position and attitude angles recursively calculated frame by frame, the initial pose data is obtained. For example, the three-dimensional depth values ​​of the feature point obtained by triangulation are distributed between 35m and 120m, and the attitude angles are represented by three components: heading, pitch, and roll, which is consistent with a relative flight altitude of approximately 80m.

[0102] The projection relationship of triangulation can be expressed as:

[0103] ;

[0104] In the formula, Let be the 3D coordinates of the feature point, and be the projection depth scale factor, i.e., the 3D depth value. The coordinates of this feature point are the homogeneous coordinates of the image.

[0105] refer to Figure 5 This embodiment of the invention further details the process of correcting the distortion of the image edge region based on the optical axis tilt angle of three consecutive frames in the initial pose data, as described in S104, to obtain the first corrected image. Figure 5 As shown, it includes S1041 to S1043.

[0106] S1041: Based on the optical axis tilt angle of three consecutive frames in the initial pose data, calculate the pitch angle offset and baseline distance change rate between adjacent frames.

[0107] Starting from the initial pose data, three consecutive frames of attitude angles and camera positions are taken. The angle difference of the optical axis in the pitch direction between adjacent frames (pitch angle offset) and the percentage change in the distance between adjacent cameras relative to the previous distance (baseline distance change rate) are calculated to characterize the imaging attitude fluctuations caused by wind disturbance and flight path curvature. For example, if the pitch angles of three consecutive frames are 2.1 degrees, 3.9 degrees, and 5.6 degrees respectively, the adjacent pitch angle offsets are approximately 1.8 degrees and 1.7 degrees; the adjacent baseline distance changes from 1.6m to 1.75m, with a baseline distance change rate of approximately 9.4%.

[0108] The rate of change of baseline distance can be expressed as:

[0109] ;

[0110] In the formula, 𝛾 is the rate of change of baseline distance. and These represent the camera spacing between adjacent frames in the preceding and following segments, respectively.

[0111] S1042: If the pitch angle offset exceeds the angle threshold calibrated based on the flight altitude, a distortion correction mapping table is constructed based on the pre-calibrated radial distortion coefficient and tangential distortion coefficient.

[0112] The angle threshold varies with flight altitude: the higher the flight altitude and the greater the ground sampling distance, the more significant the edge geometric error caused by the same pitch angle offset becomes. Therefore, the angle threshold tightens as flight altitude increases. This threshold is pre-measured by ground calibration experiments at different flight altitudes. When the pitch angle offset exceeds the angle threshold, the radial distortion coefficient and tangential distortion coefficient obtained from the camera's factory or field calibration are used to pre-calculate the correspondence between the coordinates before and after correction pixel by pixel, constructing a distortion correction mapping table. For example, at a flight altitude of 80m, the angle threshold is calibrated to 2 degrees. In S1041, 1.8 degrees does not exceed the threshold, but another adjacent offset of 2.4 degrees exceeds the threshold, triggering the construction of the mapping table; the radial distortion coefficient, for example... =−0.28、 =0.09, tangential distortion coefficient for example =0.001、 =−0.0008, the above value is only an example, the actual value depends on the lens model, focal length and calibration results.

[0113] The distortion correction model can be expressed as:

[0114] ;

[0115] ;

[0116] In the formula, (𝑥,𝑦) are normalized distortion coordinates. For the corrected coordinates, The square of the normalized radius to the optical center, , The radial distortion coefficient is... , denoted as the tangential distortion coefficient.

[0117] Radial and tangential distortion coefficients can be obtained in one step during the on-site calibration process before camera factory calibration or inspection, using a checkerboard calibration board. This involves acquiring multiple images of the calibration board in different poses, extracting the checkerboard corner points, and fitting a least-squares model of the camera imaging model based on the deviation between the ideal projection position and the actual imaging position of each corner point, along with the intrinsic parameter matrix. The results are calculated together and remain unchanged throughout the processing of the entire batch of inspection images after calibration. , Barrel or pincushion distortion, which reflects the radial symmetry of the lens, increases with the increase of the lens's wide-angle. , Tangential distortion, which reflects the non-perpendicularity between the lens optical axis and the imaging plane, is typically much smaller in magnitude than the radial coefficient. For example, the distortion obtained during the calibration of a wide-angle inspection lens... Approximately -0.28 Approximately 0.11 Approximately 0.0006 Approximately -0.0004, Negative values ​​correspond to barrel distortion. Taking a positive value has a secondary correction effect on the distortion of image edges, tangential coefficient , The order of magnitude is significantly smaller than the radial coefficient; the above values ​​are for illustrative purposes only, and the actual values ​​will vary depending on the lens model, focal length, and calibration results.

[0118] S1043: Perform reverse interpolation compensation on the pixel displacement of the image edge region according to the distortion correction mapping table to obtain the first corrected image with the edge curvature eliminated.

[0119] For image edge regions, the source coordinates before correction are looked up using the distortion correction mapping table. The source coordinates generally fall in non-integer positions. Interpolation values ​​are then used to fill in the corrected pixels, i.e., reverse interpolation compensates for pixel displacement, resulting in the first corrected image that eliminates edge curvature. For example, after pixel-by-pixel compensation for edge regions outside a normalized radius of 0.85, the bending deflection of straight lines on the edge is reduced from approximately 6 pixels to less than 1 pixel.

[0120] refer to Figure 6 The present invention provides a more detailed explanation of the process in S105 of determining whether there is an accumulated error caused by the camera extrinsic parameter drift based on the reprojection residual of the first corrected image, such as... Figure 6 As shown, it includes S1051 to S1053.

[0121] S1051: Obtain the projection position of the three-dimensional feature points of the first corrected image in the global coordinate system.

[0122] The 3D feature points recovered by S1033 are projected onto the image plane of each frame according to the camera extrinsic parameters, thus obtaining the projected positions of the 3D feature points in the global coordinate system after mapping by the camera model. For example, the projected positions of the 3D points corresponding to approximately 1500 interior points in each frame are calculated and expressed in pixels.

[0123] The projection position can be represented as:

[0124] ;

[0125] ;

[0126] In the formula, The coordinates of the i-th 3D feature point are obtained by mapping from the camera model. , , They are respectively The first component, the second component, and the third component, wherein the third component is the depth of the 3D feature point in the camera coordinate system; These are the normalized two-dimensional projected pixel coordinates. For the camera intrinsic parameter matrix, Let i be the coordinates of the i-th 3D feature point. and Let be the rotation matrix and translation vector of the camera extrinsic parameters in the global coordinate system for the j-th frame.

[0127] S1052: Calculate the pixel-level reprojection residual distribution based on the projection position and the original observation point.

[0128] The projection position is subtracted point by point from the original observation point actually detected by the feature point in the image to obtain the pixel-level deviation of each feature point, which is then summarized into the reprojection residual distribution of the entire frame or even the entire sequence. For example, the residual of a certain frame may show a distribution with large edges and small centers, and the residual of edge feature points can reach 3 to 5 pixels.

[0129] The reprojection residual can be expressed as:

[0130] ;

[0131] In the formula, Let the reprojection residual be the nth feature point. These are the pixel coordinates of the original observation point. Let its projection position be the pixel coordinates. It is a 2-norm.

[0132] S1053: If the root mean square value of the reprojection residual is greater than the error threshold set according to the camera resolution, it is determined that there is a cumulative error caused by the camera extrinsic drift.

[0133] The error threshold is set according to the camera resolution: the higher the resolution and the smaller the actual ground size corresponding to a single pixel, the stricter the allowable number of residual pixels should be. Therefore, the error threshold is calculated based on the camera resolution, derived from the ground sampling distance per pixel and the allowable geometric error. When the root mean square value of the reprojection residual is greater than the error threshold, it is determined that there is a cumulative error caused by the camera's extrinsic parameter drift. For example, for a 5472×3648 pixel camera, the error threshold is set to 2 pixels. If the root mean square value of the residual at the end of the sequence reaches 4.8 pixels, exceeding the threshold, then a cumulative error is determined to exist.

[0134] The root mean square of the reprojection residual can be expressed as:

[0135] ;

[0136] In the formula, Let be the root mean square value of the reprojection residuals, and let be the total number of feature points participating in the statistics. For the first The reprojection residual of each feature point; when If the error exceeds the error threshold, it is determined that there is a cumulative error.

[0137] refer to Figure 7 This embodiment of the invention further elaborates on the process of constructing and iteratively solving a nonlinear optimization problem involving the depth values ​​of the 3D point cloud with external camera involvement for the first corrected image sequence with accumulated errors in S106, and determining globally consistent target pose data, as follows: Figure 7 As shown, it includes S1061 to S1063.

[0138] S1061: For the first corrected image sequence with accumulated errors, construct a nonlinear optimization problem of the observation equations that include the depth values ​​of the 3D point cloud from all camera extrapolations.

[0139] The camera extrapolation of each frame in the entire sequence, along with all 3D point cloud depth values, are simultaneously used as variables to be optimized. An observation equation is constructed using the difference between the projected position of each 3D point in each frame and its original observation point. The sum of squared residuals from all observation equations is then used as the objective function to construct a nonlinear optimization problem (i.e., bundle adjustment). For example, with 1250 frames, approximately 300,000 3D points, and about 4.5 million observations, the scale of variables to be optimized reaches millions.

[0140] The objective function can be expressed as:

[0141] ;

[0142] In the formula, 𝑀 represents the total number of image frames. Let be the set of 3D points visible in frame 'x'. Let be the original observation point of the 3D point in frame , and let ⋋(⋅) be the camera projection function. and For the nth frame, the camera's external parameters, Let be the coordinates of the 𝑖th 3D point.

[0143] S1062: Assign constraint weights based on the number of shared feature points.

[0144] The more feature points observed between two frames, the more reliable their relative pose constraints. Therefore, constraint weights are assigned to the corresponding observation equations according to the number of shared feature points, with a higher weight for frames with more shared feature points, in order to suppress the interference of weakly connected frames on the global solution. For example, adjacent frames with about 1500 shared feature points are assigned a weight of 1.0, while frames with a larger span and only 200 shared feature points are assigned a weight of about 0.13.

[0145] The constraint weights can be expressed as:

[0146] ;

[0147] In the formula, Let be the constraint weights of the observation equation between frame x and frame l. This represents the number of shared feature points between two frames. This represents the maximum number of shared viewpoints across all frame pairs in the entire sequence.

[0148] S1063: The sparse Jacobian matrix is ​​solved using the Levenberg-Marquardt iterative method. The convergence rate of the pose parameters is monitored until the residual change rate is lower than the convergence criterion, thus determining globally consistent target pose data.

[0149] Leveraging the sparse distribution of camera and 3D points in the Jacobian matrix of the observation equation, a Levenberg-Marquardt iterative method is employed to adaptively adjust the damping factor between Gaussian-Newton and gradient descent methods. After each iteration, the residual change rate is monitored, and iteration stops when the residual change rate falls below the convergence criterion, resulting in globally consistent target pose data. For example, with an initial damping factor of 0.001 and a convergence criterion of residual change rate less than 1% between two consecutive iterations, convergence occurs in approximately 18 iterations, reducing the root mean square of the full-sequence reprojection residual from 4.8 pixels to 0.7 pixels.

[0150] The Levenberg-Marquardt iterative update can be represented as:

[0151] ;

[0152] In the formula, The Jacobian matrix is ​​the equation of observation. This is a diagonal weight matrix composed of constraint weights. Let be the residual vector, and be the damping factor. It is the identity matrix. This represents the increment of the variable to be optimized.

[0153] The residual rate of change can be expressed as:

[0154] ;

[0155] In the formula, The residual rate of change and Let be the objective function values ​​for the nth and (n-1)th iterations, respectively; when Stop iterating when the convergence criterion is met.

[0156] refer to Figure 8This embodiment of the invention further elaborates on the process of resampling the first corrected image onto a unified projection plane based on the target pose data in step S107 to obtain a geometrically aligned second corrected image, as follows: Figure 8 As shown, it includes S1071 to S1072.

[0157] S1071: Based on the rotation matrix and translation vector in the target pose data, establish a spatial transformation relationship from pixel coordinates to world coordinates for the first corrected image.

[0158] The rotation matrix and translation vector of each frame in the target pose data are taken, and combined with the camera intrinsic parameters to establish a mapping between the pixel coordinates of that frame and the world coordinate system, which serves as the spatial transformation relationship for projecting the image onto a unified plane. For example, the imaging plane of a certain frame in the sequence is used as the reference projection plane, and each frame establishes a homography transformation to that plane based on this plane.

[0159] Spatial transformation relationships can be expressed as:

[0160] ;

[0161] In the formula, Three-dimensional coordinates in the world coordinate system. For pixel homogeneous coordinates, The inverse of the camera intrinsic parameter matrix. and Let be the rotation matrix and translation vector of the target pose in this frame, and be the depth scale factor.

[0162] S1072: Based on the spatial transformation relationship, the first corrected image is resampled to the unified projection plane using bilinear interpolation to obtain a geometrically aligned second corrected image.

[0163] Based on spatial transformation, the corresponding position of each target pixel on the unified projection plane is retrieved from the source image. This position often falls on non-integer coordinates. The target pixel value is obtained by taking the distance weight of the four surrounding integer pixels, i.e., bilinear interpolation. After pixel-by-pixel resampling, a geometrically aligned second corrected image is obtained. For example, after resampling, the alignment residual of corresponding points in adjacent frames is controlled within 1 pixel.

[0164] Bilinear interpolation can be expressed as:

[0165] ;

[0166] In the formula, The pixel values ​​obtained through interpolation. , , , represents the value of the four integer pixels surrounding the source coordinates, and 𝑎 and 𝑏 are the decimal offsets of the source coordinates relative to the upper left integer pixels in the 𝑥 and 𝑦 directions, respectively.

[0167] refer to Figure 9 This embodiment of the invention further details the process of S108, which involves weighted fusion of pixels in the overlapping region based on the width of the overlapping region between adjacent frames in the second corrected image sequence to obtain a panoramic monitoring image with eliminated stitching seams. Figure 9 As shown, it includes S1081 to S1082.

[0168] S1081: Based on the width of the overlapping region of adjacent frames in the second corrected image sequence, establish a gradient fusion weight function with the distance from the suture line as the independent variable within the overlapping region.

[0169] First, determine the width of the overlapping region between two adjacent frames on a uniform projection plane, and then select a seam line in the center of the overlapping region. Next, establish a gradient blending weight function with the distance from the pixel to the seam line as the independent variable, so that pixels closer to the current frame are dominated by the current frame's weight, and pixels closer to the adjacent frame are dominated by the adjacent frame's weight, resulting in a smooth transition near the seam line. For example, if the overlapping region width is approximately 1900 pixels, the weight gradually changes linearly from 0 to 1 along the direction perpendicular to the seam line.

[0170] The gradient blending weight function can be expressed as:

[0171] ;

[0172] In the formula, is the fusion weight for this pixel in the current frame. The fusion weight can be between 0 and 1, and is the horizontal coordinate of the pixel. The x-axis represents the suture line. This represents the width of the overlapping region of the image.

[0173] S1082: Based on the gradient fusion weight function, the weighted average method is used to fuse the red, green and blue three-channel pixel values ​​of the overlapping area pixel by pixel to obtain a panoramic monitoring image with seamless stitching.

[0174] Within the overlapping area, the corresponding pixels of the two frames are weighted and averaged according to the gradient fusion weight function for the red, green, and blue channels respectively. After pixel-by-pixel fusion, the brightness and color at the seam are smoothly connected, resulting in a panoramic monitoring image with a continuous direction without seams. For example, after fusion, the brightness step on both sides of the seam is reduced from about 18 gray levels to less than 2 gray levels.

[0175] The weighted fusion of three channels can be expressed as:

[0176] ;

[0177] In the formula, The pixel value of the 𝑐 channel after fusion. and These are the values ​​of the L-th channel of this pixel in two adjacent frames, respectively. The gradient blending weights obtained from S1081 are R, G, and B, which represent the red, green, and blue channels, respectively.

[0178] Based on the same inventive concept, embodiments of this application also provide a batch analysis system for UAV inspection images, which may include:

[0179] The feature extraction module is used to extract feature points from each frame of the discrete image sequence captured by the UAV and generate descriptors to obtain the first feature point set.

[0180] The matching and elimination module is used to perform bidirectional matching on the descriptors of adjacent frames in the first feature point set and eliminate mismatched points to obtain a second feature point set including in-point markers.

[0181] The initial pose module is used to extract the interior coordinates of the corresponding interior points based on the interior point markers in the second feature point set, obtain the rotation matrix, translation vector and three-dimensional depth value of the feature points between adjacent images, and obtain the initial pose data.

[0182] The distortion correction module is used to correct the distortion of the image edge region based on the optical axis tilt angle of three consecutive frames in the initial pose data, and obtain the first corrected image.

[0183] The error determination module is used to determine whether there is an accumulated error caused by the camera extrinsic drift based on the reprojection residual of the first corrected image.

[0184] The global optimization module is used to construct a nonlinear optimization problem for the first corrected image sequence with accumulated errors, including the depth values ​​of the 3D point cloud with external camera participants, and solve iteratively to determine globally consistent target pose data.

[0185] The image resampling module is used to resample the first corrected image onto a uniform projection plane based on the target pose data to obtain a geometrically aligned second corrected image.

[0186] The image fusion module is used to perform weighted fusion of pixels in the overlapping area based on the width of the overlapping area of ​​adjacent frames in the second corrected image sequence, so as to obtain a panoramic monitoring image with stitching seams eliminated.

[0187] In some possible implementations, the batch analysis system for UAV inspection images according to this application may include at least a processor and a memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps in the batch analysis method for UAV inspection images according to various exemplary embodiments of this application described in this specification.

[0188] Based on the same inventive concept, this application also provides an electronic device that can realize the function of the aforementioned batch analysis method for UAV inspection images. The electronic device includes:

[0189] At least one processor and a memory connected to at least one processor. In this embodiment, the specific connection medium between the processor and the memory is not limited.

[0190] In this embodiment, the memory stores instructions executable by at least one processor. By executing the instructions stored in the memory, the at least one processor can perform the batch analysis method for UAV inspection images described above. The processor can implement the functions of each module in the batch analysis system for UAV inspection images.

[0191] The processor is the control center of the system. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the various functions and data processing of the system as a whole.

[0192] In one possible design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0193] The processor can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the batch analysis method for UAV inspection images disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0194] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited to this. The memory in the embodiments of this application can also be a circuit or any other system capable of implementing storage functions for storing program instructions and / or data.

[0195] By designing and programming the processor, the code corresponding to the batch analysis method of UAV inspection images described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the code during runtime. Figure 1 The steps of the batch analysis method for drone inspection images shown in the embodiment are as follows. How to design and program the processor is a technique well-known to those skilled in the art and will not be described further here.

[0196] Based on the same inventive concept, this application also provides a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the batch analysis method for UAV inspection images described above.

[0197] In some possible implementations, various aspects of the batch analysis method for UAV inspection images provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a system, the program code is used to cause the control device to perform the steps in the batch analysis method for UAV inspection images according to various exemplary embodiments of this application described above.

[0198] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0199] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0200] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0201] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0202] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for batch analysis of UAV inspection images, characterized in that, The method includes: Based on the discrete image sequence captured by the drone, feature points of each frame are extracted and descriptors are generated to obtain the first feature point set; Bidirectional matching is performed on the descriptors of adjacent frames in the first feature point set, and mismatched points are removed to obtain a second feature point set including in-point markers; Based on the interior point markers in the second feature point set, extract the interior point coordinates of the corresponding interior points, obtain the rotation matrix, translation vector and three-dimensional depth value of the feature points between adjacent images, and obtain the initial pose data; Based on the optical axis tilt angle of three consecutive frames in the initial pose data, distortion correction is performed on the image edge region to obtain the first corrected image; The presence of cumulative error caused by camera extrinsic drift is determined based on the reprojection residual of the first corrected image. For the first corrected image sequence with the accumulated error, a nonlinear optimization problem including the depth values ​​of the 3D point cloud outside the camera is constructed and solved iteratively to determine globally consistent target pose data; Based on the target pose data, the first corrected image is resampled onto a uniform projection plane to obtain a geometrically aligned second corrected image; The overlapping region pixels are weighted and fused based on the width of the overlapping region of adjacent frames in the second corrected image sequence to obtain a panoramic monitoring image with stitching seams eliminated.

2. The batch analysis method for UAV inspection images according to claim 1, characterized in that, Feature points are extracted from each frame of the image and descriptors are generated to obtain the first feature point set, including: For each frame of the discrete image sequence, the scale-invariant feature transform algorithm is used to extract candidate feature points whose corner response values ​​are higher than the adaptive threshold; Generate a descriptor vector based on the principal direction gradient histogram of the candidate feature points; The spatial coordinates of the candidate feature points are combined with the descriptor vector to obtain the first set of feature points.

3. The batch analysis method for UAV inspection images according to claim 1, characterized in that, Bidirectional matching is performed on the descriptors of adjacent frames in the first feature point set, and mismatched points are removed to obtain a second feature point set including inlier markers, comprising: For the feature descriptors of adjacent frames in the first feature point set, a bidirectional matching and filtering method is used to obtain matching point pairs by the nearest neighbor distance ratio method. The dynamic distance threshold is calculated based on the overlap rate of the adjacent frame images; If the Euclidean distance between the matching point pairs is greater than the dynamic distance threshold, then the random sampling consensus algorithm is used to fit the fundamental matrix, eliminate mismatched points with inconsistent geometric constraints, and obtain a second feature point set including interior point markers.

4. The batch analysis method for UAV inspection images according to claim 1, characterized in that, Based on the interior point markers in the second feature point set, extract the interior point coordinates of the corresponding interior points, obtain the rotation matrix, translation vector, and 3D depth values ​​of the feature points between adjacent images, and obtain the initial pose data, including: Extract the coordinates of the corresponding interior points based on the interior point markers in the second feature point set; Based on the inlier coordinates, the essential matrix is ​​decomposed using epipolar geometry constraints to obtain the rotation matrix and translation vector between adjacent images. Based on the rotation matrix and the translation vector, the three-dimensional depth value of the feature point is recovered using triangulation, resulting in initial pose data including camera position and attitude angle.

5. The batch analysis method for UAV inspection images according to claim 1, characterized in that, Based on the optical axis tilt angle of three consecutive frames in the initial pose data, distortion correction is performed on the image edge region to obtain a first corrected image, including: Based on the optical axis tilt angle of three consecutive frames in the initial pose data, calculate the pitch angle offset and baseline distance change rate between adjacent frames. If the pitch angle offset exceeds the angle threshold calibrated according to the flight altitude, a distortion correction mapping table is constructed based on the pre-calibrated radial distortion coefficient and tangential distortion coefficient. The pixel displacement of the image edge region is compensated by inverse interpolation according to the distortion correction mapping table to obtain a first corrected image with eliminated edge curvature.

6. The batch analysis method for UAV inspection images according to claim 1, characterized in that, Determining whether there is a cumulative error caused by camera extrinsic drift based on the reprojection residual of the first corrected image includes: Obtain the three-dimensional feature point projection positions of the first corrected image in the global coordinate system; Calculate the pixel-level reprojection residual distribution based on the projection position and the original observation point; If the root mean square value of the reprojection residual is greater than the error threshold set according to the camera resolution, it is determined that there is a cumulative error caused by the camera extrinsic drift.

7. The batch analysis method for UAV inspection images according to claim 1, characterized in that, For the first corrected image sequence with the accumulated error, a nonlinear optimization problem involving the depth values ​​of the 3D point cloud outside the camera is constructed and iteratively solved to determine globally consistent target pose data, including: For the first corrected image sequence with the aforementioned accumulated error, a nonlinear optimization problem is constructed that includes the observation equations of the 3D point cloud depth values ​​involving all camera extrapolations. Constraint weights are assigned based on the number of shared feature points; The sparse Jacobian matrix is ​​solved using the Levenberg-Marquardt iterative method. The convergence rate of the pose parameters is monitored until the residual change rate is lower than the convergence criterion, thus determining globally consistent target pose data.

8. The batch analysis method for UAV inspection images according to claim 1, characterized in that, The first corrected image is resampled onto a uniform projection plane based on the target pose data to obtain a geometrically aligned second corrected image, including: Based on the rotation matrix and translation vector in the target pose data, a spatial transformation relationship from pixel coordinates to world coordinates is established for the first corrected image. Based on the spatial transformation relationship, the first corrected image is resampled to a unified projection plane using bilinear interpolation to obtain a geometrically aligned second corrected image.

9. The batch analysis method for UAV inspection images according to claim 1, characterized in that, The overlapping region pixels in the second corrected image sequence are weighted and fused based on the width of the overlapping region of adjacent frames to obtain a panoramic monitoring image with stitching seams eliminated, including: Based on the width of the overlapping region of adjacent frames in the second corrected image sequence, a gradient fusion weight function with the distance from the suture line as the independent variable is established in the overlapping region; Based on the gradient fusion weight function, the red, green and blue three-channel pixel values ​​of the overlapping area are fused pixel by pixel using a weighted average method to obtain a panoramic monitoring image with seamless stitching.

10. A batch analysis system for UAV inspection images, characterized in that, The system includes: The feature extraction module is used to extract feature points from each frame of the discrete image sequence captured by the UAV and generate descriptors to obtain the first feature point set. The matching and elimination module is used to perform bidirectional matching on the descriptors of adjacent frames in the first feature point set and eliminate mismatched points to obtain a second feature point set including in-point markers. The initial pose module is used to extract the interior coordinates of the corresponding interior points based on the interior point markers in the second feature point set, obtain the rotation matrix, translation vector and three-dimensional depth value of the feature points between adjacent images, and obtain the initial pose data. The distortion correction module is used to correct the distortion of the image edge region based on the optical axis tilt angle of three consecutive frames in the initial pose data to obtain the first corrected image. The error determination module is used to determine whether there is an accumulated error caused by the camera extrinsic drift based on the reprojection residual of the first corrected image. The global optimization module is used to construct a nonlinear optimization problem, including the depth values ​​of the three-dimensional point cloud with external camera involvement, for the first corrected image sequence with the accumulated error, and to solve iteratively to determine globally consistent target pose data. An image resampling module is used to resample the first corrected image onto a uniform projection plane based on the target pose data to obtain a geometrically aligned second corrected image. The image fusion module is used to perform weighted fusion of pixels in the overlapping area based on the width of the overlapping area of ​​adjacent frames in the second corrected image sequence, so as to obtain a panoramic monitoring image with seams eliminated.

Images