Intelligent aerial triangulation method for swinging and scanning large-view-field infrared scanner by unmanned aerial vehicle

Abstract

According to the intelligent aerial triangulation method for the unmanned aerial vehicle swing-scanning large-view-field infrared scanner, firstly, camera installation error and POS system installation error parameters of the scanner are calculated, an imaging model of the large-view-field infrared scanner is established, and on the basis of the imaging model of the scanner, an error source and a swing-scanning strip internal error rule, an aerial triangulation algorithm is established based on the error source and the swing-scanning strip internal error rule. Establishing a self-adaptive orientation slice interpolation model; then, by analyzing the internal residual error curve of the detected and calibrated sweep strip, determining an orientation slice image needing to be extracted in the sweep strip by adopting a Douglas algorithm, and carrying out aerial triangulation block adjustment by adopting a self-adaptive orientation slice interpolation model; a block adjustment process of an infrared scanner, a precision evaluation method of a swept image and a correction method of the swept image are constructed, and three groups of experiments are designed for the three parts of space. The method is good in stability and high in accuracy, meets the requirement for high-precision mapping, and facilitates the realization of popularization and civilization of a large-view-field infrared scanner.

Description

Technical Field

[0001] The present application relates to an aerial triangulation method for a swing-sweeping large-field-of-view infrared scanner, and in particular to an intelligent aerial triangulation method for a swing-sweeping large-field-of-view infrared scanner for unmanned aerial vehicles, belonging to the technical field of aerial triangulation methods for infrared scanners. Background Art

[0002] In recent years, aerospace photogrammetry has become a highly effective technical tool for modern surveying and mapping and related industries. With the rapid development of aerospace, computer, and network communication technologies, the ability to acquire aerial survey data has continued to increase, and acquisition methods have become more diverse. Digital aerial survey cameras are developing towards large-array, large-field-of-view, high-resolution, and high-precision digital aerial imaging systems. Currently, there is considerable research on linear array push-broom imaging systems, while limited research is available on linear array swing-broom imaging sensors. With the advancement of photogrammetry and the increasing demand for wide-field-of-view imaging, the swing-broom imaging method has demonstrated its unique advantages and value: it can achieve wide-field-of-view imaging by adjusting the swing-broom angle while meeting other technical requirements, producing continuous, overlapping swing-broom swaths during flight.

[0003] Swinging, wide-field-of-view infrared scanners offer significant socioeconomic benefits. Interpreting their high-resolution swing-scan images can be used for resource surveying, environmental monitoring, and crop pest control. They also have applications in photoelectric detection and aerial reconnaissance. This scanner utilizes a linear array CCD sensor with 10 spectral bands, enabling multispectral, wide-field-of-view, and high-resolution Earth observation. Given the limited research on linear array swing-scan sensors, which offer the advantages of a large field of view, high-resolution imaging, and significant socioeconomic benefits, developing mathematical models and adjustment methods for swing-scan imaging systems is of great value.

[0004] Analytical aerial triangulation is the process of determining the exterior orientation elements of images within the image capture area. Various methods can be used to model the flight trajectory. The directional slice interpolation model selects linear array images along the entire flight trajectory as directional slices at regular intervals (typically based on the accuracy of the position data). The pose at each directional slice is considered unknown, and the poses of the linear array images at other times are interpolated from the directional slices. However, this directional slice interpolation model extracts directional slices at uniform intervals and does not yet include an adaptive selection method. Furthermore, there is currently little research on aerial triangulation for linear array swing-scan imaging systems. Therefore, developing adaptive directional slice interpolation adjustments for swing-scan imaging scanners is of great significance.

[0005] The problems that need to be solved by the existing aerial triangulation method of aerial infrared scanners and the key technical difficulties of this application include:

[0006] (1) Since there is currently little research on linear array swing-scan sensors, and swing-scan sensors have the advantages of large field of view, high-resolution imaging and huge social and economic benefits, it is of great value to develop mathematical models and adjustment methods for swing-scan imaging systems. The directional slice interpolation model of analytical aerial triangulation is to select some linear array images as directional slices on the entire flight trajectory at a certain time interval, regard the attitude of the directional slice at that moment as an unknown number, and interpolate the attitude of the linear array images at other moments based on the directional slices. However, in the directional slice interpolation model, the extraction of directional slices is carried out at the same time interval, and an adaptive selection method has not yet been included; at the same time, there is currently little research on aerial triangulation of linear array swing-scan imaging systems, so the development of adaptive directional slice interpolation adjustment for swing-scan imaging scanners is a key technical difficulty. The accuracy of POS cannot meet the requirements of high-precision geographic positioning and mapping. Therefore, in order to obtain higher positioning accuracy, aerial triangulation must be performed. Infrared scanners use linear array swing-scanning imaging, and their error sources and error patterns are different from those of linear array push-scanning imaging sensors. There is a lack of suitable directional slice interpolation methods. Existing aerial triangulation methods have low accuracy and precision, and require a large amount of computation, which brings a series of problems to the subsequent aerial triangulation modeling of linear array swing-scanning images.

[0007] (2) The swing-sweep large field of view infrared scanner adopts linear swing-sweep imaging. Due to the novel data acquisition method, its imaging principle and data processing method are different from those of traditional sensors. The swing-sweep imaging method also causes various errors of the scanner and their influences are different from those of push-broom sensors. The existing technology lacks a special study on its aerial triangulation adjustment method, lacks the combination of the scanner's imaging model and error sources, and lacks the analytical aerial triangulation measurement. There is a lack of calculation of the scanner's camera installation error and POS system installation error parameters, lacks research on the scanner's imaging model and error sources, and the internal error law of the swing-sweep strip, and lacks an adaptive directional slice interpolation model. There is a lack of designing experiments during the aerial triangulation regional network adjustment, and the object coordinates of the calibrated connection points are not used as control points, weighted virtual observation values, and initial values ​​in the aerial triangulation adjustment. The positioning accuracy is low, and the correctness and feasibility of the aerial triangulation solution are poor.

[0008] (3) The existing aerial triangulation method for the UAV sweeping large field of view infrared scanner does not establish the imaging model of the large field of view infrared scanner, the relevant coordinate system and its transformation involved in the imaging process, and the strict geometric imaging model of the infrared scanner; it lacks the directional slice interpolation model of the improved linear array push-broom camera, and cannot establish an adaptive directional slice interpolation model based on the imaging model and error law of the infrared scanner. The model does not use the error law of the residual of the swing-sweep image point after calibration, does not use the Douglas algorithm to find the inflection point of the residual curve, does not extract the directional slice of the scanning behavior where the image point at the inflection point is located, lacks the use of the adaptive directional slice interpolation model for aerial triangulation regional network adjustment, does not construct the regional network adjustment process of the infrared scanner, the accuracy evaluation method of the swing-sweep image, and the correction method of the swing-sweep image. When performing adaptive aerial triangulation, the calibrated connection points are not used as weighted virtual observation values ​​to control the aerial triangulation network. The aerial triangulation free network has an overall offset in the y direction. The accuracy of the infrared scanner aerial triangulation target solution is low, does not meet the requirements of high-precision mapping, and has poor security in promotion and application. Summary of the Invention

[0009] The swing-sweep large-field-of-view infrared scanner has great social and economic benefits. By interpreting its high-resolution swing-sweep images, it can realize resource surveying, environmental monitoring, and crop disease and pest control. In addition, it also has certain application value in photoelectric detection and aerial reconnaissance. The scanner uses a linear array CCD sensor for imaging to achieve multi-spectral, large-field-of-view, high-resolution ground observation. With the advantages of large-field-of-view, high-resolution imaging and huge social and economic benefits, the mathematical model and adjustment method of the swing-sweep imaging system developed in this application are of great value, and the adjustment of the adaptive directional slice interpolation of the developed swing-sweep imaging scanner is of great significance. Combined with the strict imaging model of the swing-sweep large-field-of-view infrared scanner, a mathematical model is constructed for it, and an adaptive directional slice interpolation method is proposed. The correctness of the method is verified by analyzing the experimental results before and after aerial triangulation. The aerial triangulation method has high correctness and accuracy, and low computational complexity. It provides an important basis for the subsequent aerial triangulation modeling of linear array swing-sweep images and has great application value.

[0010] To achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0011] Preferably, the scanner's camera installation error and POS system installation error parameters are first calculated. On this basis, an adaptive directional slice interpolation model is established based on the scanner's imaging model, error sources, and internal error patterns of the swing-scanning strip. Then, by analyzing the internal residual curve of the calibrated swing-scanning strip, the Douglas algorithm is used to determine the directional slice images that need to be extracted within a swing-scanning strip. The directional slice images are adjusted in the aerial triangulation block to improve the ground positioning accuracy of the swing-scanning images and meet the requirements of high-precision mapping.

[0012] 1) Establish an imaging model for a large field of view infrared scanner, establish the relevant coordinate systems and their transformations involved in the imaging process, and establish a strict geometric imaging model for the infrared scanner;

[0013] 2) An improved directional slice interpolation model for linear push-broom cameras was established based on the imaging model and error patterns of infrared scanners. This model employed the Douglas algorithm to find the inflection point of the residual curve according to the error patterns of the image point residuals of the calibrated push-broom image. The scanning behavior of the image point at the inflection point was extracted as the directional slice. The attitude error correction at the directional slice moment was considered unknown, while the attitude error corrections at other moments were interpolated using Lagrange polynomials.

[0014] 3) Adopting an adaptive directional slice interpolation model for aerial triangulation block adjustment, we established a block adjustment process for infrared scanners, an accuracy evaluation method for swing-scanned images, and a correction method for swing-scanned images. For the aerial triangulation part, we established three sets of comparison models: the object space coordinates of the calibrated tie points were used as control points, initial values, and weighted virtual observations in the aerial triangulation adjustment. The calibrated and triangulated results were compared in terms of image point residuals, relative accuracy, and absolute accuracy.

[0015] 4) After calibration, the image point residuals reached 8 pixels in the x-direction and 6 pixels in the y-direction, and the residuals in the x-direction were irregularly curved. After triangulation, the image point residuals were reduced to within 4 pixels, and the random errors in the x-direction were eliminated. The relative accuracy after calibration was within 4 pixels in the x-direction and 1 pixel in the y-direction. The relative accuracy after triangulation was within 2 pixels in the x-direction. When performing adaptive triangulation, it is necessary to use the calibrated connection points as weighted virtual observations to control the triangulation network to prevent the overall offset of the triangulation free network in the y-direction.

[0016] Preferably, the adjustment model of the swing-scan large-field-of-view infrared scanner: before performing aerial triangulation, the system error parameters of the infrared swing-scan scanner must be calibrated first.

[0017] The error sources that affect the positioning accuracy of the infrared pendulum scanner include: internal camera errors, including principal point offset error, focal length measurement error, and lens distortion error; external installation errors, including camera placement error, non-vertical angle deviation of the pitch and roll axes of the scanning stabilization platform, and POS system placement error; synchronization error, including the scanner's sensor synchronizer error, and the time synchronization error between the POS system and data recording; laboratory calibration includes internal orientation elements and camera lens distortion error content. Field calibration is performed after laboratory calibration, and the calibration parameters include camera placement error and POS system placement error.

[0018] Preferably, the calibration of system errors: the field calibration system parameters include: 1) camera placement error: the offset vector of the camera projection center in the scanning coordinate system, the conversion matrix between the camera coordinate system and the scanning coordinate system; 2) POS system placement error: the offset vector of the stable platform rotation center in the IMU coordinate system, the conversion matrix between the base coordinate system and the IMU coordinate system;

[0019] (1) Field geometry calibration model

[0020] The rotation matrix composed of the three attitude angles recorded by the IMU realizes the transformation from the IMU coordinate system to the navigation coordinate system. Let R IMU is the transformation matrix from the IMU coordinate system to the tangent plane coordinate system, then:

[0021] R IMU =R m R g R imu Formula 1

[0022] Where: R imu is the transformation matrix from the IMU coordinate system to the navigation coordinate system; R g is the transformation matrix from the navigation coordinate system to the geocentric coordinate system; R m is the transformation matrix from the geocentric coordinate system to the tangent plane coordinate system;

[0023] According to the imaging model of the infrared pendulum scanner, the geometric calibration model of this scanner is obtained:

[0024]

[0025] Where: (X, Y, Z) is the coordinate of a point on the image in the tangent plane coordinate system; (X GPS , Y GPS , Z GPS ) is the GPS phase center coordinate in the tangent plane coordinate system after error correction; R IMU is the transformation matrix from the IMU coordinate system to the tangent plane coordinate system; (u GPS , v GPS , w GPS ) is the offset vector of the origin of the base coordinate system in the IMU coordinate system; is the transformation matrix from the base coordinate system to the IMU coordinate system; R scan is the transformation matrix from the scanning coordinate system to the base coordinate system; (u sensor , v sensor , w sensor ) is the offset vector of the camera's projection center in the scanning coordinate system; is the transformation matrix from the camera coordinate system to the scanning coordinate system;

[0026] Time-independent (u GPS, v GPS , w GPS ), (u sensor , v sensor , w sensor ), The parameters of the scanner to be calibrated.

[0027] Preferably, the left and right swing images are calibrated separately: when the camera rotates around the longitudinal axis to perform swing imaging, the TDICCD integral directions of the left and right swing images are different, and the camera installation angles of the left and right swing images are different. During calibration, the camera installation angles of the left and right swing images are calibrated separately:

[0028]

[0029] Where: Camera mounting angle for left-side and stone-side images.

[0030] Preferably, the aerial triangulation modeling of the swing-sweep infrared scanner is carried out: the camera placement error and POS system placement error obtained from the field calibration are used as known quantities in the aerial triangulation adjustment. The imaging mode of the swing-sweep infrared scanner is a linear array swing-sweep mode, and each scanning line is relatively independent and has its own exterior orientation element.

[0031] Based on the internal error curve of each swing-scanning strip, the appropriate line array image is adaptively selected as the orientation image. The regional block adjustment of the swing-scanning large-field-of-view infrared scanner is performed after the scanner's systematic error parameters are calibrated. First, the scanner's systematic error parameters need to be calibrated: the ground three-dimensional coordinates of the connection points are obtained by forward intersection based on the POS data, the measured image point coordinates, and the initial values ​​of the systematic error parameters. The scanner's systematic error parameters are calibrated using the geometric calibration model. Suppose the coordinates of a certain image point measured on the image before calibration are (x0, y0), the projection coordinates of the image point after calibration are (x1, y1), and the image point residual before and after calibration is (△x, △y), then:

[0032]

[0033] The residuals △x of the image points involved in the calibration in each swing scanning strip are sorted according to the scanning lines to obtain the residual curve of the image points in each swing scanning strip, and the curve is smoothed;

[0034] The inflection point of the smoothed residual curve of the image point is found. The method for finding the inflection point of the curve is as follows: for a curve S, connect the two end points of S with a straight line L, calculate the distance from other points on S to L, find the maximum value among the included distances, and compare it with the critical value D. If the maximum distance is less than the critical value, all points on S except the endpoint are discarded; if the maximum distance is greater than the critical value, the point corresponding to the maximum distance is set as the new endpoint, and S is divided into two segments.

[0035] The scan line image corresponding to the image point at the inflection point on the residual curve is regarded as the directional slice to be extracted;

[0036] The flight trajectory is fitted using Lagrange polynomials: the unknown number in the aerial triangulation adjustment is the attitude correction number of the directional image. The attitude correction number of the directional image can be obtained by interpolating the attitude correction number of the directional image using Lagrange polynomials according to time to obtain the attitude correction number of other linear array images;

[0037] Using the third-order Lagrange polynomial, the attitude correction number of the four directional slices is used to describe the attitude correction number of the linear array image corresponding to a point P on the ground:

[0038]

[0039] Where:

[0040] roll p 、pitch p , yaw p is the attitude correction number of the linear array image corresponding to point P; roll j 、pitch j , yaw j is the pose correction number of the oriented image;

[0041] Substitute the imaging model of the infrared scanner to obtain the block adjustment error equation of the adaptive directional slice interpolation model:

[0042]

[0043] Where:

[0044] A P , B, E are coefficient matrices; x is the unknown vector of orientation patch attitude correction; x G is the virtual observation value vector of the control point; P, P G is the corresponding weight matrix.

[0045] Preferably, the block adjustment process of the infrared scanner is as follows: the adjustment of the infrared pendulum scanner is performed after the scanner is calibrated, and the calibrated systematic error parameters are used as known quantities in the aerial triangulation adjustment. The specific steps are as follows:

[0046] Step 1: Data acquisition: Select eight flight strip images, POS data, and field control point coordinates from two flight sorties;

[0047] Step 2: Connect point matching: Use SIFT operator to extract and match feature points in the overlapping areas of the image;

[0048] Step 3: Control point selection: measure the image point coordinates according to the control point positions;

[0049] Step 4: Initial value calculation: Based on the POS data and the initial value of the system error parameter, the connection point image point is intersected forward to obtain the object space coordinates of the connection point;

[0050] Step 5: Calibration: Solve the system error parameters based on the geometric calibration model;

[0051] Step 6: Adjustment: Bring the calibrated system error parameters and the object coordinates of the connection points into the imaging model, and use the adaptive directional slice interpolation model to solve the attitude correction number to obtain the attitude correction error value of each scanning strip.

[0052] Preferably, POS data preprocessing: for each scan strip image, the POS data records include only 330, the stable platform data include 1660, the exposure time data includes the imaging time of each scan line, the three original data records are aligned according to time, and the POS data and the stable platform data are interpolated according to the exposure time to obtain the GPS phase center coordinates, IMU attitude angle, roll angle and pitch angle of each scan line image, and obtain the attitude data of each scan line;

[0053] 1) Time alignment

[0054] The alignment of POS data and stable platform data depends on UTC time. The UTC time in the stable platform needs to be processed before use: let the internal time of the stable platform be time, the UTC time be utc, and the UTC time required for alignment with POS data be utc_real. The UTC time processing formula is:

[0055] utc_real=round(utc-time)+time Equation 7

[0056] Where: round() is approximate rounding; utc_real is the time record used to align with the UTC time in the POS data after processing; the stable platform data and the exposure time can be aligned by the internal time stamp;

[0057] 2) Posture interpolation

[0058] Interpolate the UTC time, frame roll angle, and pitch angle data in the stable platform based on the scan line exposure time and the internal time stamp of the stable platform;

[0059] The UTC time in the interpolated stable platform data is processed and matched with the UTC time in the POS, and the GPS antenna latitude, longitude, elevation, and IMU X-axis, Y-axis, and Z-axis rotation angle data in the POS are interpolated;

[0060] Preferably, image feature point extraction and matching:

[0061] 1) Extract feature points

[0062] The method uses features that remain unchanged under rotation and scaling, including the establishment of multi-scale space, detection of extreme points in scale space, positioning of key points, determination of the main direction of key points based on neighborhood features, and feature descriptor generation steps;

[0063] 2) Matching feature points

[0064] The descriptor of the feature point includes 128 dimensions, and the nearest points are searched in the high-dimensional space. The kd tree realizes the matching of feature points by establishing data index. The computational complexity of image feature point matching is proportional to the number of images that need to be matched. The matching of feature points needs to be performed on two images. High-precision POS data and stable platform scanning data are used, and the approximate overlapping relationship between the scanning strips is established using the initially obtained data to achieve image pairing.

[0065] Preferably, the initial value of the object space coordinates of the connection point is solved: the POS data of each row of the image, the scan data, the calibrated system parameters and the initial values ​​of the internal and external orientation elements of the image are interpolated to calculate the object space coordinates of the connection point, and the initial value of the object space coordinates of the connection point is calculated based on the measured coordinates of the same-name image points.

[0066] Preferably, the infrared swing-scan image is corrected by using the DEM to correct the swing-scan image to the ground according to the swing angle, pitch angle, trajectory and attitude used during the swing-scan imaging;

[0067] Image rectification is a geometric transformation between two two-dimensional images. The core is to determine the geometric relationship between the two images. Based on the coordinates of the image points in the corrected image, the coordinates of the image points in the original image are calculated from the relationship between the corrected image and the original image. The DEM is used to first determine the ground range corresponding to the image. Then, based on the imaging model, the coordinates of each three-dimensional ground point are projected onto the image to obtain the image point coordinates of the ground point in the original image. When projecting the ground point coordinates onto the image, it is necessary to iterate the scanning line.

[0068] Using TDI CCD for charge integration, the image movement speed matches the scanner's sweep speed, and the distance between the initial scan line and the optimal scan line is approximately estimated based on this constraint.

[0069] 1) Best scan line search:

[0070] ①First, set the middle scanning line of the sweeping strip image to the initial scanning line lastLine, and the initial scanning line at the beginning of the second iteration to the scanning line found last time;

[0071] ②According to the initial scan line, the corresponding posture data is obtained, and the coordinates of the image point are calculated according to the three-dimensional coordinates of the ground point. The calculation formula is as follows:

[0072]

[0073] Where: (X i , Y i , Z i ) is the object space three-dimensional coordinate; (X si , Y si , Z si ) is the projection center coordinate of the initial scan line lastLine; λ is the scale factor; R is the calculated rotation matrix of the initial scan line; (x, y) is the coordinate of the image point on the original image; f is the principal distance;

[0074] ③ The y value of the pixel coordinate calculated by ② is regarded as the distance between the scan line where it is located and the optimal scan line. If the y value is less than the given critical value, the obtained lastLine is the optimal scan line and the algorithm ends; otherwise, go to ④;

[0075] ④ Assume that the interval of a CCD detection pixel is scale, and calculate the new scan line based on y and scale:

[0076] lastLine=lastLine+y / scale Formula 9

[0077] Use the obtained lastLine as the new initial scan line and continue with step ②;

[0078] 2) Grayscale value calculation using inverse distance weighting: The grayscale value of the image point is obtained based on the image point coordinates and the scan line where it is located calculated by the projection algorithm. The grayscale value of the image point is interpolated using the inverse distance weighting method;

[0079] According to the position of the obtained image point P in the original image, the distance between point P and the eight surrounding image points is calculated, and then the image point grayscale value of point P is obtained using the inverse distance weighted method.

[0080] Compared with existing technologies, the innovations and advantages of this application are:

[0081] (1) Swing imaging can achieve large field of view imaging by adjusting the swing angle under the condition of meeting other technical indicators, and obtain continuous swing strips with overlapping degrees during the flight of the aircraft. The swing large field of view infrared scanner has great social and economic benefits. By interpreting its high-resolution swing image, it can realize resource exploration, environmental monitoring, and crop disease and pest control. In addition, it also has certain application value in photoelectric detection and aerial reconnaissance. The scanner uses a linear array CCD sensor for imaging to achieve multi-spectral, large field of view, and high-resolution ground observation. With the advantages of large field of view, high-resolution imaging and huge social and economic benefits, the mathematical model and adjustment method of the swing imaging system developed in this application are of great value, and the adjustment of the adaptive directional slice interpolation of the developed swing imaging scanner is of great significance. Combined with the strict imaging model of the swing-scan wide-field-of-view infrared scanner, a mathematical model was constructed for it, and an adaptive directional slice interpolation method was proposed. The correctness of the method was verified by analyzing the experimental results before and after aerial triangulation. The aerial triangulation method has high correctness and accuracy, and low computational complexity. It provides an important foundation for the subsequent aerial triangulation modeling of linear array swing-scanning images and has great application value.

[0082] (2) The swing-scan large field of view infrared scanner of the present application adopts linear array swing-scan imaging, and its data acquisition method is relatively novel. Its imaging principle and data processing method are different from those of traditional sensors. In addition, the swing-scan imaging method also causes various errors of the scanner and their influences are different from those of push-scan sensors. Therefore, the present application combines the imaging model and error sources of the scanner to perform analytical aerial triangulation. According to the previous simulation experiments and field geometry calibration experiments, the scanner's camera installation error and POS system installation error parameters are obtained. On this basis, an adaptive directional slice interpolation model is proposed for the imaging model of the scanner, the error sources, and the internal error laws of the swing-scan strips. By analyzing the internal residual curve of the swing-scan strips after calibration, the Douglas algorithm is used to determine the directional slice image that needs to be extracted in a swing-scan strip. Three groups of experiments were designed during the aerial triangulation regional network adjustment: the object coordinates of the calibrated connection points were used as control points, weighted virtual observation values, and initial values ​​were used in the aerial triangulation adjustment. The effects of the three methods on positioning accuracy were compared and compared with the results after calibration to verify the correctness and feasibility of the adaptive directional slice interpolation model of this application.

[0083] (3) This application first calculates the scanner's camera installation error and POS system installation error parameters, establishes an imaging model for a large-field-of-view infrared scanner, and then, based on the scanner's imaging model, error sources, and the internal error laws of the sweeping strip, establishes an adaptive directional slice interpolation model. Then, by analyzing the internal residual curve of the calibrated sweeping strip, the Douglas algorithm is used to determine the directional slice images that need to be extracted within a sweeping strip. The adaptive directional slice interpolation model is used to perform aerial triangulation block adjustment, constructing the infrared scanner's block adjustment process, the swinging image accuracy evaluation method, and the swinging image correction method. Three sets of experiments are designed for the aerial triangulation part. The method has good stability, high aerial triangulation perception efficiency, fast target detection speed, high accuracy, improves the ground positioning accuracy of the sweeping image, meets the needs of high-precision mapping, and is conducive to the popularization and civilianization of large-field-of-view infrared scanners. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is the residual curve of the image point after calibration.

[0085] Figure 2 It is a schematic diagram of the method for finding the inflection point of the smoothed image residual curve.

[0086] Figure 3 It is a schematic diagram of adaptive selection of directional slices.

[0087] Figure 4 This is the adjustment flow chart of the infrared swing scanner.

[0088] Figure 5 This is a schematic diagram of grayscale value interpolation using the inverse distance weighted method.

[0089] Figure 6 It is a schematic diagram of the residual error of the image point of test I.

[0090] Figure 7 This is a comparison chart of the results of the adaptive directional slice model and the traditional directional slice model.

[0091] Figure 8 This is a schematic diagram of the stitching of adjacent strips of images in the same flight strip after calibration and aerial triangulation.

[0092] Figure 9 This is a schematic diagram of the stitching of adjacent strip images in different flight strips after calibration and aerial triangulation.

[0093] Figure 10 It is a schematic diagram of the image square accuracy of the test after calibration. DETAILED DESCRIPTION

[0094] Below, in conjunction with the accompanying drawings, the technical solution of the intelligent aerial triangulation method of the drone swing-sweeping large-field-of-view infrared scanner provided in this application is further described so that those skilled in the art can better understand this application and implement it.

[0095] With the development of photogrammetry and the requirement for large-field-of-view imaging, the swing-scan imaging method has shown its unique advantages. Currently, there are many studies on the aerial triangulation methods of push-broom sensors, but there are relatively few studies on the aerial triangulation methods of linear array swing-scan sensors. Therefore, the research on the regional network adjustment of linear array swing-scan sensors is particularly important. This application is based on the swing-scan images of large-field-of-view infrared scanners, improves the regional network adjustment method for linear array push-broom sensors, and establishes an aerial triangulation adjustment method of adaptive directional slice interpolation based on the imaging characteristics and imaging principles of infrared cameras, so as to improve the ground positioning accuracy of the swing-scan images, meet the requirements of high-precision mapping, and realize the popularization and civilianization of large-field-of-view infrared scanners.

[0096] 1. Establish the imaging model of the large field of view infrared scanner, the relevant coordinate system and its transformation involved in the imaging process, and the strict geometric imaging model of the infrared scanner;

[0097] 2. Improve the directional slice interpolation model for linear array push-broom cameras. Based on the imaging model and error law of infrared scanners, an adaptive directional slice interpolation model is established. Based on the error law of the residual error of the calibrated swing-scan image point, the model uses the Douglas algorithm to find the inflection point of the residual curve. The scanning line where the image point at the inflection point is located is extracted as the directional slice. The attitude error correction value at the directional slice moment is regarded as an unknown number, and the attitude error correction values ​​at other moments are obtained using Lagrange polynomial interpolation.

[0098] 3. An adaptive directional slice interpolation model was used for aerial triangulation block adjustment. This included constructing a block adjustment process for infrared scanners, an accuracy evaluation method for swing-scanned images, and a correction method for swing-scanned images. Three sets of experiments were designed for the aerial triangulation process: The object-space coordinates of the calibrated tie points were used as control points, initial values, and weighted virtual observations for the aerial triangulation adjustment. The calibrated and triangulated experimental results were compared from three perspectives: image point residuals, relative accuracy, and absolute accuracy. Relative accuracy refers to the splicing accuracy of adjacent swing-scanned strips, including the relative accuracy of adjacent strips within the same flight strip and between different flight strips. Absolute accuracy was evaluated using both object-space and image-space verification methods.

[0099] 4. After calibration, the image point residuals reached 8 pixels in the x-direction and 6 pixels in the y-direction, and the x-direction residuals exhibited an irregular curve. After triangulation, the image point residuals were reduced to within 4 pixels, and random errors in the x-direction were eliminated. The relative accuracy after calibration was within 4 pixels in the x-direction and 1 pixel in the y-direction. The relative accuracy after triangulation was within 2 pixels in the x-direction. The stitching accuracy of adjacent strips in the same flight strip was higher than that of adjacent strips in different flight strips. The mean error of absolute accuracy after calibration was 3-4 pixels. The absolute accuracy of triangulation adjustments using tie points as control points and weighted virtual observations improved compared to after calibration. However, the absolute accuracy in the y-direction of triangulation using tie points as initial values ​​decreased. When performing adaptive triangulation, it is necessary to use the calibrated tie points as weighted virtual observations to control the triangulation network to prevent an overall shift in the y-direction of the triangulation free network.

[0100] 1. Adjustment model of the swing-sweep large field of view infrared scanner

[0101] In the field of aerospace photogrammetry, the accuracy of exterior orientation elements directly determines the accuracy of uncontrolled image positioning, and attitude measurement error is a significant error source affecting the accuracy of direct image positioning. The accuracy of uncontrolled image positioning is determined by the accuracy of the image's exterior orientation elements, making attitude measurement error a significant error source affecting positioning accuracy. Due to IMU attitude measurement errors, the data recorded by the POS system cannot meet the requirements of high-precision mapping. Therefore, analytical aerial triangulation is essential to improve ground positioning accuracy.

[0102] The wide-field-of-view infrared multispectral scanner utilizes a full-body swing-sweep imaging system based on a long linear array of detectors. During flight, the scanner can swing up to 50 degrees. Due to the large field of view and the high accuracy required, the scanner utilizes a TDI method with pitch compensation. Because the imaging system has numerous geometric parameters and these parameters are highly correlated, the infrared swing-sweep scanner's system error parameters must be calibrated prior to aerial triangulation.

[0103] The error sources that affect the positioning accuracy of the infrared pendulum scanner include: internal camera errors, including principal point offset error, focal length measurement error, and lens distortion error; external installation errors, including camera placement error, non-vertical angle deviation of the pitch and roll axes of the scanning stabilization platform, and POS system placement error; synchronization error, including the scanner's sensor synchronizer error, and the time synchronization error between the POS system and data recording; laboratory calibration includes internal orientation elements and camera lens distortion error content. Field calibration is performed after laboratory calibration, and the calibration parameters include camera placement error and POS system placement error.

[0104] (1) Calibration of system errors

[0105] The parameters of the field calibration system include: 1) Camera placement error: the offset vector of the camera projection center in the scanning coordinate system, and the transformation matrix between the camera coordinate system and the scanning coordinate system; 2) POS system placement error: the offset vector of the stable platform rotation center in the IMU coordinate system, and the transformation matrix between the base coordinate system and the IMU coordinate system.

[0106] (1) Field geometry calibration model

[0107] The rotation matrix composed of the three attitude angles recorded by the IMU realizes the transformation from the IMU coordinate system to the navigation coordinate system. Let R IMU is the transformation matrix from the IMU coordinate system to the tangent plane coordinate system, then:

[0108] R IMU =R m RgB imu Formula 1

[0109] Where: R imu is the transformation matrix from the IMU coordinate system to the navigation coordinate system; R g is the transformation matrix from the navigation coordinate system to the geocentric coordinate system; R m is the transformation matrix from the geocentric coordinate system to the tangent plane coordinate system;

[0110] According to the imaging model of the infrared pendulum scanner, the geometric calibration model of this scanner is obtained:

[0111]

[0112] Where: (X, Y, Z) is the coordinate of a point on the image in the tangent plane coordinate system; (X GPS , Y GPS , Z GPS ) is the GPS phase center coordinate in the tangent plane coordinate system after error correction; R IMU is the transformation matrix from the IMU coordinate system to the tangent plane coordinate system; (u GPS , v GPS , w GPS ) is the offset vector of the origin of the base coordinate system in the IMU coordinate system; is the transformation matrix from the base coordinate system to the IMU coordinate system; R scan is the transformation matrix from the scanning coordinate system to the base coordinate system; (u sensor , v sensor , w sensor ) is the offset vector of the camera's projection center in the scanning coordinate system; is the transformation matrix from the camera coordinate system to the scanning coordinate system;

[0113] Time-independent (u GPS , v GPS, w GPS ), (u sensor , v sensor , w sensor ), The parameters of the scanner to be calibrated.

[0114] (2) Separate calibration of left and right scanning images

[0115] When the camera rotates around the longitudinal axis to scan and image, the TDI CCD integration directions of the left and right swings are different. The camera installation angles of the left and right swing images are different. During calibration, the camera installation angles of the left and right swing images are calibrated separately:

[0116]

[0117] Where: Camera mounting angle for left-side and stone-side images.

[0118] (2) Aerial triangulation modeling of the swing-sweep infrared scanner

[0119] The regional block adjustment of the swing-sweep large-field-of-view infrared scanner is performed after the system parameters are calibrated. Therefore, the camera placement error and POS system placement error calibrated in the field are used as known quantities in the aerial triangulation adjustment. The imaging mode of the swing-sweep infrared scanner is linear array swing-sweep. Each scanning line is relatively independent and has its own exterior orientation elements.

[0120] In the directional slice interpolation model, the selection of directional slices is carried out according to a certain time interval, and the errors caused by irregular jitter of the flight trajectory and aircraft attitude are not taken into account. The scanner of this application has an extremely large field of view, a complex imaging model, different error patterns of different sweeping strips, and different amplitudes of attitude change rates within a specific time period. Therefore, it is not possible to simply extract several line array images at the same time interval to make directional images. It is necessary to adaptively select appropriate line array images as directional images based on the internal error curve of each sweeping strip.

[0121] The block adjustment of a swing-scanning wide-field infrared scanner is performed after the scanner's systematic error parameters (POS system placement error and camera placement error) are calibrated. First, the scanner's systematic error parameters need to be calibrated: the ground three-dimensional coordinates of the connection points are obtained by forward intersection based on the POS data, the measured image point coordinates, and the initial values ​​of the systematic error parameters. The scanner's systematic error parameters are calibrated using the geometric calibration model. Suppose the coordinates of a certain image point measured on the image before calibration are (x0, y0), the projection coordinates of the image point after calibration are (x1, y1), and the image point residuals before and after calibration are (△x, △y), then:

[0122]

[0123] The residual △x of the image points involved in the calibration in each swing scanning strip is sorted according to the scanning line, and the residual curve of the image points in each swing scanning strip is obtained, and the curve is smoothed, as shown in the following example: Figure 1 As shown;

[0124] The inflection point of the smoothed residual curve of the image point is found. The method for finding the inflection point of the curve is: for a curve S, connect the two endpoints of S with a straight line L, calculate the distance from other points on S (except the two endpoints) to L, find the maximum value among the included distances, and compare it with the critical value D. If the maximum distance is less than the critical value, all points on S except the endpoints are discarded; if the maximum distance is greater than the critical value, the point corresponding to the maximum distance is set as the new endpoint, S is divided into two segments, and the method is continued in these two segments; the algorithm steps are as follows Figure 2 As shown:

[0125] The scan line image corresponding to the image point at the inflection point on the residual curve is regarded as the directional slice to be extracted, such as Figure 3 As shown;

[0126] The flight trajectory is fitted using Lagrange polynomials: the unknown number in the aerial triangulation adjustment is the attitude correction number of the directional image. The attitude correction number of the directional image can be obtained by interpolating the attitude correction number of the directional image using Lagrange polynomials according to time to obtain the attitude correction number of other linear array images;

[0127] Using the third-order Lagrange polynomial, the attitude correction number of the four directional slices is used to describe the attitude correction number of the linear array image corresponding to a point P on the ground:

[0128]

[0129] Where:

[0130] roll p 、pitch p , yaw p is the attitude correction number of the linear array image corresponding to point P; roll j 、pitch j , yaw j is the pose correction number of the oriented image;

[0131] Substitute the imaging model of the infrared scanner to obtain the block adjustment error equation of the adaptive directional slice interpolation model:

[0132]

[0133] Where:

[0134] A P , B, E are coefficient matrices; x is the unknown vector of orientation patch attitude correction; x Gis the virtual observation value vector of the control point; P, P G is the corresponding weight matrix.

[0135] II. Block Adjustment of Swinging Large Field of View Infrared Scanner

[0136] (1) Block adjustment process of infrared scanner

[0137] The adjustment of the infrared pendulum scanner is carried out after the scanner is calibrated. The calibrated system error parameters are used as known quantities in the aerial triangulation adjustment. The process is as follows: Figure 4 The specific steps are as follows:

[0138] Step 1: Data acquisition: Select eight flight strip images, POS data, and field control point coordinates from two flight sorties;

[0139] Step 2: Connect point matching: Use SIFT operator to extract and match feature points in the overlapping areas of the image;

[0140] Step 3: Control point selection: measure the image point coordinates according to the control point positions;

[0141] Step 4: Initial value calculation: Based on the POS data and the initial value of the system error parameter, the connection point image point is intersected forward to obtain the object space coordinates of the connection point;

[0142] Step 5: Calibration: Solve the system error parameters based on the geometric calibration model;

[0143] Step 6: Adjustment: Bring the calibrated system error parameters and the object coordinates of the connection points into the imaging model, and use the adaptive directional slice interpolation model to solve the attitude correction number to obtain the attitude correction error value of each scanning strip.

[0144] (1) POS data preprocessing

[0145] For each scan strip image, the POS data records only include 330 and the stable platform data include 1660. The exposure time data contains the imaging time of each scan line. The three original data records are aligned according to time, and the POS data and stable platform data are interpolated according to the exposure time to obtain the GPS phase center coordinates, IMU attitude angle, roll angle and pitch angle of each scan line image, and obtain the attitude data of each scan line.

[0146] 1) Time alignment

[0147] The alignment of POS data and stable platform data depends on UTC time. The UTC time in the stable platform needs to be processed before use: let the internal time of the stable platform be time, the UTC time be utc, and the UTC time required for alignment with POS data be utc_real. The UTC time processing formula is:

[0148] utc_real=round(utc-time)+time Equation 7

[0149] Where: round() is approximate rounding; utc_real is the time record used to align with the UTC time in the POS data after processing; the stable platform data and the exposure time can be aligned by the internal time stamp;

[0150] 2) Posture interpolation

[0151] Interpolate the UTC time, frame roll angle, and pitch angle data in the stable platform based on the scan line exposure time and the internal time stamp of the stable platform;

[0152] The UTC time in the interpolated stable platform data is processed and matched with the UTC time in the POS, and the GPS antenna latitude, longitude, elevation, and IMU X-axis, Y-axis, and Z-axis rotation angle data in the POS are interpolated.

[0153] (2) Image feature point extraction and matching

[0154] 1. Extract feature points

[0155] The method adopts features that remain unchanged under rotation and scale scaling, including the establishment of multi-scale space, detection of extreme points in scale space, positioning of key points, determination of the main direction of key points based on neighborhood features, and feature descriptor generation steps.

[0156] 2. Matching feature points

[0157] The descriptor of the feature point includes 128 dimensions. The nearest point is searched in the high-dimensional space (128 dimensions). The kd tree realizes the matching of feature points by establishing data index. The computational complexity of image feature point matching is proportional to the number of images that need to be matched. The matching of feature points needs to be performed on two images. High-precision POS data and stable platform scanning data are used. The approximate overlapping relationship between the scanning strips is established using the initially obtained data to achieve image pairing.

[0158] (3) Calculation of the initial value of the object coordinates of the connection point

[0159] The object space coordinates of the connection points are calculated by forward intersection based on the interpolated POS data of each row of the image, the scan data, the calibrated system parameters, and the initial values ​​of the internal and external orientation elements of the image. The initial values ​​of the object space coordinates of the connection points are calculated based on the measured coordinates of the image points with the same name.

[0160] (2) Correction of infrared scanning images

[0161] According to the sweeping angle, pitch angle, trajectory and attitude used during the sweeping imaging, the sweeping image is corrected to the ground using DEM.

[0162] Image rectification is a geometric transformation between two two-dimensional images. The core is to determine the geometric relationship between the two images. According to the image point coordinates of the corrected image, the image point coordinates of the original image are calculated based on the relationship between the corrected image and the original image. The DEM is used to first determine the ground range corresponding to the image. Then, according to the imaging model, the coordinates of each three-dimensional ground point are projected onto the image to obtain the image point coordinates of the ground point in the original image. When projecting the ground point coordinates onto the image, it is necessary to iterate the scanning line.

[0163] A TDI CCD is used for charge integration. The image moving speed is matched with the scanning speed of the scanner. Based on this constraint, the distance between the initial scanning line and the optimal scanning line is approximately estimated.

[0164] 1) Best scan line search:

[0165] ①First, set the middle scanning line of the sweeping strip image to the initial scanning line lastLine, and the initial scanning line at the beginning of the second iteration to the scanning line found last time;

[0166] ②According to the initial scan line, the corresponding posture data is obtained, and the coordinates of the image point are calculated according to the three-dimensional coordinates of the ground point. The calculation formula is as follows:

[0167]

[0168] Where: (X i , Y i , Z i ) is the object space three-dimensional coordinate; (X si , Y si , Z si ) is the projection center coordinate of the initial scan line lastLine; λ is the scale factor; R is the calculated rotation matrix of the initial scan line; (x, y) is the coordinate of the image point on the original image; f is the principal distance;

[0169] ③ The y value of the pixel coordinate calculated by ② is regarded as the distance between the scan line where it is located and the optimal scan line. If the y value is less than the given critical value, the obtained lastLine is the optimal scan line and the algorithm ends; otherwise, go to ④;

[0170] ④ Assume that the interval of a CCD detection pixel is scale, and calculate the new scan line based on y and scale:

[0171] lastLine=lastLine+y / scale Formula 9

[0172] Use the obtained lastLine as the new initial scan line and continue with step ②;

[0173] 2) Grayscale value calculation using inverse distance weighting: The grayscale value of the image point is obtained based on the image point coordinates and the scan line where it is located calculated by the projection algorithm. The grayscale value of the image point is interpolated using the inverse distance weighting method;

[0174] like Figure 5 As shown in FIG, according to the position of the obtained image point P in the original image, the distance between point P and the eight surrounding image points is calculated, and then the image point grayscale value of point P is obtained by using the inverse distance weighting method.

[0175] 3. Experimental Results and Analysis

[0176] Traditional bundle block adjustment calculations require linearizing the collinearity equations, solving partial derivatives, performing matrix operations, and iterating to obtain the results. The entire process is computationally intensive. This application uses the CeresSolver secondary development library, an open source library for solving nonlinear optimization problems.

[0177] The scanned image strips of the scanner have the characteristics of small heading overlap and large lateral overlap. To ensure that they include enough connection points, images with large heading overlap in the upwind flight direction are selected to meet the three-degree overlap condition of aerial triangulation. Therefore, 8 flight strips from two flights with a total of 1,380 images are selected for the experiment. The experimental results are illustrated using the BO5 band as an example.

[0178] (1) Image point residual

[0179] 1) Image point residual after calibration

[0180] The residual errors of the image points after calibration are sorted according to the scanning lines to obtain the residual error curve of the image points.

[0181] ① After calibration, the residual error of the image point of the scanning strip image reaches 8 pixels in the x direction and 6 pixels in the y direction;

[0182] ② Comparing the curves in the x-direction and y-direction: the residual in the x-direction takes on a curved shape as the number of scan lines increases, and the curve is not a parabola, but a higher-order irregular curve; the residual in the y-direction does not change much with the increase of scan lines, and there is an obvious systematic error in the x-direction.

[0183] ③ Comparing the residual plots (a) and (b) for different scanning strips, the residual curves in the x-direction vary in shape and lack any regularity. The residual curves in the y-direction do not show much variation with the increase in scan lines. This indicates that the error in the x-direction is random and varies across different scanning strips.

[0184] 2. Image point residual after aerotriangulation using the adaptive directional slice interpolation model

[0185] After calibration, the residuals of the image points are still large, and the point accuracy cannot meet higher mapping requirements, especially in the x-direction, where there are still obvious errors. It is necessary to perform regional block adjustment after calibrating the system parameters.

[0186] The calibrated residual curve is smoothed, and the Douglas algorithm is used to find the inflection point of the curve. The scan lines corresponding to the inflection point are considered directional slices. The attitude error corrections for the scan lines between directional slices are obtained by interpolating the adjacent directional slices. Due to the complex shape of the x-direction residual curve of the calibrated image point, a third-order Lagrange polynomial is used for interpolation.

[0187] The calibrated system error parameters are introduced into the imaging model, and the adaptive directional slice interpolation model is used for the experiment. The experiment is divided into three groups: I uses the calibrated object coordinates of the tie points as control points for the aerial triangulation; II uses the calibrated object coordinates of the tie points as initial values ​​for the aerial triangulation; III uses the calibrated object coordinates of the tie points as weighted virtual observations for the aerial triangulation. The image point residuals after the aerial triangulation are sorted by scan line, and the results are as follows: Figure 6 shown.

[0188] ① In Experiment I, the residual errors of the image points in the sweep strip image after aerial triangulation were within 6 pixels in the x-direction and approximately 5 pixels in the y-direction. In Experiment II, the residual errors of the image points in the sweep strip image were within 4 pixels in both the x- and y-directions. In Experiment III, the residual errors of the image points in the sweep strip image were approximately 4 pixels in both the x- and y-directions. Comparing the results after calibration, it can be seen that positioning accuracy in both the x- and y-directions has been improved after aerial triangulation. Using the calibrated tie points as initial values ​​for the aerial triangulation adjustment yields even higher accuracy.

[0189] ② Comparing the x- and y-direction curves after aerial triangulation: The residual error ranges in the x- and y-directions are essentially the same, with neither exhibiting significant variations with the addition of scan lines. For different sweeping stripes, the x-direction residual no longer exhibits an irregular distribution. Comparing the results after calibration reveals that the x-direction residual curve has been significantly improved after aerial triangulation, eliminating errors to a certain extent and improving accuracy.

[0190] 3. Comparison between traditional directional slice model and adaptive directional slice model

[0191] The traditional directional slice interpolation model is used to conduct block adjustment experiments, and the image point residuals after the experiment are compared with the results after calibration and adaptive directional slice model. Figure 7 shown.

[0192] from Figure 7 It can be concluded that the residual curve in the x-direction after calibration is irregular and the residual is large; after applying the traditional directional slice interpolation model for aerial triangulation experiments, the residual in the x-direction is reduced, but the residual still presents an irregular curve shape, indicating that the error in the x-direction has not been eliminated; the residual in the x-direction after adaptive directional slices is significantly smaller and no longer presents an irregular curve shape, indicating that the method of adaptively selecting directional slice images for regional block adjustment based on the residual curve after calibration is correct and effective.

[0193] (2) Relative accuracy analysis

[0194] The relative accuracy analysis of infrared pan-scan images is based on the geometric stitching accuracy of adjacent pan-scan strips with overlapping areas. This application uses images of adjacent strips within the same flight band and adjacent strips from different flight bands for relative accuracy testing: the two images are automatically matched and then transformed (affine or quadratic polynomial) to remove gross errors before accuracy assessment.

[0195] Taking the BO5 band as an example, the adjacent sweep strip images of the same flight band after calibration and triangulation, and the adjacent sweep strip images of different flight bands with overlapping areas are corrected to obtain the mosaic image of adjacent strips as shown below: Figure 8 、 Figure 9 shown.

[0196] Depend on Figure 8 、 Figure 9 It can be concluded that after calibration, there is obvious misalignment in the splicing of adjacent sweep strips in the same flight strip and adjacent sweep strips in different flight strips, and the misalignment in the x-direction is obvious, which is consistent with the significantly larger residual in the x-direction. After aerial triangulation, the misalignment phenomenon in the splicing of adjacent sweep strips in the same flight strip and adjacent sweep strips in different flight strips has been significantly improved.

[0197] Since the relative accuracy of the infrared scanner is defined based on the stitching accuracy of adjacent strips, it is necessary to automatically match the corrected adjacent strip images and perform relative accuracy evaluation after eliminating gross errors.

[0198] From the relative accuracy table of adjacent strips in the same flight band, we can know that:

[0199] ① After calibration, the relative accuracy of adjacent sweep strips in the same flight band reaches 4 pixels in the x-direction and about 1 pixel in the y-direction. The relative accuracy in the y-direction is better than that in the x-direction, which is consistent with the significantly larger residual error in the x-direction after calibration.

[0200] ② After aerial triangulation, the relative accuracy of adjacent strips in the same flight strip is improved to within 2 pixels in the x-direction, while the y-direction changes little, remaining at about 1 pixel. This indicates that the relative accuracy of adjacent sweep strips in the same flight strip in the x-direction has been significantly improved after aerial triangulation.

[0201] ③ By comparing Experiment I, Experiment I and Experiment IIII, it can be concluded that the relative accuracy of using the calibrated object space coordinates of the tie points as initial values ​​for aerial triangulation is slightly higher than the relative accuracy of using the calibrated object space coordinates of the tie points as control points and weighted virtual observations.

[0202] The relative accuracy of adjacent strips across different flight strips shows that after calibration, the relative accuracy of adjacent panning strips across different flight strips reached 4 pixels in both the x and y directions. After aerial triangulation, the relative accuracy in the x direction improved to within 2 pixels, and the relative accuracy in the y direction to within 3 pixels. This indicates that the relative accuracy in both the x and y directions has improved after aerial triangulation. The accuracy results for Experiments I, II, and III were similar.

[0203] The relative accuracy table above shows that the relative accuracy in the y-direction of adjacent strips within the same flight strip is higher than that of strips in different flight strips before and after triangulation. This is due to the mismatch in the scan lines of the overlapping areas of two adjacent images in different flight strips (the larger the sweep angle, the greater the image deformation).

[0204] The scan lines on which the same-name image points in the overlapping region of adjacent strips within the same flight strip are located are essentially the same, meaning that the sweep angles at the time of imaging for these same-name image points are essentially the same. However, the scan lines on which the same-name image points in the overlapping region of adjacent strips between different flight strips are located vary significantly, meaning that the sweep angles at the time of imaging vary significantly. Larger sweep angles lead to greater image deformation. Therefore, due to image deformation, the relative accuracy in the y-direction of adjacent strip images between different flight strips is not as high as that of adjacent strip images within the same flight strip.

[0205] (3) Absolute accuracy analysis

[0206] The absolute accuracy analysis of infrared scanning images is to geometrically correct the scanning images, automatically match the corrected images with DOM (resolution of 0.1m), and then evaluate the accuracy after eliminating gross errors.

[0207] Taking the B05 band as an example, the system error parameters obtained after calibration and the attitude error correction number of Experiment II are brought into the imaging model, the image is geometrically corrected, and the corrected image is automatically matched with the DOM.

[0208] The corrected images after calibration, after aerotriangulation Experiments I, and II were fitted with the DOM, yielding fitting error curves. The absolute accuracy of the calibrated image is approximately 7 pixels; the absolute accuracy of Experiment I is lower than that of the calibration image, exceeding 10 pixels; the absolute accuracy of Experiments I and II is slightly better than that of the calibration image, at approximately 5 pixels. This indicates that using the calibrated tie point object coordinates as the initial values ​​for aerotriangulation is not a very reasonable approach; the aerotriangulation network is too free, resulting in poor absolute accuracy. Using the calibrated tie point object coordinates as weighted virtual observations yields better absolute accuracy.

[0209] The above evaluation method requires the use of DEM in image geometry correction, and when the sweep angle is large, DEM will produce large errors. Therefore, in order to more accurately determine the absolute accuracy of the sweep image before and after aerial triangulation, the image space accuracy verification method is adopted: according to the system parameters and imaging model, the control point is projected from the object space to the image space, and within a sweep strip, the difference between the projection point coordinates and the image coordinates is calculated, which is the absolute accuracy. This method can introduce as few external errors as possible and is more accurate than the object space verification method. The image space accuracy of the images after calibration, Test I, Test II, and Test III is as follows Figure 10 .

[0210] Depend on Figure 10 It can be seen that the accuracy in the x-direction is significantly improved after aerial triangulation; the x- and y-direction accuracies of Experiments I and M are higher than those after calibration. Although Experiment II can improve the accuracy in the x-direction, the accuracy in the y-direction is worse. This shows that completely releasing the object-space coordinates of the tie points after calibration from participating in aerial triangulation causes an overall offset in the y-direction coordinates. Therefore, using the tie points as control points or weighted virtual observations in the aerial triangulation calculation can obtain better absolute accuracy, which is more in line with the actual situation.

Claims

1. The intelligent aerial triangulation method of the UAV swing-scanning infrared scanner with a large field of view is characterized by: First, the scanner's camera installation error and POS system installation error parameters are calculated. Based on this, an adaptive directional slice interpolation model is established based on the scanner's imaging model, error sources, and the internal error patterns of the swing-scanning strip. Then, by analyzing the internal residual curves of the calibrated swing-scanning strip, the Douglas algorithm is used to determine the directional slice images that need to be extracted within a swing-scanning strip. This is then adjusted within the aerotriangulation block to improve the ground positioning accuracy of the swing-scanning images and meet the requirements of high-precision mapping. 1) Establish an imaging model for a large field of view infrared scanner, establish the relevant coordinate systems and their transformations involved in the imaging process, and establish a strict geometric imaging model for the infrared scanner; 2) An improved directional slice interpolation model for linear push-broom cameras was established based on the imaging model and error patterns of infrared scanners. This model employed the Douglas algorithm to find the inflection point of the residual curve according to the error patterns of the image point residuals of the calibrated push-broom image. The scanning behavior of the image point at the inflection point was extracted as the directional slice. The attitude error correction at the directional slice moment was considered unknown, while the attitude error corrections at other moments were interpolated using Lagrange polynomials. 3) Adopting an adaptive directional slice interpolation model for aerial triangulation block adjustment, we established a block adjustment process for infrared scanners, an accuracy evaluation method for swing-scanned images, and a correction method for swing-scanned images. For the aerial triangulation part, we established three sets of comparison models: the object space coordinates of the calibrated tie points were used as control points, initial values, and weighted virtual observations in the aerial triangulation adjustment. The calibrated and triangulated results were compared in terms of image point residuals, relative accuracy, and absolute accuracy. 4) After calibration, the image point residuals reached 8 pixels in the x-direction and 6 pixels in the y-direction, and the residuals in the x-direction were irregularly curved. After triangulation, the image point residuals were reduced to within 4 pixels, and the random errors in the x-direction were eliminated. The relative accuracy after calibration was within 4 pixels in the x-direction and 1 pixel in the y-direction. The relative accuracy after triangulation was within 2 pixels in the x-direction. When performing adaptive triangulation, it is necessary to use the calibrated connection points as weighted virtual observations to control the triangulation network to prevent the overall offset of the triangulation free network in the y-direction.

2. The intelligent aerial triangulation method for a UAV with a large field of view infrared scanner according to claim 1 is characterized in that: Adjustment model of the swing-scan large-field-of-view infrared scanner: Before performing aerial triangulation, the system error parameters of the infrared swing-scan scanner must be calibrated first. The error sources that affect the positioning accuracy of the infrared pendulum scanner include: internal camera errors, including principal point offset error, focal length measurement error, and lens distortion error; external installation errors, including camera placement error, non-vertical angle deviation of the pitch and roll axes of the scanning stabilization platform, and POS system placement error; synchronization error, including the scanner's sensor synchronizer error, and the time synchronization error between the POS system and data recording; laboratory calibration includes internal orientation elements and camera lens distortion error content. Field calibration is performed after laboratory calibration, and the calibration parameters include camera placement error and POS system placement error.

3. The intelligent aerial triangulation method for a UAV with a large field of view infrared scanner according to claim 1 is characterized in that: System error calibration: Field calibration system parameters include: 1) Camera placement error: the offset vector of the camera projection center in the scanning coordinate system, and the conversion matrix between the camera coordinate system and the scanning coordinate system; 2) POS system placement error: the offset vector of the stable platform rotation center in the IMU coordinate system, and the conversion matrix between the base coordinate system and the IMU coordinate system; (1) Field geometry calibration model The rotation matrix composed of the three attitude angles recorded by the IMU realizes the transformation from the IMU coordinate system to the navigation coordinate system. Let R IMU is the transformation matrix from the IMU coordinate system to the tangent plane coordinate system, then: R IMU =R m R g R imu Formula 1 Where: R imu is the transformation matrix from the IMU coordinate system to the navigation coordinate system; R g is the transformation matrix from the navigation coordinate system to the geocentric coordinate system; R m is the transformation matrix from the geocentric coordinate system to the tangent plane coordinate system; According to the imaging model of the infrared pendulum scanner, the geometric calibration model of this scanner is obtained: Where: (X, Y, Z) is the coordinate of a point on the image in the tangent plane coordinate system; (X GPS , Y GPS , Z GPS ) is the GPS phase center coordinate in the tangent plane coordinate system after error correction; R IMU is the transformation matrix from the IMU coordinate system to the tangent plane coordinate system; (u GPS , v GPS , w GPS ) is the offset vector of the origin of the base coordinate system in the IMU coordinate system; is the transformation matrix from the base coordinate system to the IMU coordinate system; R scan is the transformation matrix from the scanning coordinate system to the base coordinate system; (u sensor , v sensor , w sensor ) is the offset vector of the camera's projection center in the scanning coordinate system; is the transformation matrix from the camera coordinate system to the scanning coordinate system; Time-independent (u GPS , v GPS , w GPS ), (u sensor , v sensor , w sensor ), The parameters of the scanner to be calibrated.

4. The intelligent aerial triangulation method for a UAV with a large field of view infrared scanner according to claim 1 is characterized in that: Separate calibration of left and right swing images: When the camera rotates around the longitudinal axis to sweep and image, the TDI CCD integration direction of the left and right swing images is different, and the camera installation angle of the left and right swing images is different. During calibration, the camera installation angles of the left and right swing images are calibrated separately: Where: Camera mounting angle for left-side and stone-side images.

5. The intelligent aerial triangulation method for a UAV with a large field of view infrared scanner according to claim 1 is characterized in that: Aerial triangulation modeling of the swing-sweep infrared scanner: The camera placement errors and POS system placement errors obtained from field calibration are used as known quantities in aerial triangulation adjustment. The swing-sweep infrared scanner uses a linear array scanning method, and each scan line is relatively independent and has its own exterior orientation elements. Based on the internal error curve of each swing-scanning strip, the appropriate line array image is adaptively selected as the orientation image. The regional block adjustment of the swing-scanning large-field-of-view infrared scanner is performed after the scanner's systematic error parameters are calibrated. First, the scanner's systematic error parameters need to be calibrated: the ground three-dimensional coordinates of the connection points are obtained by forward intersection based on the POS data, the measured image point coordinates, and the initial values ​​of the systematic error parameters. The scanner's systematic error parameters are calibrated using the geometric calibration model. Suppose the coordinates of a certain image point measured on the image before calibration are (x0, y0), the projection coordinates of the image point after calibration are (x1, y1), and the image point residual before and after calibration is (△x, △y), then: The residuals △x of the image points involved in the calibration in each swing scanning strip are sorted according to the scanning lines to obtain the residual curve of the image points in each swing scanning strip, and the curve is smoothed; The inflection point of the smoothed residual curve of the image point is found. The method for finding the inflection point of the curve is as follows: for a curve S, connect the two end points of S with a straight line L, calculate the distance from other points on S to L, find the maximum value among the included distances, and compare it with the critical value D. If the maximum distance is less than the critical value, all points on S except the endpoint are discarded; if the maximum distance is greater than the critical value, the point corresponding to the maximum distance is set as the new endpoint, and S is divided into two segments. The scan line image corresponding to the image point at the inflection point on the residual curve is regarded as the directional slice to be extracted; The flight trajectory is fitted using Lagrange polynomials: the unknown number in the aerial triangulation adjustment is the attitude correction number of the directional image. The attitude correction number of the directional image can be obtained by interpolating the attitude correction number of the directional image using Lagrange polynomials according to time to obtain the attitude correction number of other linear array images; Using the third-order Lagrange polynomial, the attitude correction number of the four directional slices is used to describe the attitude correction number of the linear array image corresponding to a point P on the ground: Where: roll p 、pitch p , yaw p is the attitude correction number of the linear array image corresponding to point P; roll j 、pitch j , yaw j is the pose correction number of the oriented image; Substitute the imaging model of the infrared scanner to obtain the block adjustment error equation of the adaptive directional slice interpolation model: Where: A P , B, E are coefficient matrices; x is the unknown vector of orientation patch attitude correction; x G is the virtual observation value vector of the control point; P, P G is the corresponding weight matrix.

6. The intelligent aerial triangulation method for a UAV with a large field of view infrared scanner according to claim 1 is characterized in that: Block adjustment process for infrared scanners: Adjustment of infrared pendulum scanners is performed after the scanner is calibrated. The calibrated systematic error parameters are used as known quantities in aerial triangulation. The specific steps are as follows: Step 1: Data acquisition: Select eight flight strip images, POS data, and field control point coordinates from two flight sorties; Step 2: Connect point matching: Use SIFT operator to extract and match feature points in the overlapping areas of the image; Step 3: Control point selection: measure the image point coordinates according to the control point positions; Step 4: Initial value calculation: Based on the POS data and the initial value of the system error parameter, the connection point image point is intersected forward to obtain the object space coordinates of the connection point; Step 5: Calibration: Solve the system error parameters based on the geometric calibration model; Step 6: Adjustment: Bring the calibrated system error parameters and the object coordinates of the connection points into the imaging model, and use the adaptive directional slice interpolation model to solve the attitude correction number to obtain the attitude correction error value of each scanning strip.

7. The intelligent aerial triangulation method of the UAV swing-sweep large field of view infrared scanner according to claim 1, characterized in that POS Data preprocessing: For each scanned image strip, only 330 POS data records are included, and 1660 stable platform data records are included. The exposure time data contains the imaging time of each scan line. The three raw data records are aligned according to time, and the POS data and stable platform data are interpolated according to the exposure time to obtain the GPS phase center coordinates, IMU attitude angle, and roll and pitch angles of each scan line image, thereby obtaining the attitude data of each scan line. 1) Time alignment The alignment of POS data and stable platform data depends on UTC time. The UTC time in the stable platform needs to be processed before use: let the internal time of the stable platform be time, the UTC time be utc, and the UTC time required for alignment with POS data be utc_real. The UTC time processing formula is: utc_real=round(utc-time)+time Equation 7 Where: round() is approximate rounding; utc_real is the time record used to align with the UTC time in the POS data after processing; the stable platform data and the exposure time can be aligned by the internal time stamp; 2) Posture interpolation Interpolate the UTC time, frame roll angle, and pitch angle data in the stable platform based on the scan line exposure time and the internal time stamp of the stable platform; The UTC time in the interpolated stable platform data is processed and matched with the UTC time in the POS, and the GPS antenna latitude, longitude, elevation, and IMU X-axis, Y-axis, and Z-axis rotation angle data in the POS are interpolated.

8. The intelligent aerial triangulation method for a UAV with a large field of view infrared scanner according to claim 1 is characterized in that: Image feature point extraction and matching: 1) Extract feature points The method uses features that remain unchanged under rotation and scaling, including the establishment of multi-scale space, detection of extreme points in scale space, positioning of key points, determination of the main direction of key points based on neighborhood features, and feature descriptor generation steps; 2) Matching feature points The descriptor of the feature point includes 128 dimensions, and the nearest points are searched in the high-dimensional space. The kd tree realizes the matching of feature points by establishing data index. The computational complexity of image feature point matching is proportional to the number of images that need to be matched. The matching of feature points needs to be performed on two images. High-precision POS data and stable platform scanning data are used, and the approximate overlapping relationship between the scanning strips is established using the initially obtained data to achieve image pairing.

9. The intelligent aerial triangulation method for a UAV with a large field of view infrared scanner according to claim 1, characterized in that: Calculation of the initial value of the object space coordinates of the connection point: The object space coordinates of the connection point are calculated by performing forward intersection based on the interpolated POS data of each row of the image, the scan data, the calibrated system parameters, and the initial values ​​of the internal and external orientation elements of the image. The initial value of the object space coordinates of the connection point is calculated based on the measured coordinates of the image points with the same name.

10. The intelligent aerial triangulation method for a UAV swing-sweep large-field-of-view infrared scanner according to claim 1, characterized in that: Correction of infrared scanning images: Based on the scanning angle, pitch angle, trajectory and attitude used during scanning imaging, the DEM is used to correct the scanning images to the ground; Image rectification is a geometric transformation between two two-dimensional images. The core is to determine the geometric relationship between the two images. Based on the coordinates of the image points in the corrected image, the coordinates of the image points in the original image are calculated from the relationship between the corrected image and the original image. The DEM is used to first determine the ground range corresponding to the image. Then, based on the imaging model, the coordinates of each three-dimensional ground point are projected onto the image to obtain the image point coordinates of the ground point in the original image. When projecting the ground point coordinates onto the image, it is necessary to iterate the scanning line. Using TDI CCD for charge integration, the image movement speed matches the scanner's sweep speed, and the distance between the initial scan line and the optimal scan line is approximately estimated based on this constraint. 1) Best scan line search: ①First, set the middle scanning line of the sweeping strip image to the initial scanning line lastLine, and the initial scanning line at the beginning of the second iteration to the scanning line found last time; ②According to the initial scan line, the corresponding posture data is obtained, and the coordinates of the image point are calculated according to the three-dimensional coordinates of the ground point. The calculation formula is as follows: Where: (X i , Y i , Z i ) is the object space three-dimensional coordinate; (X si , Y si , Z si ) is the projection center coordinate of the initial scan line lastLine; λ is the scale factor; R is the calculated rotation matrix of the initial scan line; (x, y) is the coordinate of the image point on the original image; f is the principal distance; ③ The y value of the pixel coordinate calculated by ② is regarded as the distance between the scan line where it is located and the optimal scan line. If the y value is less than the given critical value, the obtained lastLine is the optimal scan line and the algorithm ends; otherwise, go to ④; ④ Assume that the interval of a CCD detection pixel is scale, and calculate the new scan line based on y and scale: lastLine=lastLine+y*scale Formula 9 Use the obtained lastLine as the new initial scan line and continue with step ②; 2) Grayscale value calculation using inverse distance weighting: The grayscale value of the image point is obtained based on the image point coordinates and the scan line where it is located calculated by the projection algorithm. The grayscale value of the image point is interpolated using the inverse distance weighting method; According to the position of the obtained image point P in the original image, the distance between point P and the eight surrounding image points is calculated, and then the image point grayscale value of point P is obtained using the inverse distance weighted method.

Images