Unmanned aerial vehicle inspection image registration and alignment method based on deep learning

Abstract

The invention discloses an unmanned aerial vehicle inspection image registration and alignment method based on deep learning, and belongs to the technical field of image processing. Comprising the steps of video acquisition and segmentation processing, semantic-space-time alignment and synchronization subsequence selection, content enhancement and alignment frame pair extraction, multi-source key point extraction and fusion, cross-video high-quality feature matching and geometric transformation estimation and registration, and solves the technical problem of high-precision frame alignment and matching in cross-view and asynchronous image sequences. The invention provides a semantic-spatio-temporal combined image sequence alignment method, which realizes high-robustness alignment of cross-sequence image frames, and enables matching to be more inclined to frame pairs in time proximity, thereby effectively inhibiting semantic interference of cross-time drift, enhancing physical rationality of a matching path, and improving matching accuracy. The robust alignment capability under asymmetric sampling, speed change and visual angle deviation in the real flight process is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a deep learning-based UAV inspection image registration and alignment method. Background Art

[0002] With the widespread application of drone imagery in surveying and mapping, agriculture, security, and disaster monitoring, how to perform high-precision matching and registration of video images from different perspectives (such as binocular cameras) or different time sequences has become a major technical challenge in the field of visual processing.

[0003] The current mainstream methods mainly rely on image registration frameworks based on local features (such as SIFT and ORB), or introduce deep learning methods such as SuperPoint and LightGlue to achieve more robust feature matching.

[0004] However, existing technologies mostly focus on the spatial consistency between single image pairs. When faced with image sequence matching tasks with time delay, perspective offset or content difference, they lack effective modeling of inter-frame temporal information, resulting in increased matching errors or alignment failures.

[0005] Conventional technologies currently ignore the crucial influence of temporal consistency constraints, such as inter-frame acquisition time and motion trends, on registration accuracy. Existing solutions struggle to effectively distinguish between semantically similar but temporally misaligned frames and temporally consistent but semantically weak frames, leading to registration mismatches and reduced overall system stability. Summary of the Invention

[0006] The purpose of this invention is to provide a deep learning-based UAV inspection image registration and alignment method, which solves the technical problem of high-precision frame alignment and matching in cross-view and asynchronous image sequences.

[0007] To achieve the above object, the present invention adopts the following technical solutions: A deep learning-based UAV inspection image registration and alignment method includes the following steps: Step 1: Use dual cameras to synchronously capture video sequences and posture information. Use timestamps and track points to perform frame sampling and segmentation on the video sequences to construct a preliminary set of video pairs. Step 2: Retrieve the video frame pairs from the video pair set, extract the semantic features and geometric matching errors of the video frames, construct a semantic + spatiotemporal joint scoring matrix, and use dynamic programming to perform frame-level alignment to obtain content-synchronized video frame pairs. Step 3: Retrieve the video frame pairs and perform enhancement processing on the aligned video pairs; Step 4: For the enhanced video, use the SuperPoint algorithm and SIFT algorithm to extract key points and descriptors respectively, integrate multi-source features through the weighted fusion function, and output a fused descriptor set; Step 5: Use the LightGlue algorithm to build a cross-video attention matching model, formulate strategies for bidirectional consistency, adaptive confidence, and geometric consistency, and screen out high-quality matching point pairs; Step 6: For high-quality matching point pairs, evaluate the matching quality and spatial distribution, select the corresponding transformation model, optimize the evaluation results using the MAGSAC++ method, and complete the geometric registration and alignment of the video.

[0008] Preferably, when executing step 2, the following steps are specifically included: Step 2-1: Retrieve a video frame pair from the video pair set; Step 2-2: Extract global semantic feature vectors from video frame pairs and perform preliminary geometric matching error processing; When processing semantic feature vectors, semantic features are extracted by convolutional neural networks; When processing geometric matching errors, the geometric error is calculated by the key point matching reprojection error based on the LightGlue algorithm; Step 2-3: Based on the semantic similarity, time decay weight and geometric consistency index, construct the semantic + spatiotemporal joint scoring matrix and calculate the joint score of each pair of video frames: ; in, For joint scoring, is the semantic similarity score between the i-th frame of video A and the j-th frame of video B, where video A and video B are a pair of video frames; Represents the preliminary geometric consistency score between two frames; represents the temporal consistency weight function; , represents the weight parameter between semantic and geometric scores; ; in, Represent the timestamps of video frame i and video frame j respectively, represents the time tolerance parameter; , the larger the value, the closer the time; Step 2-4: Based on the results of the joint scoring matrix, a dynamic programming algorithm is used to search for an ascending path to obtain the aligned subsequence with the highest joint score, and the semantically and spatiotemporally aligned video frame pairs are output, i.e., the content-synchronized video frame pairs.

[0009] Preferably, when executing step 4, the following steps are specifically included: Step 4-1: For the content-synchronized video frame pairs, use the SuperPoint algorithm and SIFT algorithm to extract key points and corresponding descriptors, which are recorded as SuperPoint descriptors and SIFT descriptors respectively; Step 4-2: Perform vector normalization on the extracted SuperPoint descriptor and SIFT descriptor respectively; Step 4-3: Construct a weighted fusion function to fuse the normalized SuperPoint descriptor, SIFT descriptor and key point space encoding into a feature descriptor, and output a fused feature descriptor set.

[0010] Preferably, when executing step 5, the following steps are specifically included: Step 5-1: Retrieve the fused descriptor set and use the Transformer-based LightGlue model to calculate the matching similarity matrix between key points; Step 5-2: According to the bidirectional maximum matching principle, select the best matching key point pairs; Step 5-3: Based on the statistical characteristics of the matching scores, the confidence threshold is adaptively set and pairs below the confidence threshold are eliminated; Step 5-4: Build an estimated geometric transformation model, further screen matching pairs that meet the geometric consistency requirements, and build and output a set of quality matching point pairs.

[0011] Preferably, when executing step 6, the following steps are specifically included: Step 6-1: For high-quality matching point pairs, evaluate the number of matching point pairs and spatial distribution uniformity, and select the transformation estimation method based on the preset threshold; Step 6-2: Use the MAGSAC++ algorithm to eliminate false matches and determine the set of geometrically consistent interior points; Step 6-3: Use a robust optimization method to estimate the view transformation matrix on the interior point set; Step 6-4: Apply the estimated transformation matrix to perform geometric transformation on the video frame to complete pixel-level registration and alignment of the video frame.

[0012] The deep learning-based drone inspection image registration and alignment method described in the present invention solves the technical problem of high-precision frame alignment and matching in cross-view and asynchronous image sequences. The present invention provides a semantic-spatiotemporal joint image sequence alignment method. By constructing a joint scoring matrix containing semantic similarity, geometric consistency score and temporal consistency weight function, high-robust alignment of cross-sequence image frames is achieved. The temporal consistency weight function automatically adjusts the frame pair score according to the index difference or timestamp difference between frames, making the matching more inclined to temporally adjacent frame pairs, thereby effectively suppressing the semantic interference of cross-temporal drift, enhancing the physical rationality of the matching path, and significantly improving the robust alignment capability under asymmetric sampling, speed changes and perspective deviations during real flight. The present invention also considers the triple factors of joint semantics, geometry and time, and can accurately identify image pairs with similar semantic content, temporal synchronization and geometric consistency in the image sequence, significantly improving the subsequent feature matching and image registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the main flow chart of the present invention; Figure 2 is a flow chart of step 2 of the present invention; Figure 3 is a flow chart of step 4 of the present invention; Figure 4 It is a flow chart of step 5 of the present invention Figure 5 It is a flow chart of step 6 of the present invention. DETAILED DESCRIPTION

[0014] Depend on Figure 1-Figure 5 The figure shows a deep learning-based drone inspection image registration and alignment method. In this embodiment, the video data collected by the dual cameras on the drone will be transmitted to an FPGA controller, and the FPGA controller will send the video data to the background server for processing through wireless communication modules such as WiFi.

[0015] The process includes the following steps: Step 1: Use dual cameras to synchronously capture video sequences and posture information. Use timestamps and track points to perform frame sampling and segmentation on the video sequences to construct a preliminary set of video pairs. The video sequence captured by the dual cameras is the left and right visible light camera video streams, which are represented as , the attitude information is the attitude and position data output by the IMU / GPS / SLAM device on the drone, expressed as ; In this embodiment, the two video streams are timed with a unified global clock t, and each frame in the video sequence is accompanied by a timestamp, which is recorded as , respectively, A and B represent the video data of two cameras.

[0016] When performing timestamp alignment, the main focus is to align the time bases of the frames of the two video data channels to eliminate asynchronous jitter on the acquisition end. Specifically, for each frame index k, find the closest peer frame index. The calculation formula is as follows: ; in, They represent the timestamps of the frames captured by the two cameras respectively; j is the peer frame index synchronized with k.

[0017] When sampling frames of a video sequence, either timed sampling or key frame detection is used. If timed sampling is used, frames are extracted at fixed time intervals. In this embodiment, one frame may be extracted every second. If key frame detection is used, frame extraction is triggered only when the posture or visual changes dramatically. The specific triggering condition is as follows: or ; in, Represents the body posture quaternion, is the image matrix, represents the Frobenius norm; These are all preset thresholds.

[0018] When segmenting the route, the video is divided into several independent segments according to the preset inspection waypoints or mission stages to facilitate subsequent batch processing. In this embodiment, GPS track points are used to mark the segment boundaries.

[0019] The video pair set is represented as , Indicates each paragraph, The number of video frame pairs extracted and successfully matched in segment s.

[0020] Step 2: Retrieve the video frame pairs from the video pair set, extract the semantic features and geometric matching errors of the video frames, construct a semantic + spatiotemporal joint scoring matrix, and use dynamic programming to perform frame-level alignment to obtain content-synchronized video frame pairs. When executing step 2, the specific steps include: Step 2-1: Retrieve a video frame pair from the video pair set; Step 2-2: Extract global semantic feature vectors from video frame pairs and perform preliminary geometric matching error processing; When processing semantic feature vectors, semantic features are extracted by convolutional neural networks; When processing geometric matching errors, the geometric error is calculated by the key point matching reprojection error based on the LightGlue algorithm; For each pair of original image frames , extracting semantic features and estimating preliminary geometric consistency, respectively.

[0021] In this embodiment, semantic feature extraction specifically uses a lightweight convolutional neural network (such as MobileNet, EfficientNet) to extract a frame-level global semantic feature vector.

[0022] The semantic similarity calculation is specifically to use cosine similarity to measure the similarity between the semantic features of two frames to obtain a semantic similarity matrix.

[0023] The geometric error estimation specifically uses the LightGlue algorithm to perform preliminary feature matching on the two frames of images and calculate the key point reprojection error (such as the root mean square error RMSE) as a reflection of the geometric consistency score.

[0024] Step 2-3: Based on the semantic similarity, time decay weight and geometric consistency index, construct the semantic + spatiotemporal joint scoring matrix and calculate the joint score of each pair of video frames: ; in, For joint scoring, is the semantic similarity score between the i-th frame of video A and the j-th frame of video B, where video A and video B are a pair of video frames; Represents the preliminary geometric consistency score between two frames; represents the temporal consistency weight function; , represents the weight parameter between semantic and geometric scores; ; in, Represent the timestamps of video frame i and video frame j respectively, represents the time tolerance parameter; , the larger the value, the closer the time; In this embodiment, the time consistency weight function It can effectively solve the problem of incorrectly pairing frames that "look alike" visually when relying solely on image semantics or geometric matching, even if they should not appear together in time. It solves the problem of frame-level registration of multi-source UAV images under complex conditions such as semantic similarity, geometric offset and temporal asynchrony.

[0025] Step 2-4: Based on the results of the joint scoring matrix, a dynamic programming algorithm is used to search for an ascending path to obtain the aligned subsequence with the highest joint score, and the semantically and spatiotemporally aligned video frame pairs are output, i.e., the content-synchronized video frame pairs.

[0026] In this embodiment, the dynamic programming algorithm may adopt a weighted dynamic time warping (DTW) or soft-DTW algorithm. First, the dynamic programming algorithm searches for an ascending path from the upper left corner to the lower right corner in an M×N scoring matrix such that the sum of the joint scores on the path is maximized. The formula of the dynamic programming algorithm is as follows: ; Among them, Path is the path.

[0027] Then, a sliding window operation is applied on the path to calculate the local average score, and the continuous subsequence with the highest score is selected to output the final aligned frame pair set, i.e. The calculation formula for calculating the local average score is as follows: ; Where L represents the length of the selected subsequence.

[0028] The set of aligned frame pairs is represented as , Represents the final aligned frame pair.

[0029] Step 3: Retrieve the video frame pairs and perform enhancement processing on the aligned video pairs; In this embodiment, for Frame pair enhancement can include brightness adjustment, edge refinement, and texture enhancement operations, such as Retinex-based brightness normalization, U-Net / ResUNet models for detail restoration, and GuidedFilter or HistogramMatching for color and style consistency.

[0030] The final processed image frame pair set is expressed as: .

[0031] Step 4: For the enhanced video, use the SuperPoint algorithm and SIFT algorithm to extract key points and descriptors respectively, integrate multi-source features through the weighted fusion function, and output a fused descriptor set; When executing step 4, the specific steps include: Step 4-1: For the content-synchronized video frame pairs, use the SuperPoint algorithm and SIFT algorithm to extract key points and corresponding descriptors, which are recorded as SuperPoint descriptors and SIFT descriptors respectively; against Video frame pairs in , the following two algorithms are used to extract key points and corresponding descriptors: Algorithm 1: Use the SuperPoint algorithm to extract deep learning-driven dense key points and their descriptors, recorded as the SuperPoint feature descriptor set.

[0032] Algorithm 2: Use the SIFT algorithm to extract scale-invariant feature points and their traditional descriptors, which are recorded as the SIFT feature descriptor set.

[0033] Step 4-2: Perform vector normalization on the extracted SuperPoint descriptor and SIFT descriptor respectively; In this embodiment, in order to eliminate the inconsistency of scale and norm of different descriptors, the SuperPoint feature descriptor set and the SIFT feature descriptor set need to be vector normalized respectively so that they are distributed in a unified space. Specifically, the L2 normalization method or the unit sphere projection method can be used.

[0034] Step 4-3: Construct a weighted fusion function to fuse the normalized SuperPoint descriptor, SIFT descriptor and key point space encoding into a feature descriptor, and output a fused feature descriptor set.

[0035] In this embodiment, the weighted fusion function The specific formula is as follows: ; in, are weights, The spatial encoding of the two-dimensional coordinate position of the key points, is the normalized SuperPoint feature descriptor set, is the normalized SIFT feature descriptor set.

[0036] In this embodiment, the fusion parameters It should be adjusted based on the actual scenario. The specific method is: If the image texture is weak, increase the SuperPoint weight; If the geometric distortion is strong, increase the SIFT weight; If the spatial structure of the viewpoint needs to be considered during matching, spatial coding weights are introduced.

[0037] The final output fusion feature descriptor set is expressed as: ,in represents the fusion features of video frame A in the k-th pair of video frames, Represents the fusion features of video frame B in the kth pair of video frames.

[0038] Step 5: Use the LightGlue algorithm to build a cross-video attention matching model, formulate strategies for bidirectional consistency, adaptive confidence, and geometric consistency, and screen out high-quality matching point pairs; When executing step 5, the specific steps include: Step 5-1: Retrieve the fused descriptor set and use the Transformer-based LightGlue model to calculate the matching similarity matrix between key points; In the call Afterwards, it is input into the Transformer-based LightGlue matching model.

[0039] The LightGlue matching model can calculate the similarity matrix between key point descriptors through the attention mechanism across images, and output potential matching pairs and their matching confidence scores. The output results in the family are expressed as: ; in, Represents the i-th key point of image A; The jth key point of image B; , used to represent the confidence score of the corresponding match, which is output by the LightGlue matching model.

[0040] Step 5-2: Based on the bidirectional maximum matching principle, select the best matching key point pairs; and then effectively remove duplicate pairings and low-confidence interference items.

[0041] In this embodiment, in order to ensure the uniqueness and stability of the matching, the initial matching set Execute the bidirectional maximum consistency matching (MutualNearestMatching) strategy, retaining the " yes The best match for yes The matching result of the best match of "is recorded as the consistent matching set .

[0042] Step 5-3: Based on the statistical characteristics of the matching scores, the confidence threshold is adaptively set and pairs below the confidence threshold are eliminated; In this embodiment, the matching score Perform distribution statistics, set the adaptive threshold Ts, and eliminate low-confidence pairs. The specific conditions are as follows: ; In this embodiment, Ts may be set to a certain multiple of the mean value of the matching score plus the standard deviation.

[0043] Step 5-4: Build an estimated geometric transformation model, further screen matching pairs that meet the geometric consistency requirements, and build and output a set of quality matching point pairs.

[0044] In this embodiment, after adaptive screening, the point pair set On the other hand, geometric consistency verification is performed to further eliminate false matches caused by local noise or background repetition. This embodiment adopts one of the following strategies: Preliminary estimation of affine / homography transformation model; Calculate the reprojection error for each pair of matching points; Matching pairs whose reprojection error exceeds a certain threshold are eliminated.

[0045] The final retained set is recorded as the final high-quality matching point pair set: .

[0046] Step 6: For high-quality matching point pairs, evaluate the matching quality and spatial distribution, select the corresponding transformation model, optimize the evaluation results using the MAGSAC++ method, and complete the geometric registration and alignment of the video.

[0047] When executing step 6, after obtaining a high-quality matching point pair set After that, perform global transformation estimation and optimization to complete the pixel-level alignment between the two frames and output the final geometric registration result, which specifically includes the following steps: Step 6-1: For high-quality matching point pairs, evaluate the number of matching point pairs and spatial distribution uniformity, and select the transformation estimation method based on the preset threshold; Matching point set Conduct quality assessments, including: Quantity evaluation: number of matching point pairs Whether the estimation requirement is met, that is, whether it is greater than 10.

[0048] Distribution uniformity assessment: Divide the image into N×N regions and count the number of points in each region. If the point distribution is concentrated in a few regions (at least 75% of the regions contain ≥1 matching point), it indicates local redundancy and difficulty in fitting a stable transformation. Select the appropriate transformation model type based on the evaluation results: If the view angle is planar, choose a homography, if the change is more local, just use an affine transformation.

[0049] Step 6-2: Use the MAGSAC++ algorithm to eliminate false matches and determine the set of geometrically consistent interior points; Based on the results of the transformation model selected in step 6-1, the MAGSAC++ algorithm is used to perform robust parameter estimation, specifically including: Randomly sample matching point pairs; Iteratively fit the transformation matrix and calculate the reprojection error; Automatically optimize the cost function based on statistical inference of error distribution; Output the maximum consistent set of points and the final transformation matrix H or A.

[0050] The MAGSAC++ algorithm used in this embodiment can adaptively adjust fault-tolerant parameters, effectively process data containing a large amount of noise or mismatches, and has better estimation accuracy than traditional technologies.

[0051] Step 6-3: Use a robust optimization method to estimate the view transformation matrix on the interior point set; According to the set of interior points, the transformation matrix is ​​refined using the least squares method or nonlinear optimization method to output the final perspective transformation matrix, which is expressed as: , ; in, are the homogeneous coordinate points in the target image and the original image respectively.

[0052] For affine models: , .

[0053] The least square method or nonlinear optimization method are both existing technologies and will not be described in detail.

[0054] Step 6-4: Apply the estimated transformation matrix to perform geometric transformation on the video frame to complete pixel-level registration and alignment of the video frame.

[0055] In this embodiment, the transformation matrix H or A is specifically applied to the original video frame image. Perform geometric transformations to generate Aligned image : ; in, Represents affine / homography geometric mapping and interpolation operations.

[0056] The deep learning-based drone inspection image registration and alignment method described in the present invention solves the technical problem of high-precision frame alignment and matching in cross-view and asynchronous image sequences. The present invention provides a semantic-spatiotemporal joint image sequence alignment method. By constructing a joint scoring matrix containing semantic similarity, geometric consistency score and temporal consistency weight function, high-robust alignment of cross-sequence image frames is achieved. The temporal consistency weight function automatically adjusts the frame pair score according to the index difference or timestamp difference between frames, making the matching more inclined to temporally adjacent frame pairs, thereby effectively suppressing the semantic interference of cross-temporal drift, enhancing the physical rationality of the matching path, and significantly improving the robust alignment capability under asymmetric sampling, speed changes and perspective deviations during real flight. The present invention also considers the triple factors of joint semantics, geometry and time, and can accurately identify image pairs with similar semantic content, temporal synchronization and geometric consistency in the image sequence, significantly improving the subsequent feature matching and image registration accuracy.

Claims

1. A deep learning-based UAV inspection image registration and alignment method, characterized by: The steps include: Step 1: Use dual cameras to synchronously capture video sequences and posture information. Use timestamps and track points to perform frame sampling and segmentation on the video sequences to construct a preliminary set of video pairs. Step 2: Retrieve the video frame pairs from the video pair set, extract the semantic features and geometric matching errors of the video frames, construct a semantic + spatiotemporal joint scoring matrix, and use dynamic programming to perform frame-level alignment to obtain content-synchronized video frame pairs. Step 3: Retrieve the video frame pairs and perform enhancement processing on the aligned video pairs; Step 4: For the enhanced video, use the SuperPoint algorithm and SIFT algorithm to extract key points and descriptors respectively, integrate multi-source features through the weighted fusion function, and output a fused descriptor set; Step 5: Use the LightGlue algorithm to build a cross-video attention matching model, formulate strategies for bidirectional consistency, adaptive confidence, and geometric consistency, and screen out high-quality matching point pairs; Step 6: For high-quality matching point pairs, evaluate the matching quality and spatial distribution, select the corresponding transformation model, optimize the evaluation results using the MAGSAC++ method, and complete the geometric registration and alignment of the video.

2. The deep learning-based UAV inspection image registration and alignment method according to claim 1, characterized in that: When executing step 2, the specific steps include: Step 2-1: Retrieve a video frame pair from the video pair set; Step 2-2: Extract global semantic feature vectors from video frame pairs and perform preliminary geometric matching error processing; When processing semantic feature vectors, semantic features are extracted by convolutional neural networks; When processing geometric matching errors, the geometric error is calculated by the key point matching reprojection error based on the LightGlue algorithm; Step 2-3: Based on the semantic similarity, time decay weight and geometric consistency index, construct the semantic + spatiotemporal joint scoring matrix and calculate the joint score of each pair of video frames: ； in, For joint scoring, is the semantic similarity score between the i-th frame of video A and the j-th frame of video B, where video A and video B are a pair of video frames; Represents the preliminary geometric consistency score between two frames; represents the temporal consistency weight function; , represents the weight parameter between semantic and geometric scores; ； in, Represent the timestamps of video frame i and video frame j respectively, represents the time tolerance parameter; , the larger the value, the closer the time; Step 2-4: Based on the results of the joint scoring matrix, a dynamic programming algorithm is used to search for an ascending path to obtain the aligned subsequence with the highest joint score, and the semantically and spatiotemporally aligned video frame pairs are output, i.e., the content-synchronized video frame pairs.

3. The deep learning-based UAV inspection image registration and alignment method according to claim 1, characterized in that: When executing step 4, the specific steps include: Step 4-1: For the content-synchronized video frame pairs, use the SuperPoint algorithm and SIFT algorithm to extract key points and corresponding descriptors, which are recorded as SuperPoint descriptors and SIFT descriptors respectively; Step 4-2: Perform vector normalization on the extracted SuperPoint descriptor and SIFT descriptor respectively; Step 4-3: Construct a weighted fusion function to fuse the normalized SuperPoint descriptor, SIFT descriptor and key point space encoding into a feature descriptor, and output a fused feature descriptor set.

4. The deep learning-based UAV inspection image registration and alignment method according to claim 1, characterized in that: When executing step 5, the specific steps include: Step 5-1: Retrieve the fused descriptor set and use the Transformer-based LightGlue model to calculate the matching similarity matrix between key points; Step 5-2: According to the bidirectional maximum matching principle, select the best matching key point pairs; Step 5-3: Based on the statistical characteristics of the matching scores, the confidence threshold is adaptively set and pairs below the confidence threshold are eliminated; Step 5-4: Build an estimated geometric transformation model, further screen matching pairs that meet the geometric consistency requirements, and build and output a set of quality matching point pairs.

5. The deep learning-based UAV inspection image registration and alignment method according to claim 1, characterized in that: When executing step 6, the specific steps include: Step 6-1: For high-quality matching point pairs, evaluate the number of matching point pairs and spatial distribution uniformity, and select the transformation estimation method based on the preset threshold; Step 6-2: Use the MAGSAC++ algorithm to eliminate false matches and determine the set of geometrically consistent interior points; Step 6-3: Use a robust optimization method to estimate the view transformation matrix on the interior point set; Step 6-4: Apply the estimated transformation matrix to perform geometric transformation on the video frame to complete pixel-level registration and alignment of the video frame.

Images