Method and device for eliminating high-altitude image jitter based on static frame feature

Abstract

The invention discloses a method and a device for eliminating high-altitude image jitter based on a static frame feature. The method comprises the following steps of: extracting frames from an input video and extracting a feature point set of each frame; taking the kth frame as a template, counting the occurrence times of feature points through multi-frame proximity matching, and generating a static feature template; feature points are extracted frame by frame and subjected to Lloylomb ratio test with a template, and after dynamic feature points are filtered, a homography matrix is established to achieve frame alignment; detecting a difference set region of the template frame and the current frame, performing clipping compensation, and rewriting the video after image defect is repaired; and iteratively processing until the video ends. The method is superior to a traditional method depending on manual point selection and anchor point fixation in the aspects of dynamic interference suppression, automation degree and picture integrity.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle image processing, and more specifically, to a method and device for eliminating high-altitude image jitter based on a static frame feature formula. Background Art

[0002] Shooting videos with high-altitude drones is currently an important way to collect data in the field of road traffic. However, when drones are hovering in the sky to shoot videos, rotation and jitter displacement will inevitably occur due to factors such as airflow changes and body vibration, resulting in rotation and translation between the previous and next frames of the video. This will change the true coordinates of target objects such as vehicles and pedestrians, introduce additional errors to subsequent detection and tracking video extraction tasks, and make the data unusable.

[0003] At present, video stabilization technology is often used to try to solve the above problems. Existing video stabilization technology often eliminates local image jitter between adjacent video frames. For drone hovering shooting scenes, the focus is on road traffic participants within a certain range below the drone. Ideally, the drone will not move, and the coordinate changes of target objects such as vehicles and pedestrians are only caused by the movement of the objects themselves, that is, the coordinates of the background points corresponding to the same position in the real world should remain consistent in different video frames.

[0004] A video stabilization method for a drone high-altitude hovering shooting scene is disclosed in the patent with the authorization announcement number CN117575966B. The main method is to first separate the video frame by frame and perform grayscale preprocessing, take the first frame as the template frame, manually select the focus feature area in the template frame, detect the corner points in each area as anchor points, and the coordinate relationship of each anchor point constitutes the template frame space coordinate system. The affine transformation matrix between each frame of the original video and the template frame is estimated, and the lost anchor point pairs are compensated. According to the above information, a robust estimation model is constructed with the affine transformation matrix used to stabilize each frame of the image as the parameter to be optimized, and the weighted iteration of each anchor point pair is performed based on the idea of ​​robust estimation. Then the model is solved to obtain the affine transformation matrix that satisfies the root mean square error threshold for each video frame. Finally, the original video is transformed frame by frame and written into a new video. This method can eliminate the influence of the drone's own motion and achieve an ideal video stabilization effect. However, the patent relies on manual selection of anchor points, which may introduce subjective errors, and the compensation mechanism is only for anchor points that fail to track, which is easily affected by dynamic object occlusion. At the same time, it uses affine transformation to align, which may cause local information loss due to occlusion of dynamic objects, and it also requires iterative calculation of robust estimation weights, which has high computational complexity. In addition, it has strict restrictions on the number and position of anchor points, and may fail in scenes with sparse features. Summary of the invention

[0005] An object of the present invention is to solve at least the above problems and to provide at least the advantages which will be described hereinafter.

[0006] In order to achieve these purposes and other advantages according to the present invention, a method for eliminating high-altitude image jitter based on a static frame feature formula is provided, comprising: S1, obtain the video, extract the video frame, and extract the feature point set of each video frame; S2, using the extracted feature point set of the k-th video frame as a template, performing proximity matching calculation on the feature point sets of other video frames and the feature point set of the k-th video frame. When a successfully matched feature point appears in the two feature point sets, it is detected whether the feature point has an identifier. If there is no identifier, a unique identifier is generated for the feature point, and the number of occurrences of the feature point is initialized to 2. If there is an identifier, the number of occurrences of the feature point is increased by 1. S3, repeat S2 until all other video frames have been subjected to proximity matching calculation with the kth video frame; S4, retaining the feature points whose number of occurrences in the feature point set of the kth video frame exceeds a preset threshold, thereby generating a static feature template; S5, extracting the feature point set of the current video frame from the video frame one by one, performing Lloyd's ratio test calculation on the feature point set and the static feature template, filtering the dynamic feature points, retaining the static feature points, establishing a homography matrix between the current video frame and the k-th video frame according to the positions of the static feature points of the current video frame and the positions of the static feature points in the static feature template, and aligning the current video frame with the k-th video frame according to the homography matrix; S6, detecting a difference region between the k-th video frame and the current video frame, cropping and compensating the difference region in the k-th video frame to the current video frame, and then rewriting the current video frame into the video; S7, repeat S5~S6 until the last frame of the video.

[0007] Preferably, in S1, a SURF feature point detector is used to extract feature points of each video frame, thereby generating a feature point set belonging to the video frame.

[0008] Preferably, the step of performing proximity matching calculation on the feature point sets of other video frames and the feature point set of the kth video frame in S2 specifically includes the following steps: S21. Based on the CUDA parallel acceleration algorithm, a kd-tree spatial index is constructed for the feature point set of the kth frame. The point cloud is divided into 32×32 grid units. The segmentation dimension of each node selects the axis with the largest variance. The maximum capacity of a leaf node is 16 points, and the search depth is limited to 20 layers. S22, using a dynamic radius matching strategy, dynamically calculating the matching radius according to the resolution of the input frame: , Taking the feature point of the kth frame as the center, the kd tree is used to quickly retrieve candidate matching points in other video frames whose spatial distance is less than r; S23. Hamming distance screening based on double constraints: If the number of candidate matching pairs is >500, retain the top 8% matching pairs with a Hamming distance less than 25 and the smallest distance; if the number of candidates is ≤500, retain the matching pairs with a Hamming distance less than 30; eliminate false matches through the probability weighted RANSAC algorithm; dynamically adjust the sampling weight according to the distribution density of feature points, and reduce the weight of high-density areas by 30%. The high-density area refers to the area with more than 50 feature points in the 32×32 grid unit; set the maximum number of iterations to 3000, and terminate early if the model error change is less than 0.1% within 100 consecutive iterations; use the LMeds optimizer to calculate the basic matrix and select the model with the smallest median error: , in is the homography matrix, and are the coordinates of the matching point pairs.

[0009] S24, statistical standard deviation σ of the displacement vector of feature points between frames: , If σ>2.5 pixels, it is determined as an abnormal motion area and all matching points in the area are removed, where N is the number of matching point pairs. For the i For the displacement vector of the matching point between frames, is the mean vector of the displacement vectors of all matching point pairs, It is the square of the Euclidean distance between the displacement vector of a single pair of matching points and the mean vector, which is used to quantify the degree of deviation of the displacement.

[0010] Preferably, in S1, the video frame extraction is performed once every preset time interval.

[0011] Preferably, the method of extracting a feature point set of each video frame in S1 includes: S11, construct a 3-layer Gaussian pyramid with scaling ratios of 0.5, 1.0, and 1.5 respectively. l Layer Image I l Through the l -1 layer image is Gaussian smoothed and downsampled to get: , Among them, G(m,n) is a 5×5 Gaussian kernel with a standard deviation of σ=1.2. The parameters m and n are the coordinate offsets used to traverse the Gaussian kernel. The 1.5-fold layer is implemented by upsampling the original image through bilinear interpolation. S12. In each layer of the pyramid image, use the FAST-9 algorithm to detect corner points, take the brightness threshold T=20, retain the corner point with the largest response value in the 3×3 neighborhood, avoid repeated detection in dense areas, sort the corner points detected in each layer by response value, and dynamically adjust the threshold to ensure that 400±50 feature points are extracted in each layer, and a total of 1200±200 feature points are extracted per frame; S13. For each feature point, calculate the grayscale centroid of its neighborhood : , The neighborhood radius of each feature point is r = 15 pixels, and the feature point direction ; S14, according to the feature point direction θ, the feature point neighborhood is rotated to align with the main direction, using 256 pairs of predefined random point pairs (( x i , y i )and( x j , y j )), compare the grayscale values ​​in the rotated neighborhood to generate a binary descriptor k: , Finally, a 256-bit binary descriptor is generated; S15. Map the coordinates of the feature points of each pyramid layer back to the original image scale. At the original image scale, perform 3×3 neighborhood screening on the feature points in the overlapping area, retain the points with the largest response value, and avoid redundancy.

[0012] Preferably, Lloyd's Ratio Test calculation with a multiplier of 0.7 is applied in S5.

[0013] The present invention also provides a device for eliminating high-altitude image jitter based on a static frame feature formula, comprising: The feature point extraction module obtains the video, extracts frames from the video, and extracts the feature point set of each video frame; The proximity matching module uses the extracted feature point set of the k-th video frame as a template, performs proximity matching calculations on the feature point sets of other video frames and the feature point set of the k-th video frame. When a successfully matched feature point appears in the two feature point sets, it is detected whether the feature point has an identifier. If there is no identifier, a unique identifier is generated for the feature point, and the number of occurrences of the feature point is initialized to 2. If there is an identifier, the number of occurrences of the feature point is increased by 1. The first repetition module repeatedly calls the proximity matching module until all other video frames have been subjected to proximity matching calculations with the kth video frame; The static feature template generation module retains the feature points in the feature point set of the kth video frame whose number of occurrences exceeds a preset threshold, thereby generating a static feature template; The video frame alignment module extracts the feature point set of the current video frame from the video frame one by one, performs Lloyd's ratio test calculation on the feature point set and the static feature template, filters the dynamic feature points, retains the static feature points, establishes the homography matrix between the current video frame and the k-th video frame according to the position of the static feature points of the current video frame and the position of the static feature points in the static feature template, and aligns the current video frame with the k-th video frame according to the perspective transformation matrix; The difference compensation module detects the difference area between the k-th video frame and the current video frame, crops and compensates the difference area in the k-th video frame to the current video frame, and then rewrites the current video frame into the video; The second repetition module repeatedly calls the video frame alignment module and the difference compensation module until the last frame of the video.

[0014] The present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0015] The present invention also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above method is implemented.

[0016] The present invention includes at least the following beneficial effects: the present invention significantly improves the robustness, continuity and computational efficiency of video stabilization while reducing human intervention through automated feature screening, dynamic interference filtering and difference compensation mechanism, and is particularly suitable for complex scenes where dynamic objects frequently appear during high-altitude photography by drones.

[0017] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a flow chart of the method for eliminating high-altitude image jitter based on static frame features according to the present invention. DETAILED DESCRIPTION

[0019] It should be noted that, in order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, the present invention provides a method for eliminating high-altitude image jitter based on a static frame feature formula, comprising: S1, obtain the video, extract the video frame, and extract the feature point set of each video frame; Specifically, drones can be used to shoot traffic flow videos at an altitude of 100m or 300m on urban roads, with a shooting time of more than 2 minutes; The following rules can be used to extract frames from the video: 100m high-altitude video is extracted at intervals of 2s, and 300m high-altitude video is extracted at intervals of 1s. The extracted video frames constitute an image set P, which contains a total of n images: P = {p1, p2, …p n}, To detect feature points on the image set P, you can use the SURF feature point detector and record the corresponding feature points to generate the total feature point set W: W = {w p1 , w p2 ,…,w pn}, Among them, w pn Represents the set of feature points generated by the nth video frame in the image set P.

[0021] The generated feature point set includes both static object feature points and dynamic object feature points. From the perspective of spatiotemporal continuity, since the drone shooting method is high-altitude fixed-point shooting, the same static object in all frames should be in the same spatial position. Therefore, the feature points of these static objects can be formed into a set as a reference for comparison of all video frames.

[0022] S2, using the extracted feature point set of the k-th video frame as a template, performing proximity matching calculation on the feature point sets of other video frames and the feature point set of the k-th video frame. When a successfully matched feature point appears in the two feature point sets, it is detected whether the feature point has an identifier. If there is no identifier, a unique identifier is generated for the feature point, and the number of occurrences of the feature point is initialized to 2. If there is an identifier, the number of occurrences of the feature point is increased by 1. Specifically, a feature point library is created, and the initial feature point library is an empty set; Get the feature point set of the kth video frame as the template reference, and pk and w pi (1≤i≤n and i≠k) perform feature point proximity matching calculation; Record the successfully matched feature points. If the current feature point is not marked, proceed to step a. Otherwise, proceed to step b: a. Generate a unique identifier for the feature point of the current record and initialize the number of occurrences to 2; b. Add 1 to the number of occurrences of the feature point in the current record.

[0023] S3, repeat S2 until all other video frames have been subjected to proximity matching calculation with the kth video frame; S4, retaining the feature points whose number of occurrences in the feature point set of the kth video frame exceeds a preset threshold, thereby generating a static feature template; Specifically, the number of times all feature points appear is counted, high-frequency feature points are retained, low-frequency feature points are removed, and high-frequency feature points are written into the feature point library to form a static feature template. The judgment rule for high-frequency feature points is as follows: the feature points are sorted from high to low according to the number of occurrences, and the top 50% of the feature points whose number is greater than or equal to 1 / 2 of the extracted frames are taken as high-frequency feature points, and the others are low-frequency feature points.

[0024] S5, extracting the feature point set of the current video frame from the video frame one by one, performing Lloyd's ratio test calculation on the feature point set and the static feature template, filtering the dynamic feature points, retaining the static feature points, establishing a homography matrix between the current video frame and the k-th video frame according to the positions of the static feature points of the current video frame and the positions of the static feature points in the static feature template, and aligning the current video frame with the k-th video frame according to the homography matrix; Specifically, a multiplier of 0.7 is applied in the Lowe's ratio test.

[0025] S6, detecting a difference region between the k-th video frame and the current video frame, cropping and compensating the difference region in the k-th video frame to the current video frame, and then rewriting the current video frame into the video; S7, repeat S5~S6 until the last frame of the video.

[0026] In this embodiment, dynamic feature points are filtered by the Lowe's ratio test, combined with static feature template generation (retaining high-frequency static feature points), to effectively distinguish static backgrounds from dynamic objects (such as vehicles and pedestrians). In view of the problem that the prior art relies on manual selection of anchor points, and the compensation mechanism is only for anchor points that fail to track, which is easily affected by occlusion of dynamic objects. This solution can automatically remove the interference features of temporary occlusion or moving objects by counting the frequency of feature points and dynamic filtering, significantly improving the image stabilization robustness of complex dynamic scenes. In this embodiment, the static feature template is generated by multi-frame feature point matching statistics, without the need to manually select anchor points. In view of the problem that the prior art needs to manually select the focus area anchor point in the first frame, relies on manual operation and may introduce subjective errors, this solution is fully automated, reduces labor costs, and improves the scalability of the algorithm (suitable for large-scale video processing). In this embodiment, the difference set area compensation crops the difference area between the template frame and the current frame and compensates it to the current frame. In view of the fact that the existing technology only aligns through affine transformation, which may cause the loss of local information of the picture due to the occlusion of dynamic objects, this solution compensates for the difference set area, repairs the picture defects caused by occlusion or movement, and improves the visual coherence of the output video. In this embodiment, based on homography matrix alignment and static template matching, redundant calculations are reduced. In view of the problem that the existing technology requires iterative calculation of robust estimation weights and has high computational complexity, this solution reduces the resource consumption of real-time calculations through pre-screening of static templates and dynamic feature filtering. At the same time, the homography matrix is ​​more suitable for complex perspective transformation scenes (such as drone overhead perspective) and has higher image stabilization accuracy. In this embodiment, it does not rely on the number of specific anchor points (such as the existing technology fixes 12 pairs of anchor points) and supports adaptive feature point matching. In view of the fact that the existing technology has strict restrictions on the number and position of anchor points, it may fail in scenes with sparse features. The new solution generates templates through multi-frame statistics, which can adapt to the feature distribution in different scenes, and expands the application potential of drone image stabilization technology in high-dynamic, low-texture environments (such as sparse traffic scenes).

[0027] In general, this embodiment significantly improves the robustness, continuity and computational efficiency of video stabilization while reducing human intervention through automated feature screening, dynamic interference filtering and difference compensation mechanism. It is particularly suitable for complex scenes where dynamic objects frequently appear in high-altitude drone photography.

[0028] Furthermore, the feature point set of other video frames is subjected to proximity matching calculation with the feature point set of the kth video frame in S2, which specifically includes the following steps: S21. Based on the CUDA parallel acceleration algorithm, a kd-tree spatial index is constructed for the feature point set of the kth frame. The point cloud is divided into 32×32 grid units. The segmentation dimension of each node selects the axis with the largest variance. The maximum capacity of a leaf node is 16 points, and the search depth is limited to 20 layers. S22, using a dynamic radius matching strategy, dynamically calculating the matching radius according to the resolution of the input frame: , Taking the feature point of the kth frame as the center, the kd tree is used to quickly retrieve candidate matching points in other video frames whose spatial distance is less than r; S23. Hamming distance screening based on double constraints: If the number of candidate matching pairs is >500, retain the top 8% matching pairs with a Hamming distance less than 25 and the smallest distance; if the number of candidates is ≤500, retain the matching pairs with a Hamming distance less than 30; eliminate false matches through the probability weighted RANSAC algorithm; dynamically adjust the sampling weight according to the distribution density of feature points, and reduce the weight of high-density areas by 30%. The high-density area refers to the area with more than 50 feature points in the 32×32 grid unit; set the maximum number of iterations to 3000, and terminate early if the model error change is less than 0.1% within 100 consecutive iterations; use the LMeds optimizer to calculate the basic matrix and select the model with the smallest median error: , in is the homography matrix, and are the coordinates of the matching point pairs.

[0029] S24, statistical standard deviation σ of the displacement vector of feature points between frames: , If σ>2.5 pixels, it is determined as an abnormal motion area and all matching points in the area are removed, where N is the number of matching point pairs. For the i For the displacement vector of the matching point between frames, is the mean vector of the displacement vectors of all matching point pairs, It is the square of the Euclidean distance between the displacement vector of a single pair of matching points and the mean vector, which is used to quantify the degree of deviation of the displacement.

[0030] In this embodiment, the kd tree index divides the feature points into grid units (32×32), selects the axis with the largest variance in the segmentation dimension, limits the search depth (20 layers), and significantly accelerates the spatial retrieval efficiency of large-scale feature points, especially for high-resolution drone images (such as 4K / 8K). Dynamic radius matching (8 / 12 / 15 pixel classification) avoids redundant calculations of fixed search radius at different resolutions, reducing the time complexity by about 30%~50% (compared with traditional global search). CUDA parallel acceleration uses GPU computing resources to further optimize the matching speed and meet the real-time stabilization requirements of drone videos. The threshold is dynamically adjusted according to the number of candidate matches (the top 8% and the distance <25 are retained when the number of candidates is >500; the distance is retained when the number of candidates is ≤500) to effectively filter out mismatches caused by texture repetition or noise. The sampling weight is dynamically adjusted according to the distribution density of the feature points (the weight of the high-density area is reduced by 30%) to avoid the dominant model fitting of the dense area mismatch and improve the estimation accuracy of the basic matrix (homography matrix). The LMeds optimizer selects the model with the smallest median error. Compared with the mean error of traditional RANSAC, it has a higher tolerance for outliers and improves the robustness of the model by about 20%. Calculate the standard deviation of the displacement vector of the matching point pair σ , quantify the inter-frame motion consistency. σ >2.5 pixels, it is determined to be an abnormal displacement area caused by dynamic objects (such as vehicles and pedestrians), and all matching points in the area are directly removed. Combined with the Lowe's ratio test, dynamic feature points are further filtered so that the homography matrix is ​​aligned only based on static background features to avoid the interference of dynamic objects on the stabilization results (compared with the existing technology that relies on manual point selection, the robustness is improved by about 35%). Traditional RANSAC requires a fixed maximum number of iterations (such as 3000 times), while this solution reduces redundant iterations through dynamic termination conditions, reducing the calculation time by about 40%~60%. Hierarchical feature processing (kd tree index → ​​Hamming distance screening → RANSAC optimization) gradually removes low-quality matching points, reduces the data scale of subsequent model fitting, and saves memory and computing resources. The dynamic radius matching strategy adapts to different resolution inputs (720P to 8K) to avoid mismatching caused by too large search radius at low resolution. The weight of high-density areas is reduced by 30% to prevent overfitting of local features in texture-rich scenes such as urban roads, and improve the applicability of the algorithm under complex backgrounds (such as dense buildings and vegetation).

[0031] In general, this embodiment significantly improves the real-time, accuracy and robustness of drone high-altitude image stabilization through efficient feature retrieval (kd tree + dynamic radius), high-precision mismatch elimination (double Hamming + RANSAC optimization) and dynamic interference suppression (abnormal area elimination). Compared with the existing technology (such as CN117575966 B relying on manual point selection and fixed number of anchor points), this solution performs better in complex dynamic scenes (traffic, pedestrians) and low-texture environments (such as sparse roads), while reducing computing resource consumption, and is suitable for large-scale drone video real-time processing needs.

[0032] In another embodiment, the method of extracting a feature point set of each video frame in S1 includes: S11, construct a 3-layer Gaussian pyramid with scaling ratios of 0.5, 1.0, and 1.5 respectively. l Layer Image I l Through the l -1 layer image is Gaussian smoothed and downsampled to get: , Among them, G(m,n) is a 5×5 Gaussian kernel with a standard deviation of σ=1.2. The parameters m and n are the coordinate offsets used to traverse the Gaussian kernel. The 1.5-fold layer is implemented by upsampling the original image through bilinear interpolation. S12. In each layer of the pyramid image, use the FAST-9 algorithm to detect corner points, take the brightness threshold T=20, retain the corner point with the largest response value in the 3×3 neighborhood, avoid repeated detection in dense areas, sort the corner points detected in each layer by response value, and dynamically adjust the threshold to ensure that 400±50 feature points are extracted in each layer, and a total of 1200±200 feature points are extracted per frame; S13. For each feature point, calculate the grayscale centroid of its neighborhood : , The neighborhood radius of each feature point is r = 15 pixels, and the feature point direction ; S14, according to the feature point direction θ, the feature point neighborhood is rotated to align with the main direction, using 256 pairs of predefined random point pairs (( x i , y i )and( x j , y j )), compare the grayscale values ​​in the rotated neighborhood to generate a binary descriptor k: , Finally, a 256-bit binary descriptor is generated; S15, mapping the coordinates of the feature points of each pyramid layer back to the original image scale, performing 3×3 neighborhood screening on the feature points in the overlapping area at the original image scale, retaining the points with the largest response value to avoid redundancy.

[0033] In this embodiment, a three-layer Gaussian pyramid (scaling ratios are 0.5, 1.0, and 1.5) is constructed, and multi-scale images are generated by downsampling and upsampling. The low-level pyramid (0.5 times) detects large-scale features (such as the edge of a building), the middle level (1.0 times) captures conventional features, and the high-level (1.5 times) enhances details through bilinear interpolation to adapt to the scene of mixed near and far targets in the high-altitude images of drones. Gaussian smoothing (standard deviation σ=1.2) effectively suppresses image noise and avoids feature misdetection caused by slight shaking of the drone. The pyramid layered processing reduces repeated calculations. Compared with traditional single-scale detection, the calculation time is reduced by about 30%. In this embodiment, the FAST-9 algorithm is used to detect corner points, and the threshold is dynamically adjusted to ensure that 400±50 feature points are extracted per layer, and redundancy is avoided by neighborhood screening. The FAST-9 algorithm only needs to compare the brightness of pixels in the 3×3 neighborhood, and the detection speed is 5~10 times faster than SIFT / SURF, which meets the real-time processing requirements. The threshold is dynamically adjusted according to the response value sorting of each layer (such as the brightness threshold T=20) to ensure that the number of feature points is stable (a total of 1200±200 / frame), avoiding insufficient features in sparse scenes or overload in dense scenes. At the original image scale, the overlapping area is screened by 3×3 neighborhoods, and the points with the largest response values ​​are retained, reducing redundant feature points by about 40%, and improving the subsequent matching efficiency. In this embodiment, the grayscale centroid direction of the feature point neighborhood is calculated (radius r=15 pixels), and the neighborhood is rotated to align with the main direction. The feature point direction is corrected by the grayscale centroid direction, so that the binary descriptor is insensitive to image rotation. Experiments show that the rotation error tolerance is improved to ±15° (compared to ±5° in the traditional method), and the neighborhood after rotation generates a binary descriptor (256 pairs of random points), and the consistency of the descriptor is improved by about 25%, reducing mismatches caused by changes in viewing angles. In this embodiment, a 256-bit binary descriptor is used, which is generated by comparing the grayscale values ​​of random point pairs in the neighborhood after rotation. Each feature point only requires 32 bytes (256 bits), which reduces the storage space by 93.75% compared to SIFT (128-dimensional floating point, 512 bytes), and is suitable for large-scale video processing. The calculation speed of the Hamming distance (XOR + bit count) is 20 to 50 times faster than the Euclidean distance. Combined with CUDA acceleration, the matching efficiency is significantly improved.

[0034] In general, this embodiment significantly improves the robustness, efficiency and scene adaptability of UAV high-altitude image feature extraction through multi-scale feature detection, efficient corner point screening, rotation-invariant descriptor generation and lightweight storage design. Compared with the existing technology, this solution performs better in complex dynamic scenes (such as urban traffic) and low-texture environments (such as long-distance shooting), while reducing computing resource consumption, providing high-quality feature input for subsequent jitter elimination algorithms (such as static template matching and homography matrix alignment), thereby improving the overall video stabilization effect.

[0035] Based on the same inventive concept, the present invention also provides a device for eliminating high-altitude image jitter based on a static frame feature formula. The device can be a personal computer, a server, or other devices that implement the aforementioned method for eliminating high-altitude image jitter based on a static frame feature formula.

[0036] The device for eliminating high-altitude image jitter based on the static frame feature formula of the present invention comprises: The feature point extraction module obtains the video, extracts frames from the video, and extracts the feature point set of each video frame; The proximity matching module uses the extracted feature point set of the k-th video frame as a template, performs proximity matching calculations on the feature point sets of other video frames and the feature point set of the k-th video frame. When a successfully matched feature point appears in the two feature point sets, it is detected whether the feature point has an identifier. If there is no identifier, a unique identifier is generated for the feature point, and the number of occurrences of the feature point is initialized to 2. If there is an identifier, the number of occurrences of the feature point is increased by 1. The first repetition module repeatedly calls the proximity matching module until all other video frames have been subjected to proximity matching calculations with the kth video frame; The static feature template generation module retains the feature points in the feature point set of the kth video frame whose number of occurrences exceeds a preset threshold, thereby generating a static feature template; The video frame alignment module extracts the feature point set of the current video frame from the video frame one by one, performs Lloyd's ratio test calculation on the feature point set and the static feature template, filters the dynamic feature points, retains the static feature points, establishes the homography matrix between the current video frame and the k-th video frame according to the position of the static feature points of the current video frame and the position of the static feature points in the static feature template, and aligns the current video frame with the k-th video frame according to the perspective transformation matrix; The difference compensation module detects the difference area between the k-th video frame and the current video frame, crops and compensates the difference area in the k-th video frame to the current video frame, and then rewrites the current video frame into the video; The second repetition module repeatedly calls the video frame alignment module and the difference compensation module until the last frame of the video.

[0037] All relevant contents of each step involved in the embodiment of the aforementioned method for eliminating high-altitude image jitter based on static frame features can be referred to the functional description of the functional modules corresponding to the device in the embodiment of the present application, and will not be repeated here.

[0038] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0039] The present invention also provides an electronic device, comprising: at least one processor, and a memory connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned method for eliminating high-altitude image jitter based on static frame features. The electronic device can be any terminal device including a mobile phone, a laptop computer, a desktop computer, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, etc.

[0040] The present invention also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for eliminating high-altitude image jitter based on the static frame feature formula is implemented.

[0041] Through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present invention, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0042] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for eliminating high-altitude image jitter based on static frame features, characterized in that: include: S1, obtain the video, extract the video frame, and extract the feature point set of each video frame; S2, using the extracted feature point set of the k-th video frame as a template, performing proximity matching calculation on the feature point sets of other video frames and the feature point set of the k-th video frame. When a successfully matched feature point appears in the two feature point sets, it is detected whether the feature point has an identifier. If there is no identifier, a unique identifier is generated for the feature point, and the number of occurrences of the feature point is initialized to 2. If there is an identifier, the number of occurrences of the feature point is increased by 1. S3, repeat S2 until all other video frames have been subjected to proximity matching calculation with the kth video frame; S4, retaining the feature points whose number of occurrences in the feature point set of the kth video frame exceeds a preset threshold, thereby generating a static feature template; S5, extracting the feature point set of the current video frame from the video frame one by one, performing Lloyd's ratio test calculation on the feature point set and the static feature template, filtering the dynamic feature points, retaining the static feature points, establishing a homography matrix between the current video frame and the k-th video frame according to the positions of the static feature points of the current video frame and the positions of the static feature points in the static feature template, and aligning the current video frame with the k-th video frame according to the homography matrix; S6, detecting a difference region between the k-th video frame and the current video frame, cropping and compensating the difference region in the k-th video frame to the current video frame, and then rewriting the current video frame into the video; S7, repeat S5~S6 until the last frame of the video.

2. The method for eliminating high-altitude image jitter based on static frame features as claimed in claim 1, characterized in that: In S1, the SURF feature point detector is used to extract the feature points of each video frame, thereby generating a set of feature points belonging to the video frame.

3. The method for eliminating high-altitude image jitter based on static frame feature formula according to claim 1, characterized in that: The method of performing proximity matching calculation on the feature point sets of other video frames and the feature point set of the kth video frame in S2 specifically includes the following steps: S21. Based on the CUDA parallel acceleration algorithm, a kd-tree spatial index is constructed for the feature point set of the kth frame. The point cloud is divided into 32×32 grid units. The segmentation dimension of each node selects the axis with the largest variance. The maximum capacity of a leaf node is 16 points, and the search depth is limited to 20 layers. S22, using a dynamic radius matching strategy, dynamically calculating the matching radius according to the resolution of the input frame: ， Taking the feature point of the kth frame as the center, the kd tree is used to quickly retrieve candidate matching points in other video frames whose spatial distance is less than r; S23. Hamming distance screening based on double constraints: If the number of candidate matching pairs is >500, retain the top 8% matching pairs with a Hamming distance less than 25 and the smallest distance; if the number of candidates is ≤500, retain the matching pairs with a Hamming distance less than 30; eliminate false matches through the probability weighted RANSAC algorithm; dynamically adjust the sampling weight according to the distribution density of feature points, and reduce the weight of high-density areas by 30%. The high-density area refers to the area with more than 50 feature points in the 32×32 grid unit; set the maximum number of iterations to 3000, and terminate early if the model error change is less than 0.1% within 100 consecutive iterations; use the LMeds optimizer to calculate the basic matrix and select the model with the smallest median error: ， in is the homography matrix, and is the coordinates of the matching point pair; S24, statistical standard deviation σ of the displacement vector of feature points between frames: ， If σ>2.5 pixels, it is determined as an abnormal motion area and all matching points in the area are removed, where N is the number of matching point pairs. For the i For the displacement vector of the matching point between frames, is the mean vector of the displacement vectors of all matching point pairs, It is the square of the Euclidean distance between the displacement vector of a single pair of matching points and the mean vector, which is used to quantify the degree of deviation of the displacement.

4. The method for eliminating high-altitude image jitter based on static frame feature formula according to claim 1, characterized in that: In S1, the video frame is extracted once every preset time interval.

5. The method for eliminating high-altitude image jitter based on static frame feature formula according to claim 1, characterized in that: The method of extracting a feature point set of each video frame in S1 includes: S11, construct a 3-layer Gaussian pyramid with scaling ratios of 0.5, 1.0, and 1.5 respectively. l Layer Image I l Through the l -1 layer image is Gaussian smoothed and downsampled to get: ， Among them, G(m,n) is a 5×5 Gaussian kernel with a standard deviation of σ=1.

2. The parameters m and n are the coordinate offsets used to traverse the Gaussian kernel. The 1.5-fold layer is implemented by upsampling the original image through bilinear interpolation. S12. In each layer of the pyramid image, use the FAST-9 algorithm to detect corner points, take the brightness threshold T=20, retain the corner point with the largest response value in the 3×3 neighborhood, avoid repeated detection in dense areas, sort the corner points detected in each layer by response value, and dynamically adjust the threshold to ensure that 400±50 feature points are extracted in each layer, and a total of 1200±200 feature points are extracted per frame; S13. For each feature point, calculate the grayscale centroid of its neighborhood : ， The neighborhood radius of each feature point is r = 15 pixels, and the feature point direction ; S14, according to the feature point direction θ, the feature point neighborhood is rotated to align with the main direction, using 256 pairs of predefined random point pairs (( x i , y i )and( x j , y j )), compare the grayscale values ​​in the rotated neighborhood to generate a binary descriptor k: ， Finally, a 256-bit binary descriptor is generated; S15. Map the coordinates of the feature points of each pyramid layer back to the original image scale. At the original image scale, perform 3×3 neighborhood screening on the feature points in the overlapping area, retain the points with the largest response value, and avoid redundancy.

6. The method for eliminating high-altitude image jitter based on static frame feature formula according to claim 1, characterized in that: S5 is calculated using the Lloyd's ratio test with a multiplier of 0.

7.

7. A device for eliminating high-altitude image jitter based on static frame characteristics, characterized in that: include: The feature point extraction module obtains the video, extracts frames from the video, and extracts the feature point set of each video frame; The proximity matching module uses the extracted feature point set of the k-th video frame as a template, performs proximity matching calculations on the feature point sets of other video frames and the feature point set of the k-th video frame. When a successfully matched feature point appears in the two feature point sets, it is detected whether the feature point has an identifier. If there is no identifier, a unique identifier is generated for the feature point, and the number of occurrences of the feature point is initialized to 2. If there is an identifier, the number of occurrences of the feature point is increased by 1. The first repetition module repeatedly calls the proximity matching module until all other video frames have been subjected to proximity matching calculations with the kth video frame; The static feature template generation module retains the feature points whose number of occurrences in the feature point set of the kth video frame exceeds a preset threshold, thereby generating a static feature template; The video frame alignment module extracts the feature point set of the current video frame from the video frame one by one, performs Lloyd's ratio test calculation on the feature point set and the static feature template, filters the dynamic feature points, retains the static feature points, establishes the homography matrix between the current video frame and the k-th video frame according to the position of the static feature points of the current video frame and the position of the static feature points in the static feature template, and aligns the current video frame with the k-th video frame according to the perspective transformation matrix; The difference compensation module detects the difference area between the k-th video frame and the current video frame, crops and compensates the difference area in the k-th video frame to the current video frame, and then rewrites the current video frame into the video; The second repetition module repeatedly calls the video frame alignment module and the difference compensation module until the last frame of the video.

8. An electronic device, characterized in that include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor performs the method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images