A video shake correction method and system based on high-altitude perspective

Abstract

The application relates to a high-altitude perspective-based video jitter correction method and system, wherein the method comprises the following steps: extracting feature points from a pretreated detection frame, optimizing the distribution of the feature points, and obtaining a feature point set. The feature point trajectories are tracked frame by frame through a motion tracking rule, and a feature point trajectory set is obtained. According to the motion trajectories of each feature point in the feature point trajectory set, a polar coordinate system comprising a plurality of feature points is generated, and the polar coordinate system is converted into a rectangular coordinate system. The feature points in the rectangular coordinate system are clustered through a density clustering algorithm, and a target feature point set is obtained. According to the target feature point set, the transformation relationship between adjacent frames is calculated, the current frame is inversely transformed based on the transformation relationship, and a corrected video frame is output. The high-altitude perspective-based video jitter correction method in the related art has the problem of low correction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video jitter correction, in particular to a video jitter correction method and system based on high-altitude view. BACKGROUND

[0002] With the wide application of mobile shooting devices, unmanned aerial vehicles and high-altitude monitoring cameras, the problem of video jitter is increasingly prominent, which seriously affects the picture quality and the accuracy of subsequent intelligent analysis. In particular, under high-altitude view, slight jitter will be greatly magnified. For example, when shooting at an altitude of 120 meters, an error of only 1 pixel in the position of a target vehicle may result in an instantaneous speed error of up to 9.8 km / h. Therefore, high-precision video jitter correction technology is of great significance for unmanned aerial vehicle aerial photography, traffic monitoring and other application scenarios.

[0003] At present, video correction technologies mainly include digital correction based on image content and hybrid correction. However, the existing video jitter correction methods for high-altitude view still have the problem of low correction accuracy. SUMMARY

[0004] The embodiments of the present application provide a video jitter correction method and system based on high-altitude view, to at least solve the problem of low correction accuracy of the video jitter correction method based on high-altitude view in the related art.

[0005] In a first aspect, the embodiments of the present application provide a video jitter correction method based on high-altitude view, which comprises:

[0006] extracting feature points from the preprocessed detection frame, optimizing the distribution of the feature points, and obtaining a feature point set;

[0007] tracking the feature point trajectories frame by frame through a motion tracking rule, and obtaining a feature point trajectory set;

[0008] generating a polar coordinate system including a plurality of feature points according to the motion trajectories of each feature point in the feature point trajectory set, and converting the polar coordinate system into a rectangular coordinate system;

[0009] clustering the feature points in the rectangular coordinate system through a density clustering algorithm, and obtaining a target feature point set;

[0010] calculating the transformation relationship between adjacent frames according to the target feature point set, performing inverse transformation matrix on the current frame based on the transformation relationship, and outputting a corrected video frame.

[0011] In an embodiment, before extracting feature points from the preprocessed detection frame, the method comprises:

[0012] The first frame of the video is taken as a first frame, and if the displacement between the first frame and the next frame exceeds a preset displacement threshold or the deflection angle between the first frame and the next frame exceeds a preset deflection angle threshold, a detection frame is determined through video clipping or frame selection;

[0013] The frame width, frame height, frame rate and total frame number of the video are parsed and recorded by a video decoding module;

[0014] The detection frame is preprocessed, wherein the preprocessing includes converting the detection frame into a grayscale image;

[0015] Based on the grayscale image, a feature point set is initialized and a trajectory set , wherein the feature point set includes the unique ID and initial coordinates of each feature point, represents the nth feature point, and the initial feature point set backup , the trajectory set includes inter-frame motion trajectory data of each feature point, including the frame number, coordinates, displacement distance and motion direction of the current trajectory point.

[0016] In an embodiment, the feature points are extracted from the preprocessed detection frame, the feature points are distributed and optimized, and a feature point set is obtained, including:

[0017] The autocorrelation matrix of each pixel point in the grayscale image is calculated by a feature point detection algorithm, based on the autocorrelation matrix, the corner point response degree is determined according to the minimum eigenvalue, and the corner point response degree is sorted from high to low, and the pixel points of the preset rank are selected as the initial feature point set;

[0018] According to the imaging characteristics of the high-altitude view angle, the initial feature point set is optimized by the maximum number of corner points, the corner point quality threshold, the minimum corner point spacing and the detection window size;

[0019] The distribution of the optimized initial feature point set is optimized, wherein the distribution optimization includes: dividing the grayscale image into a plurality of image blocks, each image block including at least one feature point; removing invalid feature points in the sky and shadow area; and retaining the number of static feature points of the ground and building area and the number of feature points of dynamic targets according to the priority;

[0020] The initial feature point set after distribution optimization is taken as a feature point set, wherein the feature point set includes the ID number of a plurality of feature points and the coordinate information in the detection frame.

[0021] In an embodiment, the feature point trajectories are tracked frame by frame based on a motion tracking rule, and a feature point trajectory set is obtained, including:

[0022] define a window size, a pyramid layer number, and an iteration termination condition parameter;

[0023] predict and track the positions of all feature points in the previous detection frame by a sparse optical flow method to obtain the positions of each feature point in the current frame;

[0024] based on the defined window size, the pyramid layer number, and the iteration termination condition parameter, obtain a feature point trajectory set by a pyramid optimization algorithm.

[0025] In an embodiment, the generating a polar coordinate system including a plurality of feature points according to the displacement distance and the motion direction of each feature point from the first frame to the last frame in the feature point trajectory set, and converting the polar coordinate system into a rectangular coordinate system, comprises:

[0026] extracting the first frame coordinates and the last frame coordinates of a plurality of feature points according to the overall motion trajectory of each feature point in the feature point trajectory set;

[0027] generating a polar coordinate system including a plurality of feature points based on the displacement distance and the motion direction according to the determined displacement distance of the first frame coordinates and the last frame coordinates, and the determined motion direction according to the direction angle of the first frame coordinates and the last frame coordinates;

[0028] converting the polar coordinate system into a rectangular coordinate system according to the mapping relationship between the polar coordinate system and the rectangular coordinate system.

[0029] In an embodiment, the converting the polar coordinate system into a rectangular coordinate system according to the mapping relationship between the polar coordinate system and the rectangular coordinate system, comprises:

[0030] calculating the displacement distance of the first frame coordinates and the last frame coordinates by the Euclidean distance according to the first frame coordinates and the last frame coordinates;

[0031] calculating the ratio of the vertical displacement of the first frame coordinates and the last frame coordinates to the displacement distance, performing inverse sine processing based on the ratio to obtain a direction angle, and determining the motion direction according to the direction angle;

[0032] converting the polar coordinate system into a rectangular coordinate system by a trigonometric function relationship based on the displacement distance and the motion direction, wherein the y-axis in the rectangular coordinate system represents the motion direction of the trajectory, the x-axis represents the displacement distance, and the feature point data set P is set as , and each has coordinates .

[0033] In an embodiment, the clustering the feature points in the rectangular coordinate system by a density clustering algorithm to obtain a target feature point set comprises:

[0034] determine a neighborhood radius parameter and a minimum point number parameter of core points within the neighborhood radius of the density algorithm according to a density distribution of the feature points in the rectangular coordinate system;

[0035] perform density clustering on the feature points in the rectangular coordinate system through reachability of the density algorithm based on the neighborhood radius parameter and the minimum point number parameter;

[0036] obtain a main density cluster including core points as the target feature point set according to a result of the density clustering.

[0037] In an embodiment, determining the neighborhood radius parameter and the minimum point number parameter of core points within the neighborhood radius of the density algorithm comprises:

[0038] calculate distances of feature point coordinates in the rectangular coordinate system to k-neighbor points, generate a distance curve according to distance ordering from large to small, wherein the k-neighbor points are obtained through a k-distance function;

[0039] identify a distance value corresponding to a maximum inflection point as the neighborhood radius parameter through moving average smoothing processing of the distance curve;

[0040] determine the minimum point number parameter of core points within the neighborhood radius based on the k-neighbor points;

[0041] perform density clustering on the feature points in the rectangular coordinate system through reachability of the density algorithm, comprising:

[0042] traverse each feature point in the rectangular coordinate system according to the neighborhood radius parameter and the minimum point number parameter;

[0043] if a point number within a neighborhood radius of a current feature point is greater than or equal to the minimum point number parameter, the feature point is marked as a core point; if the point number within the neighborhood radius of the current feature point is less than the minimum point number parameter, the current feature point is a noise point;

[0044] if the current feature point is a core point, create a new clustering cluster, and add all feature points within its neighborhood to a candidate set;

[0045] perform iteration on the candidate set, for a feature point in the candidate set that has not been allocated a cluster, mark it as visited, and expand the candidate set according to core points until a current clustering cluster meets an expansion criterion, to obtain a clustering result.

[0046] In an embodiment, the method further comprises:

[0047] According to a corresponding relationship between the current frame feature point coordinates in the target feature point set and the previous frame feature corner point coordinates, a linear equation is constructed, the linear equation is converted into a matrix, the matrix is solved by a least square method, and an affine transformation matrix between adjacent frames is constructed;

[0048] According to the affine transformation matrix, an accumulated transformation matrix is obtained;

[0049] An affine transformation matrix of the first two rows of the accumulated transformation matrix is extracted, and an inverse transformation matrix is obtained by inverting the affine transformation matrix;

[0050] The current frame is subjected to the inverse transformation matrix, the current frame is corrected to the reference view angle of the first frame, and a corrected video frame is output.

[0051] In a second aspect, an embodiment of the present application provides a video jitter correction system based on high-altitude view, the system comprising: a feature point set module, a feature point trajectory set module, a rectangular coordinate system module, a clustering module and a correction module; wherein:

[0052] The feature point set module is configured to extract feature points from the preprocessed detection frame, optimize the distribution of the feature points, and obtain a feature point set.

[0053] The feature point trajectory set module is configured to track feature point trajectories frame by frame according to a motion tracking rule, and obtain a feature point trajectory set.

[0054] The rectangular coordinate system module is configured to generate a polar coordinate system comprising a plurality of feature points according to the motion trajectories of the feature points in the feature point trajectory set, and convert the polar coordinate system into a rectangular coordinate system.

[0055] The clustering module is configured to cluster the feature points in the rectangular coordinate system by a density clustering algorithm, and obtain a target feature point set.

[0056] The correction module is configured to calculate a transformation relationship between adjacent frames according to the target feature point set, perform inverse affine transformation on a current frame based on the transformation relationship, and output a corrected video frame.

[0057] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a video jitter correction method based on high-altitude view according to the first aspect.

[0058] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the high-altitude perspective-based video shake correction method according to the first aspect.

[0059] The high-altitude perspective-based video shake correction method and system provided by the embodiments of the present application have at least the following technical effects.

[0060] The embodiments of the present application extract feature points from the preprocessed detection frame and perform distribution optimization, ensure that the feature points uniformly cover the effective area, avoid local concentration, track the trajectory frame by frame combined with the motion tracking rule, obtain the feature point trajectory set, and can accurately capture the feature point motion trajectory. The trace data is converted into a polar coordinate system and further converted into a rectangular coordinate system, so as to adapt to the requirements of the input of the clustering algorithm. Then, the stable feature point set is screened through density clustering, which can automatically screen out the most stable feature point set in the scene, effectively avoid the interference of dynamic targets (such as vehicles and crowds) on motion estimation, and improve the correction accuracy and robustness in complex dynamic environment. Finally, the transformation relationship between adjacent frames is calculated based on the target feature point set, and the current frame is inversely transformed based on the transformation relationship, and the corrected video frame is output. The transformation relationship and inverse transformation matrix technology are adopted to realize the global stable alignment of all frames relative to the first frame, which can effectively eliminate the problem that the slight shake in high-altitude shooting is greatly amplified, and significantly improve the picture stability. The high-altitude perspective-based video shake correction method in the related art has the problem of low correction accuracy.

[0061] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0062] The drawings described herein are intended to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0063] Figure 1 is a flowchart of a high-altitude perspective-based video shake correction method;

[0064] Figure 2 is a flowchart of step S102 according to an exemplary embodiment;

[0065] Figure 3 is a schematic diagram of a polar coordinate system according to an exemplary embodiment;

[0066] Figure 4 is a schematic diagram of a rectangular coordinate system according to an exemplary embodiment;

[0067] Figure 5 is a flow chart of step S104 according to an exemplary embodiment

[0068] Figure 6 is a k-distance graph according to an exemplary embodiment;

[0069] Figure 7 is a smoothed k-distance graph according to an exemplary embodiment;

[0070] Figure 8 is a schematic diagram of density clustering results according to an exemplary embodiment;

[0071] Figure 9 is a structural block diagram of a video shake correction system based on high-altitude perspective according to an exemplary embodiment;

[0072] Figure 10 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0073] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0074] Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those of ordinary skill in the art, the present application can be applied to other similar scenarios without creative efforts based on these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacture or production changes based on the technical content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the present application.

[0075] In the present application, "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0076] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terms "a", "an", "one", "this", and similar referents in the context of describing the application are to be construed to be inclusive, both singular and plural, unless otherwise indicated. The terms "comprising", "comprises" and "comprised of" as well as conjugations thereof, are used synonymously with the term "including" or "containing" or "having" and are inclusive or open-ended and do not exclude additional, unrecited elements or method steps. The terms "connected", "coupled", "attached", and the like, unless otherwise defined, are used in the context of the present application to mean the physical, electrical, and / or magnetic coupling of one element to another element. The term "plurality" means two or more. The term "and / or" describes associating objects and means that three separate "and" or "or" cases can exist. The character " / " is generally used to mean "or". The terms "first", "second", "third", etc. are used only to distinguish similar objects, and do not represent a specific order or sequence.

[0077] With the wide application of mobile shooting devices, unmanned aerial vehicles and high-altitude surveillance cameras, video jitter problems are increasingly prominent, seriously affecting the picture quality and the accuracy of subsequent intelligent analysis. In particular, under high-altitude view, a small jitter will be greatly magnified. For example, when shooting at an altitude of 120 meters, a 1-pixel error in the position of a target vehicle can result in a transient speed error of up to 9.8 km / h. Therefore, high-precision video jitter correction technology is of great significance for unmanned aerial vehicle aerial photography, traffic monitoring and other application scenarios.

[0078] Currently, mainstream video anti-shake and jitter correction technologies mainly include the following categories:

[0079] 1. Hardware anti-shake method based on sensors. This method integrates gyroscopes, accelerometers and other inertial measurement units (IMU) to sense the motion state of the device in real time and drive the gimbal or lens to make physical compensation. Its advantages are fast response speed and no dependence on image content, and it is commonly used in high-end cameras, unmanned aerial vehicles and other devices. However, this method has limitations such as high hardware cost, complex system integration, certain requirements for device size and power consumption, etc.

[0080] 2. Digital anti-shake method based on image content. This method directly analyzes the video frame sequence, estimates the global or local motion through image processing and computer vision algorithms, and performs digital correction on the picture. Common techniques include:

[0081] Global motion estimation: Estimate the inter-frame affine, perspective or homography transformation based on feature point matching (SIFT, SURF, ORB, etc.) or block matching, to realize overall picture stabilization.

[0082] Motion compensation based on optical flow: Estimate pixel-level motion using sparse or dense optical flow, suitable for handling complex jitter and local deformation.

[0083] Deep learning-based methods: In recent years, convolutional neural networks (CNN), temporal networks (RNN), etc. have been used for end-to-end learning of video stabilization mapping, which can handle non-linear and complex scenarios, but requires high data and computing power.

[0084] 3. Hybrid stabilization method. Combine hardware sensors and image analysis, use IMU data to provide preliminary motion estimation, and then use image content to optimize correction effect, balancing real-time and accuracy, but system design and calibration are more complex.

[0085] Although the above methods have achieved certain results in different scenarios, there are still the following shortcomings:

[0086] 1. Strong dependence on external hardware, increasing device cost and maintenance difficulty;

[0087] 2. Sensitive to scene and environmental parameters, such as accurate calibration or dependence on static background, limited adaptability;

[0088] 3. Susceptible to dynamic target interference, such as traffic, crowd and other moving objects that can affect feature point tracking and motion estimation, leading to correction failure or picture distortion;

[0089] 4. Some methods lack adaptability to high-altitude large field of view and complex jitter types, making it difficult to meet the high-precision application requirements of unmanned aerial vehicles and other platforms.

[0090] In summary, there is an urgent need to develop a new video jitter correction method that only relies on visual information, can adaptively identify stable regions, is robust to dynamic targets, and is suitable for high-altitude large field of view scenarios, to improve the accuracy of video analysis on platforms such as unmanned aerial vehicles and high-altitude cameras, and to solve the problem of low correction accuracy in related technologies based on high-altitude video jitter correction methods.

[0091] Based on the above, the embodiments of the present application provide a video jitter correction method and system based on high-altitude view. To solve the above problems.

[0092] In this paper, it should be understood that the terms involved can be technical means or other summary technical terms for implementing part of the invention, for example, the terms can include:

[0093] Motion tracking rule: The tracking criterion of feature points between frames established by Lucas-Kanade (LK) sparse optical flow algorithm and other algorithms, including three basic assumptions of constant brightness, small motion and neighborhood consistency, used to ensure the continuity and reliability of the feature point trajectory.

[0094] Polar coordinate system: The coordinate system that takes the initial position of the feature point as the origin and uses the displacement distance (r) and the motion direction angle (θ) to describe the trajectory, which can directly reflect the motion amplitude and direction characteristics and is convenient for subsequent clustering analysis.

[0095] Cartesian coordinate system: The coordinate system that converts the (r, θ) of the polar coordinate into the two-dimensional plane coordinate (x, y), which is used to adapt to the input requirements of the density clustering algorithm.

[0096] Density clustering algorithm: The unsupervised learning method based on neighborhood density represented by DBSCAN (Density-Based Spatial Clustering of Applications with Noise), which divides clusters by the density distribution characteristics of data points and can identify high-density regions of any shape and automatically filter noise points in low-density regions.

[0097] Autocorrelation matrix: The matrix calculated from the image gradient in the Shi-Tomasi corner detection, whose eigenvalue determines the corner response degree and is used to evaluate the stability of the local image structure.

[0098] Pyramid optimization algorithm: A multi-scale image processing technology that optimizes the calculation efficiency and accuracy by constructing a multi-layer representation of the image (from a high-resolution original image to a low-resolution approximate image). It solves the tracking failure problem caused by large displacement and is particularly suitable for fast motion scenes under the high-altitude view of the unmanned aerial vehicle.

[0099] Affine transformation matrix: Describes the rotation, scaling and translation transformation between adjacent frames, which is obtained by fitting the corresponding relationship of feature points by the least square method.

[0100] Cumulative transformation matrix: A 3x3 homogeneous coordinate matrix generated by continuously multiplying the affine transformation matrix between frames, which records the global motion trajectory from the first frame to the current frame and is used for reference calculation of inverse correction.

[0101] Inverse transformation matrix: The inverse matrix of the cumulative transformation matrix, which is used to map the current frame picture back to the view angle of the initial reference frame to realize the elimination of jitter.

[0102] In a first aspect, the embodiments of the present application provide a video jitter correction method based on a high-altitude view, Figure 1 A flowchart of a video jitter correction method based on a high-altitude view is shown in Figure 1 The method comprises:

[0103] Step S101, extracting feature points from the pre-processed detection frame, optimizing the distribution of the feature points, and obtaining a feature point set.

[0104] Step S102, tracking the feature point trajectories frame by frame through a motion tracking rule, and obtaining a feature point trajectory set.

[0105] Step S103, generating a polar coordinate system including a plurality of feature points according to the motion trajectories of the feature points in the feature point trajectory set, and converting the polar coordinate system into a rectangular coordinate system.

[0106] Step S104, clustering the feature points in the rectangular coordinate system through a density clustering algorithm, and obtaining a target feature point set.

[0107] Step S105, calculating a transformation relationship between adjacent frames according to the target feature point set, performing inverse transformation matrix on the current frame based on the transformation relationship, and outputting a corrected video frame.

[0108] To sum up, the embodiment of the present application provides a video jitter correction method based on a high-altitude perspective. The method does not need to rely on external hardware sensors, but only uses visual information of the video itself. Through consistent analysis and abnormal elimination of the motion trajectories of the inter-frame feature points, stable feature points of the background static region are automatically selected, and then high-precision estimation and compensation of video jitter are realized. For the application scenarios of high-altitude perspective, large field of view, and rich dynamic targets such as unmanned aerial vehicles and high-altitude ball machines, the present application can effectively suppress the picture instability problem caused by factors such as device jitter and wind disturbance, and significantly improve the stability of the video and the accuracy of subsequent intelligent analysis (such as target detection and speed identification). The video jitter correction method based on a high-altitude perspective in the related art has the problem of low correction accuracy.

[0109] Before step S101, extracting feature points from the pre-processed detection frame, the method comprises:

[0110] Step one, taking the first frame of the video as the first frame. If the displacement between the first frame and the next frame exceeds the preset displacement threshold, or the deflection angle between the first frame and the next frame exceeds the preset deflection angle threshold, then the detection frame is determined through video editing or frame selection.

[0111] Optionally, the initial frame of the first detected corner point needs to be found first. Generally, the first frame of the video is the detection frame by default. If it is found that the video has a large displacement or rotation (the unmanned aerial vehicle starts recording the video before it is fixed) at the beginning (between the first frame and the next frame), the video can be edited or the subsequent video frame can be selected as the detection frame.

[0112] Step two, analyzing and recording the frame width, frame height, frame rate, and total frame number of the video through a video decoding module.

[0113] Optionally, after determining the detection frame, the video file is opened and read using a video decoding module (such as OpenCV's VideoCapture class) to ensure that the video data can be accessed frame by frame. Basic video parameters, including frame width, frame height, frame rate (fps), and total number of frames (N), are parsed and recorded to provide a basis for subsequent processing and output.

[0114] Step 3: Preprocess the detection frames, including converting the detection frames into grayscale images.

[0115] Optionally, since subsequent feature point detection and sparse optical flow methods both require grayscale images, the detection frames in the video stream are converted into grayscale images. Let the original color frame be ( Then the grayscale frame is .

[0116] Step 4: Initialize the feature point set based on the grayscale image. With trajectory set , where the set of feature points Includes the unique ID and initial coordinates of each feature point. This represents the nth feature point, and the initial feature point set is backed up. Trajectory set It includes inter-frame motion trajectory data for each feature point, including the frame number, coordinates, displacement distance, and direction of motion of the current trajectory point.

[0117] Optionally, a set of feature points is defined based on the grayscale image. , representing a total of n feature points. This represents the i-th feature point, and the initial feature point set is backed up. Define the set of feature point trajectories. , It represents The complete trajectory of the feature points, each t is a list, storing the trajectory information of the i-th feature point from the first frame to the last frame of the video. Table 1 shows the data structure fields for storing feature points, and Table 2 shows the data structure fields for storing feature point trajectories.

[0118] Table 1: Data Structure Fields for Storing Feature Points

[0119]

[0120] Table 2: Data Structure Fields for Storing Feature Point Trajectories

[0121]

[0122] The steps one to four above, by detecting the displacement and deflection of the first frame and the secondary frame (to avoid subsequent correction failure caused by initial jitter), combining video parameter analysis (to ensure that the processing flow matches the video specification), converting the detection frame into a gray image (to adapt to the input of the feature point detection algorithm), and initializing the feature points and the trajectory set (to provide a structured data basis for motion tracking), can effectively avoid the interference of unstable frames, ensure the compatibility of subsequent algorithms for videos of any resolution, improve the efficiency and accuracy of trajectory analysis through standardized data storage, and lay a foundation for high-robustness video stabilization.

[0123] In an embodiment, step S101, feature points are extracted from the preprocessed detection frame, the feature points are distributed and optimized, and a feature point set is obtained. Specifically, the following steps are included:

[0124] Step S1011, the autocorrelation matrix of each pixel point in the gray image is calculated by a feature point detection algorithm, the corner point response degree is determined according to the minimum eigenvalue based on the autocorrelation matrix, and the corner point response degree is sorted from high to low, and the pixel points of a preset rank are selected as the initial feature point set.

[0125] Optionally, on the gray image of the detection frame, a Shi-Tomasi corner point detection algorithm is used to extract potential feature points. The algorithm calculates the eigenvalues of the structure tensor by analyzing the gray scale changes of the local window of the image, and selects the pixel points with the minimum eigenvalue greater than a set threshold as the corner points.

[0126] First, the detection image is converted into a gray image. For each pixel point (x, y), the Sobel operator is used to calculate the gradient in the x and y directions to obtain the amplitude A of the gradient and the direction B of the gradient. For each pixel point (x, y), an autocorrelation matrix M is calculated.

[0127]

[0128] is a window function used for weighted calculation of the contribution of the local area. is the local displacement (offset) of the pixel point in the image, is the brightness value of the image at position . Since there is no weight distinction for the detection of corner points in the background of the video under high-altitude view, the window function of the present application is selected as a rectangular window.

[0129] The eigenvalues λ1 and λ2 of the autocorrelation matrix M are calculated, and the Shi-Tomasi response function is:

[0130]

[0131] Compared with the Harris corner point detection algorithm, Shi-Tomasi is more stable when calculating the corner response. The corner responses of all pixels are sorted, and the corner points with preset rankings are selected as the initial feature point set, for example, the top 5 corner points are selected as the final corner point result.

[0132] In step S1012, the initial feature point set is optimized according to the imaging characteristics of the high-altitude view angle, by the maximum number of corner points, the corner quality threshold, the minimum distance between corner points, and the detection window size.

[0133] Optionally, according to the characteristics of small ground targets and sparse texture under the high-altitude view angle, parameter settings are required, including the maximum number of corner points (max Corners), the corner quality threshold (quality Level), the minimum distance (minDistance), and the window size (block Size).

[0134] Specifically, the quality Level is appropriately reduced, the max Corners value is increased, the minDistance is adjusted according to the actual scene to ensure that enough uniformly distributed corner points are detected, and the block Size value is increased to make the algorithm more sensitive to larger area features.

[0135] Through testing, the quality Level (corner quality threshold) is limited to between 0.01 and 0.1, which can detect a sufficient number of corner points under the 120m unmanned aerial vehicle view angle for subsequent processing, the max Corners (maximum number of detected corner points) can be set to between 50 and 100, and the present application can be selected as 50 or 100 (in fact, 10 or so different position corner points can achieve correction), the block Size is set to 5-10, and the present application can be selected as 7. Due to the characteristics of sparse texture, the minDistance (minimum distance between corner points (pixels)) setting needs to be determined according to the resolution, and the following is the determination formula:

[0136]

[0137] In the formula, w is the width and h is the height.

[0138] In step S1013, the optimized initial feature point set is distributed optimized, wherein the distribution optimization includes: dividing the gray image into a plurality of image blocks, each image block including at least one feature point; removing invalid feature points in the sky and shadow area; and retaining the number of static feature points of the ground and building area and the number of feature points of dynamic targets according to the priority.

[0139] Optionally, since the video taken from high-altitude perspective often contains large areas of sky, roads, buildings and other regions, the texture of some regions is single and the corner points are few. In order to improve the robustness of subsequent tracking and motion estimation, the detected corner points should be distributed and optimized. Specifically, it includes:

[0140] Spatially block the corner points to ensure that there are corner points in each region, avoiding too many corner points in a local area.

[0141] Remove the corner points located in invalid areas such as sky and shadow, which can be combined with a simple image or region or mask method (preprocessed by code).

[0142] For static background regions such as ground and buildings, prefer to keep the corner points and reduce the number of corner points on dynamic targets (such as vehicles and pedestrians).

[0143] Step S1014, the initial feature point set after distribution optimization is taken as a feature point set, wherein the feature point set includes ID numbers of multiple feature points and coordinate information in the detection frame.

[0144] Optionally, each corner point is assigned a unique ID, and its coordinate information in the detection frame is recorded, and the ID and coordinate information are recorded in the feature point set . At the same time, the initial position of the corner point is taken as the starting point of subsequent trajectory tracking, and the feature point trajectory set is initialized , providing a data basis for subsequent feature point tracking and outlier rejection. All corner point information and initial trajectory information are stored in a structured manner, facilitating batch processing and analysis in subsequent steps.

[0145] Step S101 filters high-stability corner points through autocorrelation matrix and minimum eigenvalue response. It can improve the quality of feature points, optimize the detection parameters combined with the characteristics of high-altitude perspective, adapt to sparse texture scenes, and then evenly distribute and filter the regions to ensure that the feature points cover the effective static regions and remove the interference. Finally, a high-robustness feature point set is constructed, so that the feature points are evenly distributed under high altitude, effectively reducing dynamic targets, providing reliable input of static background for subsequent motion tracking, and improving the anti-shake accuracy.

[0146] Figure 2 is a flowchart of step S102 shown according to an example embodiment, as Figure 2 shown, step S102, tracks the feature point trajectory frame by frame through the motion tracking rule to obtain a feature point trajectory set. Step S102 specifically includes the following steps:

[0147] Step S1021, define the window size, pyramid layer number and iteration termination condition parameters.

[0148] Optionally, in order to make these feature point sets The motion of these corner points (pixels) needs to be captured by some calculation method, so Lucas-Kanade (LK) sparse optical flow method is selected for feature point tracking. LK optical flow method is suitable for sparse feature point tracking, has high calculation efficiency, and is suitable for resource-limited platforms such as unmanned aerial vehicles.

[0149] Optical flow is a kind of motion mode, which refers to the apparent movement of an object between an observer and the background at a certain angle. LK optical flow calculation evaluates the deformation between two images, which is based on three assumptions of the optical characteristics of object movement: 1. constant brightness, the appearance of the pixels of the target in the image scene remains unchanged when the target moves between frames. Used to obtain the basic equation of the optical flow method; 2. time continuity or small motion, the movement of the image over time is relatively slow. 3. consistent optical flow in the field. The same surface in a scene has similar motion in the local field, and the projection on the image plane is also in the adjacent area, and the velocities of adjacent points are consistent.

[0150] For high-altitude view (unmanned aerial vehicle / high-altitude ball machine, etc.), the brightness of the background of the video depends on the weather factor, and the weather almost does not change in a short time (two frames), so it meets assumption 1. The time continuity of the video shot by the unmanned aerial vehicle meets assumption 2. For the video shot by the unmanned aerial vehicle, the surface optical flow of the static background (buildings, sky and roads, etc.) is consistent, so it meets assumption 3. Therefore, the LK optical flow method can effectively track the feature point set Motion trajectory of each frame.

[0151] The main parameters include window size (win Size), pyramid layer number (max Level), iteration termination condition (criteria), etc. For the characteristics of small target size, sparse background texture and possibly large motion service under the high-altitude view of the unmanned aerial vehicle, the window size and the pyramid layer number need to be increased to improve the adaptability to large-scale motion and the tracking robustness. They can be set as:

[0152]

[0153] Step S1022, position prediction and tracking of all feature points of the previous detection frame are performed by the sparse optical flow method to obtain the position of each feature point in the current frame.

[0154] Optionally, in the video sequence, for each frame, the position prediction and tracking of all valid corner points of the previous frame are performed by the LK sparse optical flow method to obtain the new position of each corner point in the current frame.

[0155] Assume that the feature point set A pixel point in the feature point set The pixel coordinate of the current frame is (x, y), the brightness of the point at this time t is I(x, y), and the velocity components of the optical flow in the horizontal and vertical directions are represented by u(x, y) and v(x, y).

[0156] After a time interval , the brightness of the corresponding point of the point becomes Under the premise that the motion is small, the Taylor formula is expanded:

[0157]

[0158] When is small enough, it approaches 0:

[0159]

[0160] The sparse optical flow assumption is that the brightness is constant in a small local area of (u, v), so: , which is simplified as: , and the least squares method is used:

[0161]

[0162] The final optical flow (velocity vector) obtained by solving is:

[0163]

[0164] The matrix form is as follows:

[0165]

[0166] The final obtained is the optical flow corresponding to the LK algorithm. By using the corner point of the previous frame prev_gray, the corresponding point in the current frame gray can be found.

[0167] Step S1023, based on the defined window size, pyramid layer number and iteration termination condition parameters, the trajectory set of each feature point is obtained through the pyramid optimization algorithm.

[0168] Optionally, in the high-altitude perspective, the background is mainly composed of static ground buildings, roads and dynamic pedestrians and vehicles, in the corner point detection of step S1023, the parts of the buildings, roads and vehicles are often taken as the corner points, and the vehicles often have a high motion speed in the video. However, the constraint condition of the LK algorithm is: small speed, constant brightness and region consistency. However, when the object motion speed is high, the small speed assumption is not established, and the algorithm has a relatively large error. At this time, the motion speed of the object in the image needs to be considered. The most intuitive method used by the present application is to reduce the size of the image (pyramid method).

[0169] Specifically, the main steps of the pyramid method are: establishing a pyramid, tracking based on the pyramid, and iteration process. First, an image pyramid is established for each frame image, and the original image is at the bottom and the lowest resolution is at the top layer;

[0170] The optical flow of the top layer (Lm layer) is calculated, and the optical flow of each point in the top layer image is obtained by minimizing the matching error sum in the field range of each point:

[0171]

[0172] The size of the image is defined as half of the original size each time, and the Lm layer is scaled, and the 0th layer is the original image. Assuming that the displacement of the original image is d, the displacement of each layer is :

[0173]

[0174] The calculation result (displacement condition) of the optical flow of the top layer is fed back to the Lm-1 layer as the estimation g of the initial optical flow value of the layer.

[0175] The feedback is repeated along the pyramid until the bottom layer (original image) of the pyramid.

[0176] For each layer L, the calculation of the optical flow of each point is based on the minimization of the matching error sum of all points in the neighborhood.

[0177] For example, the general speed limit of urban roads is 30-60km / h, and under 120m high, the instantaneous speed of 1 pixel / frame corresponds to the actual vehicle speed of 9.8km / h. In order to make the vehicle motion speed as observed as possible, taking the (1920, 1080) resolution video as an example, the number of pyramid layers is controlled to be 3-5, that is, the pixel point is scaled by 4-16 times to control the vehicle movement speed of each frame to be 1-2 frames. The corner points in the image can be successfully tracked and recorded to the feature point trajectory set .

[0178] Steps S1021-S1023 adapt to the high-altitude large displacement characteristics by predefining the optical flow calculation window and the number of pyramid layers, realize the cross-frame position prediction of the feature points by using the sparse optical flow method, ensure the trajectory continuity, and further combine the pyramid layer optimization to compress the fast motion to the trackable range, thereby significantly improving the robustness and accuracy of the trajectory tracking.

[0179] In an embodiment, step S103 generates a polar coordinate system including a plurality of feature points according to the motion trajectories of each feature point in the feature point trajectory set, and converts the polar coordinate system into a rectangular coordinate system. Specifically, the following steps are included:

[0180] Step S1031 extracts the first frame coordinates and the last frame coordinates of the plurality of feature points according to the overall motion trajectories of each feature point in the feature point trajectory set.

[0181] Optionally, the determination criteria of the corner point abnormal point are determined first. For the high-altitude view video, the abnormality of the corner point is mainly determined according to the trajectory motion direction and the displacement distance, for example, the high-altitude shooting of the road.

[0182] For the unmanned aerial vehicle shooting video, the main background is composed of geographical buildings, roads and vehicles and pedestrians, which makes the types of corner points can be divided into static corner points (buildings, road signs, etc.) and dynamic corner points (vehicles, pedestrians, etc.). Obviously, for the reference of video correction, it is necessary to use static corner points as reference as much as possible and exclude the interference of dynamic corner points.

[0183] Since the unmanned aerial vehicle will be slightly deviated due to the movement of the vehicle body itself, wind influence and other problems, the absolute pixel movement distance of the corner point cannot be directly used as the judgment standard. Therefore, the relative movement between the static corner point and the dynamic corner point is selected as the reference, and the basis for judging the relative movement of the corner point is the trajectory motion direction and the displacement distance of the corner point.

[0184] Trajectory motion direction: the angle between the connecting line of the corner point from the initial frame to the current frame and the standard coordinate system.

[0185] Displacement distance: refers to the cumulative Euclidean distance of the corner point from the initial frame to the current frame.

[0186] Based on the complete motion trajectory of each feature point in the feature point trajectory set, the first frame coordinates and last frame coordinates of multiple feature points are extracted. In a Cartesian coordinate system, with the y-axis representing the trajectory motion direction and the x-axis representing the displacement distance, the corner point dataset P is set as follows: each Coordinates .

[0187] Step S1032: Determine the displacement distance based on the coordinates of the first frame and the last frame, determine the motion direction based on the directional angle between the coordinates of the first frame and the last frame, and generate a polar coordinate system including multiple feature points based on the displacement distance and the motion direction.

[0188] Optionally, for each corner point Calculate its value from the initial frame To the last frame The displacement distance (r) and the direction of motion ( ), and expressed in polar coordinates, assuming that at the corner point is The pixel coordinates are ( ),exist The pixel coordinates are ( Then (taking the first quadrant as an example):

[0189] Suppose we have a set of detected corner points. For example, when filming road videos from the perspective of the National College Entrance Examination (Gaokao), the video includes corner points of buildings and corner points of vehicles and pedestrians. A polar coordinate system is drawn based on this. Figure 3 This is a schematic diagram of a pole coordinate system according to an exemplary embodiment, such as... Figure 3 As shown, the polar coordinate system intuitively reflects the motion amplitude and directional characteristics of each corner point.

[0190] Step S1033: Convert the polar coordinate system to a rectangular coordinate system according to the mapping relationship between the polar coordinate system and the rectangular coordinate system. Specifically, this includes:

[0191] The displacement distance between the first frame coordinates and the last frame coordinates is calculated using Euclidean distance.

[0192] Calculate the ratio of the vertical displacement to the displacement distance between the coordinates of the first frame and the coordinates of the last frame. Perform arcsine processing based on the ratio to obtain the direction angle. Determine the direction of motion based on the direction angle.

[0193] Based on displacement distance and motion direction, the polar coordinate system is converted to a rectangular coordinate system through trigonometric function relationships. In the rectangular coordinate system, the y-axis represents the trajectory motion direction, and the x-axis represents the displacement distance. The feature point dataset P is set as follows. each Coordinates .

[0194] Optionally, the displacement distance and the motion direction are calculated by the following formula:

[0195]

[0196] For subsequent clustering analysis, the features in the polar coordinate system are converted into the rectangular coordinate system , which is shown by the following formula:

[0197]

[0198] Figure 4 The schematic diagram of the rectangular coordinate system is shown according to an example embodiment, as shown in Figure 4 The shooting device for high-altitude view is usually provided with a wide-angle lens, and mirror image distortion (barrel / pillow) is prone to occur at the edge of the picture, which causes the geometric position of the edge pixels to be inconsistent with the real position, so that the error of motion estimation is larger, and abnormal points are prone to be introduced. Therefore, the final trajectory of the corner point is defined to allow a certain deviation of the motion direction and the displacement distance .

[0199] Steps S1031-S1033 calculate the displacement distance and the motion direction by extracting the feature point coordinates of the first and last frames, construct the polar coordinate system to intuitively represent the motion mode, distinguish the static background from the dynamic target, and then convert it into the rectangular coordinate system to adapt to the input requirements of the subsequent clustering algorithm, thereby improving the accuracy of motion analysis.

[0200] Figure 5 The flowchart of step S104 is shown according to an example embodiment, as shown in Figure 5 Step S104, the feature points in the rectangular coordinate system are clustered by a density clustering algorithm to obtain a target feature point set. Specifically, the following steps are included:

[0201] Step S1041, according to the density distribution of the feature points in the rectangular coordinate system, the neighborhood radius parameter of the density algorithm and the minimum point number parameter of the core points in the neighborhood radius are determined. Specifically, the following steps are included:

[0202] Step one, calculate the distance from the feature point coordinates in the rectangular coordinate system to the k-nearest neighbor points, sort according to the distance from large to small, and generate a distance curve, wherein the k-nearest neighbor points are obtained by a k-distance function;

[0203] Step two, smooth the distance curve by moving average processing, and identify the distance value corresponding to the maximum inflection point as the neighborhood radius parameter;

[0204] Step three, determine the minimum point number parameter of the core points in the neighborhood radius based on the k-nearest neighbor points. ​

[0205] Optionally, as can be known from the above step S103, high-altitude perspective video shooting often has problems such as mirror distortion and unmanned aerial vehicle shaking, and the corner point dataset is often a non-convex dataset, and the DBSCAN clustering method often performs better than the K-means clustering and Birch hierarchical clustering methods on the non-convex dataset, and the DBSCAN can also be used as an anomaly detection algorithm to find noise points.

[0206] The DBSCAN algorithm mainly has two parameters: Eps (neighborhood radius) and MinPts (minimum number of points in the neighborhood radius of a core object). The Eps neighborhood of a point p is :

[0207]

[0208] A core object refers to that if the number of sample points in the Eps neighborhood of a point p is greater than or equal to MinPts, the object is called a core object.

[0209] Before starting the DBSCAN algorithm, appropriate values of the two parameters Eps and MinPts need to be selected. For the selection method of the neighborhood radius Eps, we use the K-distance function to select the value. First, the value of k needs to be selected, and k is usually taken as:

[0210]

[0211] The characteristic number of the present application can be 2, so k=3 is selected. The distance from each point to its Kth nearest point is calculated, and then the distances are sorted from large to small and plotted, and finally a larger distance with a larger elbow point is selected as the value of Eps. Taking the scene of step S104 as an example, Figure 6 is a k-distance curve diagram according to an example embodiment, as Figure 6 shown, the dashed line is the elbow point, and the straight line is the k-distance curve. After drawing the k-distance curve diagram (elbow diagram), the elbow point is determined by first-order difference finding the maximum inflection point, and manual observation can also be used for judgment.

[0212] However, since the video samples of the present application are usually non-uniform samples, the drawn curve has too many twists and turns, resulting in low accuracy of the inflection point judgment, so when using the k-distance function to judge, a layer of smoothing processing is needed, and the moving average smoothing method is used to make the curve smoother to judge the effective inflection point: Figure 7 is a smoothed k-distance curve diagram according to an example embodiment, as Figure 7As shown, k = 3, the smoothing window = 15, the solid line is the original k-distance curve, the long dashed line is the curve after smoothing (15), the dotted line is the 3rd largest inflection point, and eps is about 227.725. After smoothing, Eps can be determined as 227.725. In actual detection, the effect of the largest inflection point as Eps is not necessarily the best. A plurality of inflection points can be calculated at one time, and the inflection point close to the middle point of the curve is sometimes better. After being put into application, the same type of video can be uniformly processed, Eps is determined first, and then the same Eps is used to calculate the video samples of the same type. Eps can also be determined by manual observation, such as when it is found that half of the inflection points in the actual video are dynamic corner points, the median of the curve is selected as the inflection point.

[0213] The value of MinPts is k + 1, that is:

[0214]

[0215] The range of MinPts in the application can be 3-6. It can be selected as 4 or 5. After the two parameters Eps and MinPts are determined, DBSCAN clustering begins.

[0216] Step S1042, based on the neighborhood radius parameter and the minimum point number parameter, performing density clustering on the feature points in the rectangular coordinate system through density algorithm reachability. Specifically, the following steps are included:

[0217] Step one, according to the neighborhood radius parameter and the minimum point number parameter, traversing each feature point in the rectangular coordinate system;

[0218] Step two, if the number of points in the neighborhood radius of the current feature point is greater than or equal to the minimum point number parameter, the feature point is marked as a core point; if the number of points in the neighborhood radius of the current feature point is less than the minimum point number parameter, the current feature point is a noise point;

[0219] Step three, if the current feature point is a core point, a new cluster is created, and all feature points in its neighborhood are added to the candidate set;

[0220] Step four, through iterative traversal of the candidate set, for the feature points in the candidate set that have not been assigned a cluster, marking them as visited, and expanding the candidate set according to the core point, until the current cluster meets the expansion criteria, to obtain the clustering result.

[0221] Optionally, the clustering principle of DBSCAN is the maximum density connected sample set derived from density reachability, that is, a cluster . Any DBSCAN cluster has at least one core object core, and if the number of core objects is greater than one, these core objects must be able to form a density reachable sequence .

[0222] After the calculation steps of step S104 and step S1041, the two-dimensional feature data of the corner point set P and two DBSCAN parameters: Eps and MinPts can be obtained. DBSCAN clustering is performed on the corner points.

[0223] The calculation step is specifically:

[0224] 1. Given the neighborhood radius (Eps) and the minimum number of points (MinPts) in the neighborhood radius that become core objects, select Eps as 60-90 and MinPts as 3-6 under the sample video.

[0225] 2. Start from an arbitrary point p, mark it as “marked”, check whether it is a core point, if not, mark it as a noise point. Otherwise, create a new cluster for p , and put all objects in the Eps domain of p into the candidate set .

[0226] 3. Iteratively add objects in N that do not belong to other clusters to C, in the process, for the objects in N marked as “unmarked” , mark it as “marked”, and check its Eps neighborhood, if it is also a core object, then the objects in the Eps domain of p are added to N. Continue to add objects to C until C cannot be expanded.

[0227] 4. Randomly select the next access object from the remaining objects, repeat step 3 until all objects are accessed.

[0228] Step S1043, according to the result of density clustering, obtain the main density cluster including the core points as the target feature point set.

[0229] Optionally, Figure 8 is a schematic diagram of the density clustering result according to an exemplary embodiment, as Figure 8 shown, according to the DBSCAN clustering result, the corner points belonging to the main cluster (squares (cluster 0) in the figure) are selected as the stable feature point set , the equilateral triangle cluster 1, and the inverted triangle is cluster 2, and x is noise.

[0230] Steps S1041-S1043 group the feature points in the rectangular coordinate system by the density clustering algorithm, automatically distinguish the static background (cluster 0) and the dynamic target (cluster 1 and cluster 2). First, the optimal neighborhood parameter is determined based on the k-neighbor distance analysis, which can adapt to the density changes of different scenes, and then the stable feature point cluster is constructed through the core point expansion mechanism to ensure the motion consistency. Finally, the static background feature points are accurately identified in the complex dynamic scene. Through the density analysis of the corner feature points by the DBSCAN clustering algorithm, the most stable feature point set in the scene can be automatically screened out, the interference of the dynamic target (such as vehicles and crowds) on the motion estimation is effectively avoided, and the correction accuracy and robustness in the complex dynamic environment are improved.

[0231] In an embodiment, step S105, according to the target feature point set, the transformation relationship between adjacent frames is calculated, and based on the transformation relationship, the inverse transformation matrix of the current frame is output, and the corrected video frame is output. Specifically, the following steps are included:

[0232] Step S1051, according to the corresponding relationship between the current frame feature point coordinates and the previous frame feature corner point coordinates in the target feature point set, a linear equation is constructed, the linear equation is converted into a matrix, the matrix is solved by the least square method, and an affine transformation matrix between adjacent frames is constructed.

[0233] Optionally, through step S104, the stable feature point set can be obtained In order to estimate the affine transformation between frames, the feature point set coordinates Corner_curr and Corner_prev of the current frame and the previous frame are needed, and the data format of Corner_curr is:

[0234] In which represents the current frame coordinate of the i-th stable corner point, and the Corner_prev set is similar.

[0235] Suppose an affine transformation matrix (M) is a 2*3 floating point matrix, which represents the affine transformation from the original point set to the target click. The first two columns of the matrix are the rotation and scaling parts, and the last column is the translation part.

[0236] According to the input point set Corner_curr and Corner_prev, a linear equation system is constructed. For each point pair (cPoint, pPoint), the following two equations need to be constructed:

[0237]

[0238] After the construction, the linear equation system needs to be converted into a matrix form A*X=B, where: A is a 2N*6 matrix, N is the number of point pairs, i.e. the number of corner points. Each row of A corresponds to a point pair, containing the coordinates of the original points and a constant term. X is a 6*1 matrix, representing the parameters of the affine transformation matrix to be solved. B is a 2N*1 matrix, containing the coordinates of the target points.

[0239] Solve the matrix X using the least squares method to construct the estimated affine transformation matrix M:

[0240]

[0241] The goal of the least squares method is to find an optimal affine transformation matrix that makes the original point set as close as possible to the target point set after transformation. By minimizing the sum of the squares of the residuals, an optimal estimation result can be obtained.

[0242] Step S1052, obtain the cumulative transformation matrix according to the affine transformation matrix.

[0243] Optionally, assuming that the video correction takes the first frame as the reference target, all subsequent frames need to be corrected to the perspective of the first frame. However, the inter-frame affine transformation matrix only takes the adjacent two frames as the calculation target. When the frame number difference is too large, the correction trajectory needs to be determined through the cumulative transformation from the first frame to the current frame. Therefore, a cumulative transformation matrix variable needs to be maintained.

[0244] The maintenance process is as follows: the cumulative transformation matrix (abbreviated as tm) is a 3X3 matrix:

[0245]

[0246] Assuming that M is the new 2X3 affine transformation matrix obtained through step S1051, representing the transformation of the current frame relative to the previous frame:

[0247]

[0248] Then the new tm is:

[0249]

[0250] Every frame, the new transformation M is multiplied on the left of the cumulative transformation, representing the total transformation from the initial frame to the current frame.

[0251] Step S1053, extract the affine transformation matrix of the first two rows of the cumulative transformation matrix, and obtain the inverse transformation matrix by inverting the affine transformation matrix.

[0252] Step S1054, perform the inverse transformation matrix on the current frame to correct the current frame to the reference perspective of the first frame, and output the corrected video frame.

[0253] Optionally, since the cumulative transformation matrix (tm) represents the transformation relationship from the first frame to the current frame, in order to correct the current frame back to the reference view angle of the first frame, the inverse transformation of the cumulative transformation needs to be applied to the current frame, and the specific implementation steps are as follows:

[0254] 1. Extract the affine part, the cumulative transformation (tm) is a 3x3 homogeneous coordinate transformation matrix, and we need to extract the (2x3) affine matrix of the first two rows.

[0255] 2. Inverse the affine matrix to obtain the inverse transformation matrix .

[0256] 3. Apply the inverse transformation matrix to the current frame, which can "pull back" the picture of the current frame to the stable view angle of the first frame, realizing picture correction.

[0257] 4. Write into the input video, write the corrected frame into the output video file, and complete the correction process of one frame.

[0258] Steps S1051-S1054 realize the global stable alignment of all frames relative to the first frame by using the cumulative affine transformation and inverse mapping technology, which can effectively eliminate the problem that the slight shaking in high-altitude shooting is greatly amplified, and significantly improves the picture stability and the accuracy of subsequent intelligent analysis (such as vehicle flow speed identification).

[0259] In summary, the embodiment of the application provides a video shaking correction method based on high-altitude view, which extracts feature points from the preprocessed detection frame and optimizes the distribution, ensures that the feature points uniformly cover the effective area, avoids local concentration, combines with the motion tracking rule to track the trajectory frame by frame, obtains the feature point trajectory set, and can accurately capture the feature point motion trajectory. The trace data is converted into a polar coordinate system and further converted into a rectangular coordinate system to adapt to the requirements of the input of the clustering algorithm. Then, the stable feature point set is selected by density clustering, which can automatically select the most stable feature point set in the scene, effectively avoid the interference of dynamic targets (such as vehicles and crowds) on motion estimation, and improve the correction accuracy and robustness in complex dynamic environment. Finally, the transformation relationship between adjacent frames is calculated based on the target feature point set, the inverse transformation matrix of the current frame is calculated based on the transformation relationship, and the corrected video frame is output. By using the transformation relationship and inverse transformation matrix technology, the global stable alignment of all frames relative to the first frame is realized, which can effectively eliminate the problem that the slight shaking in high-altitude shooting is greatly amplified, and significantly improve the picture stability. The video shaking correction method based on high-altitude view in the related art has the problem of low correction accuracy.

[0260] Further, the application has the following effects:

[0261] 1. Complete visual information, no need for external hardware. This application only relies on the image content of the video frame itself for analysis and correction, without the need for gyroscopes, IMU and other hardware sensors, greatly reducing the system integration complexity and hardware cost, facilitating rapid deployment and promotion on various unmanned aerial vehicles, monitoring equipment.

[0262] 2. No need for static background assumption, wide adaptability. This application does not rely on the staticity or specific background of the scene, and can adapt to various complex scenes, including dynamic target dense, light change and other practical application environments, with stronger universality and adaptability.

[0263] 3. Easy to integrate and extend. The algorithm flow is clear, the calculation efficiency is high, it is convenient to integrate with existing video processing, target detection and other systems, and can flexibly adjust parameters according to actual needs, supporting video data of multiple resolutions and frame rates.

[0264] 4. Improve the accuracy of subsequent intelligent analysis. Through high-precision picture correction, the target detection, tracking and speed estimation errors caused by shaking are effectively reduced, especially in high-altitude large field of view, the influence of 1 pixel error on speed estimation is minimized, greatly improving the practical value of unmanned aerial vehicle video analysis.

[0265] In summary, this application does not need to rely on external hardware sensors, only uses the visual information of the video itself, analyzes the consistency and excludes abnormalities through the motion trajectory of the feature points between frames, automatically selects the stable feature points of the background static area, and then realizes high-precision estimation and compensation of video shaking. For high-altitude aerial vehicles, high-altitude aerial vehicles and other high-altitude, large field of view, dynamic target rich application scenarios. This application can effectively suppress the picture instability problem caused by device shaking, wind disturbance and other factors, and significantly improve the stability of the video and the accuracy of subsequent intelligent analysis (such as target detection, speed identification, etc.).

[0266] In a second aspect, an embodiment of the present application provides a video shaking correction system based on high-altitude view. Figure 9 is a structural block diagram of a video shaking correction system based on high-altitude view according to an example embodiment. As shown in Figure 9 , the system includes: a feature point set module 910, a feature point trajectory set module 920, a rectangular coordinate system module 930, a clustering module 940 and a correction module 950; wherein:

[0267] The feature point set module 910 is configured to extract feature points from the preprocessed detection frame, optimize the distribution of the feature points, and obtain a feature point set.

[0268] The feature point trajectory set module 920 is configured to track the feature point trajectory frame by frame through a motion tracking rule, and obtain a feature point trajectory set.

[0269] The rectangular coordinate system module 930 is configured to generate a polar coordinate system including a plurality of feature points according to the motion trajectories of the feature points in the feature point trajectory set, and convert the polar coordinate system into a rectangular coordinate system.

[0270] The clustering module 940 is configured to cluster the feature points in the rectangular coordinate system by using a density clustering algorithm, and obtain a target feature point set.

[0271] The rectification module 950 is configured to calculate a transformation relationship between adjacent frames according to the target feature point set, perform inverse affine transformation on the current frame based on the transformation relationship, and output a rectified video frame.

[0272] To sum up, the video shake rectification system based on high-altitude view provided in the application introduces the unsupervised learning method such as density clustering through the feature point set module 910, the feature point trajectory set module 920, the rectangular coordinate system module 930, the clustering module 940, and the rectification module 950, automatically identifies and eliminates the moving targets or abnormal feature points, and enhances the adaptability of the algorithm to the complex dynamic scene. Meanwhile, the affine or homographic transformation model is used to model and rectify the inter-frame global motion, which can be compatible with multiple types of shakes (such as rotation, translation, scaling, etc.), realizes efficient and robust anti-shake of the high-altitude and wide-view video, and has simple overall scheme structure, easy hardware and software integration, good universality and promotion value, and is suitable for various practical application scenarios such as unmanned aerial vehicle aerial photography, city monitoring, and traffic flow analysis. The traditional anti-shake method is broken through, and the method has high precision, self-adaptation, strong robustness, and easy use.

[0273] It should be noted that the video shake rectification system based on high-altitude view provided in the embodiment is used to realize the above-mentioned embodiments, and will not be described again. As used above, the terms “module”, “unit”, “sub-unit”, and the like can be a combination of software and / or hardware that realizes a predetermined function. Although the apparatus described in the above embodiments is preferably realized in software, realization in hardware, or a combination of software and hardware is also possible and conceived.

[0274] In a third aspect, the embodiments of the application provide an electronic device, Figure 10 is a block diagram of an electronic device according to an exemplary embodiment. As Figure 10 indicated, the electronic device can include a processor 101 and a memory 102 having stored computer program instructions.

[0275] In particular, the processor 101 can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the application.

[0276] The memory 102 can include mass storage for data or instructions. By way of example, and not limitation, the memory 102 can include a Hard Disk Drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash drive, a Compact Disk Drive (CD Drive), a DVD Drive, a tape drive, a USB drive, or any combination of two or more of these. The memory 102 can be removable and / or non-removable (or fixed) as appropriate. The memory 102 can be internal or external as appropriate. In certain embodiments, the memory 102 is a non-volatile memory. In certain embodiments, the memory 102 includes a Read-Only Memory (ROM) and a Random Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH memory, or any combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended Data Output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0277] The memory 102 can be used to store or buffer various data files required for processing and / or communication, and possible computer program instructions executed by the processor 101.

[0278] The processor 101 reads and executes the computer program instructions stored in the memory 102 to implement any one of the above-mentioned embodiments of the high-altitude perspective-based video shake correction method.

[0279] In an embodiment, a high-altitude perspective-based video shake correction device can further include a communication interface 103 and a bus 100. As shown in the figure, the processor 101, the memory 102, and the communication interface 103 are connected through the bus 100 and complete communication with each other. Figure 10

[0280] The communication interface 103 is used to implement communication between various modules, devices, units, and / or equipment in the embodiments of the present application. The communication interface 103 can also implement data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations, etc.

[0281] ​Bus 100 includes hardware, software, or both, to couple various components of a high-altitude perspective-based video shake correction apparatus to each other. Bus 100 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, a local bus. By way of example and not limitation, bus 100 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or another suitable bus or combination of two or more of these. Where appropriate, bus 100 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0282] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having stored thereon a program, wherein the program is executed by a processor to implement the method of the first aspect.

[0283] More specifically, the computer readable storage medium can include, but is not limited to, a portable disc, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0284] In possible implementation manners, the present application can also be implemented in the form of a program product, which comprises program codes for causing a terminal device to perform the steps of the high-altitude perspective-based video shake correction method provided in the first aspect when the program product is run on the terminal device.

[0285] Wherein, the program codes for executing the present application can be written in any combination of one or more programming languages, which can be executed entirely on the user device, partially on the user device, as a separate software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0286] The technical features of the above-described embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above-described embodiments are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered as falling within the scope of the present disclosure.

[0287] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A high-altitude perspective-based video shake correction method, characterized by, The method is used for high-altitude, wide-field-of-view scenarios, and the method includes: Feature points are extracted from the preprocessed detection frame, and the distribution of these feature points is optimized to obtain a feature point set. This includes: calculating the autocorrelation matrix of each pixel in the grayscale image using a feature point detection algorithm; determining the corner responsivity based on the minimum eigenvalue of the autocorrelation matrix; sorting the corner responsivity from high to low; and selecting pixels with a preset ranking as the initial feature point set. Based on the imaging characteristics of a high-altitude viewpoint, the initial feature point set is optimized using the maximum number of corner points, a corner quality threshold, a minimum corner point spacing, and a detection window size. The optimized initial feature point set is then distributed and optimized, including: dividing the grayscale image into several image blocks, each containing at least one feature point; removing invalid feature points from the sky and shadow areas; retaining the number of static feature points in the ground and building areas and the number of feature points in dynamic targets according to priority; and using the optimized initial feature point set as the feature point set, which includes the ID numbers of multiple feature points and their coordinate information in the detection frame. The feature points in the feature point set are tracked frame by frame using motion tracking rules to obtain the feature point trajectory set; Based on the motion trajectory of each feature point in the feature point trajectory set, a polar coordinate system including multiple feature points is generated, and the polar coordinate system is converted into a rectangular coordinate system; The feature points in the Cartesian coordinate system are clustered using a density clustering algorithm to obtain the target feature point set; Based on the target feature point set, the transformation relationship between adjacent frames is calculated. Based on the transformation relationship, the current frame is subjected to an inverse affine transformation, and the corrected video frame is output.

2. The method of claim 1, wherein the method comprises: The step of generating a polar coordinate system including multiple feature points based on the displacement distance and movement direction of each feature point from the first frame to the last frame in the feature point trajectory set, and converting the polar coordinate system into a rectangular coordinate system, includes: Based on the full motion trajectory of each feature point in the feature point trajectory set, extract the first frame coordinates and last frame coordinates of multiple feature points; The displacement distance is determined based on the coordinates of the first frame and the coordinates of the last frame, the direction of motion is determined based on the angle between the directions of the coordinates of the first frame and the coordinates of the last frame, and a polar coordinate system including multiple feature points is generated based on the displacement distance and the direction of motion. Based on the mapping relationship between the polar coordinate system and the rectangular coordinate system, the polar coordinate system is converted into a rectangular coordinate system.

3. The video jitter correction method based on a high-altitude perspective according to claim 2, characterized in that, The step of converting the polar coordinate system to a rectangular coordinate system based on the mapping relationship between the polar coordinate system and the rectangular coordinate system includes: Based on the coordinates of the first frame and the coordinates of the last frame, the displacement distance between the coordinates of the first frame and the coordinates of the last frame is calculated using Euclidean distance. Calculate the ratio of the vertical displacement of the first frame coordinates and the last frame coordinates to the displacement distance, perform arcsine processing based on the ratio to obtain the direction angle, and determine the motion direction based on the direction angle; Based on the displacement distance and motion direction, the polar coordinate system is converted to a rectangular coordinate system using trigonometric function relationships. In the rectangular coordinate system, the y-axis represents the trajectory motion direction, and the x-axis represents the displacement distance. The feature point dataset P is defined as {...} }, each Coordinates ).

4. The video jitter correction method based on a high-altitude perspective according to claim 1, characterized in that, The step of clustering feature points in the Cartesian coordinate system using a density clustering algorithm to obtain a target feature point set includes: Based on the density distribution of feature points in the Cartesian coordinate system, determine the neighborhood radius parameter and the minimum number of core points within the neighborhood radius for the density algorithm; Based on the neighborhood radius parameter and the minimum number of points parameter, density clustering is performed on the feature points in the Cartesian coordinate system using the reachability of the density algorithm; Based on the density clustering results, the principal density clusters, including the core points, are obtained as the target feature point set.

5. The video jitter correction method based on a high-altitude perspective according to claim 4, characterized in that, The parameters for determining the neighborhood radius and the minimum number of core points within the neighborhood radius in the density algorithm include: Calculate the distance from the coordinates of the feature point in the Cartesian coordinate system to the k nearest neighbor, sort the distances by size, and generate a distance curve. The k nearest neighbor is obtained by the k-distance function. The distance curve is smoothed by moving average, and the distance value corresponding to the maximum inflection point is identified as the neighborhood radius parameter. The minimum number of points within the neighborhood radius is determined based on the k nearest neighbor points; Density clustering of feature points in the Cartesian coordinate system is performed based on the reachability of the density algorithm, including: Based on the neighborhood radius parameter and the minimum number of points parameter, traverse each feature point in the Cartesian coordinate system; If the number of points within the neighborhood radius of the current feature point is greater than or equal to the minimum number of points parameter, then the feature point is marked as a core point; if the number of points within the neighborhood radius of the current feature point is less than the minimum number of points parameter, then the current feature point is a noise point. If the current feature point is the core point, a new cluster is created and all feature points in its neighborhood are added to the candidate set. By iteratively traversing the candidate set, feature points of unassigned clusters are marked as visited, and the candidate set is expanded based on the core points until the current cluster meets the expansion criteria, thus obtaining the clustering result.

6. The video jitter correction method based on a high-altitude perspective according to claim 1, characterized in that, The step of tracking feature point trajectories frame by frame using motion tracking rules to obtain a set of feature point trajectories includes: Define the window size, number of pyramid levels, and iteration termination condition parameters; The position of each feature point in the current frame is obtained by predicting and tracking the position of all feature points in the previous detection frame using the sparse optical flow method. Based on the defined window size, the number of pyramid layers, and the iteration termination condition parameters, the set of trajectory points for each feature point is obtained through the pyramid optimization algorithm.

7. The video jitter correction method based on a high-altitude perspective according to claim 1, characterized in that, The step of calculating the transformation relationship between adjacent frames based on the target feature point set, performing an inverse affine transformation on the current frame based on the transformation relationship, and outputting the corrected video frame includes: Based on the correspondence between the coordinates of the feature points in the current frame and the coordinates of the feature corner points in the previous frame in the target feature point set, a linear equation is constructed, the linear equation is converted into a matrix, the matrix is ​​solved by the least squares method, and the affine transformation matrix between adjacent frames is constructed. According to the affine transformation matrix ， Obtain the cumulative transformation matrix; Extract the affine transformation matrix from the first two rows of the cumulative transformation matrix, and invert the affine transformation matrix to obtain the inverse transformation matrix; Perform an inverse transform matrix on the current frame to correct the current frame to the reference viewpoint of the first frame, and output the corrected video frame.

8. The video jitter correction method based on a high-altitude perspective according to claim 1, characterized in that, Before extracting feature points from the preprocessed detection frame, the method further includes: The first frame of the video is taken as the first frame. If the displacement between the first frame and the next frame exceeds a preset displacement threshold, or the deflection angle between the first frame and the next frame exceeds a preset deflection angle threshold, the detection frame is determined by video editing or frame selection. The video decoding module parses and records the video's frame width, frame height, frame rate, and total number of frames. The detection frame is preprocessed, wherein the preprocessing includes converting the detection frame into a grayscale image; Initialize the feature point set based on the grayscale image. With trajectory set The set of feature points Includes the unique ID and initial coordinates of each feature point. This represents the nth feature point, and the initial feature point set is backed up. The set of trajectories The data includes the inter-frame motion trajectory data of each feature point, including the frame number, coordinates, displacement distance, and direction of motion of the current trajectory point.

9. A video jitter correction system based on a high-altitude perspective, characterized in that, The system is used for high-altitude, wide-field-of-view scenarios. The system includes: a feature point set module, a feature point trajectory set module, a rectangular coordinate system module, a clustering module, and a correction module; wherein: The feature point set module is used to extract feature points from the preprocessed detection frame, optimize the distribution of the feature points, and obtain a feature point set. Specifically, it is used to: calculate the autocorrelation matrix of each pixel in the grayscale image using a feature point detection algorithm; determine the corner responsivity based on the minimum eigenvalue of the autocorrelation matrix; sort the corner responsivity from high to low; and select pixels with a preset ranking as the initial feature point set. Based on the imaging characteristics of the high-altitude viewpoint, the initial feature point set is optimized by the maximum number of corner points, the corner quality threshold, the minimum corner point spacing, and the detection window size. The optimized initial feature point set is then optimized for distribution, wherein the distribution optimization includes: dividing the grayscale image into several image blocks, each image block containing at least one feature point; removing invalid feature points in the sky and shadow areas; retaining the number of static feature points in the ground and building areas and the number of feature points in dynamic targets according to priority; and using the optimized initial feature point set as the feature point set, wherein the feature point set includes the ID number of multiple feature points and their coordinate information in the detection frame. The feature point trajectory set module is used to track the feature points in the feature point set frame by frame according to motion tracking rules to obtain the feature point trajectory and acquire the feature point trajectory set. The rectangular coordinate system module is used to generate a polar coordinate system including multiple feature points based on the motion trajectory of each feature point in the feature point trajectory set, and to convert the polar coordinate system into a rectangular coordinate system. The clustering module is used to cluster feature points in the Cartesian coordinate system using a density clustering algorithm to obtain a target feature point set. The correction module is used to calculate the transformation relationship between adjacent frames based on the target feature point set, perform an inverse affine transformation on the current frame based on the transformation relationship, and output the corrected video frame.

Images