An image stabilization method and system based on pixel displacement compensation

Abstract

This invention relates to the field of image stabilization technology, specifically disclosing an image stabilization method and system based on pixel displacement compensation. The method includes: acquiring the attitude data of a virtual gimbal and the global motion vectors of video frames, identifying key attitude segments prone to compensation boundary conflicts; collecting the spatial distribution characteristics of pixel displacements in the key attitude segments, extracting boundary-sensitive regions corresponding to the characteristic attitude points; generating non-rectangular compensation boundary shapes, dynamic mask prototypes, and continuous deformation rules based on the boundary-sensitive regions; and outputting a stabilized image sequence based on the dynamic mask prototypes and deformation rules. The system includes an attitude acquisition module, a compensation analysis module, a rule determination module, and a stabilization and reinforcement module. This invention helps reduce compensation boundary conflicts under large-angle attitudes and is suitable for complex flight scenarios of UAVs.

Description

Technical Field

[0001] This invention relates to the field of image stabilization technology, specifically to an image stabilization method and system based on pixel displacement compensation. Background Technology

[0002] Traditional drones rely on mechanical gimbals for image stabilization, but mechanical gimbals are bulky, consume a lot of power, are expensive, and are prone to mechanical jamming when rotating at large angles, making them unsuitable for the application needs of miniaturized and lightweight drones.

[0003] Virtual gimbal technology based on software algorithms acquires video streams through camera modules and combines them with attitude sensing data for pixel displacement compensation. It requires no additional mechanical structure, effectively reducing equipment complexity and cost.

[0004] However, the fixed compensation boundaries in existing virtual gimbal image stabilization solutions lead to frequent boundary conflicts at large angles of motion. Current electronic image stabilization technologies generally employ fixed rectangular compensation areas, whose boundaries are pre-defined and do not dynamically adjust with the drone's attitude. When the drone performs large-angle pitch or roll movements, the pixel displacement increment exceeds the tolerance range of the rectangular boundary, resulting in insufficient overlap between the compensation area and the effective image area. This leads to problems such as image cropping, loss of critical information (e.g., missing details of power lines during inspections), or image edge stretching and distortion, affecting the accuracy of subsequent image analysis and task judgment.

[0005] Therefore, the present invention provides an image stabilization method and system based on pixel displacement compensation. Summary of the Invention

[0006] The purpose of this invention is to provide an image stabilization method and system based on pixel displacement compensation to solve the aforementioned background problems.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] An image stabilization method based on pixel displacement compensation includes the following steps:

[0009] Acquire the attitude data and pixel displacement of the virtual gimbal within the motion range, perform attitude sensitivity analysis on the pixel displacement under different attitudes, and identify key attitude segments for compensating boundary conflicts.

[0010] The spatial distribution features of pixel displacements corresponding to key pose segments are collected, and the pose-compensation range is constructed. By analyzing the compensation capability of the pose-compensation range, the boundary sensitive areas of dynamic changes of feature pose points are extracted.

[0011] Based on the boundary-sensitive region, non-rectangular compensation boundary morphology applicable to different poses is extracted. Based on the non-rectangular compensation boundary morphology, a dynamic mask prototype driven by feature pose points is generated, as well as deformation rules that continuously change with feature pose points.

[0012] Based on the dynamic mask prototype and deformation rules, adaptive deformation control processing is performed on the compensation region to obtain the angular compensation region of dynamic change of two-dimensional attitude points. Image stabilization processing is then performed in the angular compensation region to output a stabilized image sequence.

[0013] As a further technical solution of the present invention: the posture sensitivity analysis is performed as follows:

[0014] Pitch and roll angles are extracted from attitude data to obtain sensitivity values;

[0015] Construct a set of data points with pitch and roll angles as coordinates and sensitivity values ​​as features;

[0016] A two-dimensional attitude plane composed of pitch angle and roll angle is constructed based on the data point set, and density-based spatial clustering analysis is performed on the sensitivity characteristics of all data points in the two-dimensional attitude plane.

[0017] Spatial clustering analysis was used to divide the entire attitude plane into a low-sensitivity main region and a high-sensitivity outlier region.

[0018] Identify the range of high-sensitivity outlier regions that are continuously distributed in the pitch-roll plane, and define the range as the key attitude segment where compensation boundary conflicts are likely to occur.

[0019] As a further technical solution of the present invention: the method for obtaining the sensitivity value is as follows:

[0020] Acquire continuous video frames from the virtual gimbal and perform pixel displacement analysis to calculate the global motion vector of the continuous image frames as the pixel displacement;

[0021] Decompose the horizontal pixel displacement increment and the vertical pixel displacement increment from the global motion vector;

[0022] Calculate the ratio of the lateral pixel displacement increment to the pitch angle increment to obtain the sensitivity of the lateral pixel displacement to the pitch angle.

[0023] Calculate the ratio of the vertical pixel displacement increment to the roll angle increment to obtain the sensitivity of the vertical pixel displacement to the roll angle.

[0024] The calculated sensitivity of the lateral pixel displacement to the pitch angle and the sensitivity of the vertical pixel displacement to the roll angle are bound to the corresponding pitch angle and roll angle, respectively, to form a set of data points with pitch angle and roll angle as coordinates and sensitivity value as feature.

[0025] As a further technical solution of the present invention, the process of performing the compensation capability analysis is as follows:

[0026] Based on the pitch and roll angles corresponding to key attitude segments, the attitude data is bound to the motion instability coefficient to establish the attitude-compensation range.

[0027] Traverse all key attitude segments, draw a surface where the pixel compensable range varies with pitch and roll angles, and identify local maxima and minima of the attitude-compensation range on the surface.

[0028] The two-dimensional attitude points corresponding to the local minimum points are used as feature attitude points;

[0029] Obtain the global instability coefficient corresponding to the feature attitude point, as well as the motion instability coefficient of the strip region;

[0030] By comparing and analyzing the motion instability coefficient and the global instability coefficient, the boundary sensitive area is determined.

[0031] As a further technical solution of the present invention: the process of obtaining the motion instability coefficient of the strip-shaped region is as follows:

[0032] Extract pixel displacement vector data from multiple consecutive frames within each key pose segment;

[0033] Each frame of the image is divided into three concentric band-shaped regions: the central region, the transition region, and the edge region.

[0034] Calculate the average magnitude and average orientation angular variance of all pixel displacement vectors within each strip region;

[0035] The instability coefficient is obtained by multiplying the variance of the average modulus and the orientation angle.

[0036] As a further technical solution of the present invention: the process of extracting the non-rectangular compensation boundary shape is as follows:

[0037] Based on the characteristic attitude points, and the spatial location and motion instability coefficient of all boundary sensitive regions marked under the characteristic attitude points, the boundary sensitive region is modeled as a repulsive energy field;

[0038] Obtain the initial rectangular compensation region boundary of the image, discretize the boundary into dense and uniformly distributed control points, and connect the control points to form an elastic polygonal contour.

[0039] Based on each control point on the elastic polygonal profile, extract the total repulsive force vector of each control point in the repulsive energy field of all boundary sensitive regions.

[0040] Iterative deformation and constraint solving are performed on all control points based on the total repulsive force vector to extract the non-rectangular compensation boundary shape.

[0041] As a further technical solution of the present invention, the method for performing the iterative deformation and constraint solution is as follows:

[0042] Based on the total repulsive force vector at each repulsion point, the constraint rules for iterative deformation are determined;

[0043] Calculate the local curvature of the elastic polygonal profile and perform smoothing control on the elastic polygonal profile;

[0044] Based on constraint rules and smoothing control, the control force of all control points converges or reaches the maximum number of iterations, and the optimal non-rectangular compensation boundary shape is established under the current characteristic attitude point.

[0045] As a further technical solution of the present invention, the method for performing image stabilization processing in the angle compensation region is as follows:

[0046] Weight fusion and allocation are performed based on the angled compensation region that forms a dynamic change in two-dimensional attitude points;

[0047] Position correction processing is performed on pixels within the angled compensation area after weighted fusion allocation;

[0048] The video frames based on position correction are feathered and fused to output a continuously stabilized image sequence.

[0049] As a further technical solution of the present invention: the method for obtaining the angle compensation region is as follows:

[0050] Obtain the initially adjusted video frames and add boundary constraints to the compensation boundaries; based on the video frames with added boundary constraints and the pose-compensation range, establish angle quantization rules:

[0051] By using angular quantization rules, real-time two-dimensional attitude points are transformed into compensation region boundary angle parameters, forming an angular compensation region that dynamically changes with the two-dimensional attitude points.

[0052] An image stabilization system based on pixel displacement compensation includes the following modules:

[0053] Attitude acquisition module: used to acquire the attitude data and pixel displacement of the virtual gimbal within the motion range, perform attitude sensitivity analysis on the pixel displacement under different attitudes, and identify key attitude segments for compensating for boundary conflicts.

[0054] Compensation Analysis Module: Used to collect the spatial distribution features of pixel displacements corresponding to key pose segments, construct the pose-compensation range, and extract the boundary sensitive areas of dynamic changes of feature pose points by analyzing the compensation capability of the pose-compensation range.

[0055] Rule determination module: Based on the boundary sensitive region, extract the non-rectangular compensation boundary shape applicable to different postures, generate a dynamic mask prototype driven by feature posture points based on the non-rectangular compensation boundary shape, and deformation rules that change continuously with feature posture points.

[0056] Stabilization and reinforcement module: Based on the dynamic mask prototype and deformation rules, adaptive deformation control processing is performed on the compensation area to obtain the angular compensation area of ​​dynamic change of two-dimensional attitude points, and image stabilization processing is performed in the angular compensation area to output the stabilized image sequence.

[0057] The beneficial effects of this invention are as follows:

[0058] 1. By acquiring attitude data such as pitch and roll angles of the virtual gimbal and combining them with the global motion vectors of video frames for attitude sensitivity analysis, and clustering the attitude-sensitivity data points, it is helpful to identify high-sensitivity outlier areas that are prone to compensation boundary conflicts. This effectively solves the problem that traditional methods have difficulty in locating high-risk attitude ranges and reduces the blindness of compensation operations.

[0059] 2. By collecting the spatial distribution features of pixel displacement in key attitude segments, the image is divided into three strip regions: center, transition, and edge, and the motion instability coefficient is calculated. This allows for the quantification of the motion activity and directional consistency in different regions. Simultaneously, based on the attitude-compensation range, the boundary sensitive regions under the feature attitude points are extracted, high-risk regions prone to conflict during the compensation process are located, and the compensation priority is clarified.

[0060] 3. Based on the boundary-sensitive region, a repulsive energy field is constructed to drive the initial rectangular compensation boundary to iteratively transform into a non-rectangular compensation boundary. A dynamic mask prototype library and continuous deformation rules driven by feature attitude points are generated, reducing the limitations of fixed rectangular compensation boundaries in traditional image stabilization. Dynamic mask matching under different attitudes is achieved through bilinear interpolation algorithm, which is beneficial to adapting to the compensation needs of different flight attitudes of UAVs. It alleviates the problem that traditional fixed boundaries are prone to insufficient compensation or boundary conflicts under large-angle attitudes, and improves the adaptability of compensation boundaries and attitudes.

[0061] 4. Based on the dynamic mask prototype and deformation rules, adaptive deformation control is achieved by comparing the compensation modulus with the initial compensation boundary range, resulting in an angular compensation region that dynamically changes with the two-dimensional attitude point. At the same time, sub-pixel level bilinear interpolation is combined to achieve accurate pixel position correction. With inter-frame weighted fusion and edge feathering, it can compensate for the offset caused by the increase in horizontal and vertical pixel displacement. Furthermore, real-time performance is ensured by pre-calculating key attitude segment mask templates, meeting the real-time stabilization requirements of UAVs during dynamic flight. Attached Figure Description

[0062] The invention will now be further described with reference to the accompanying drawings.

[0063] Figure 1 This is a flowchart of an image stabilization method based on pixel displacement compensation according to the present invention;

[0064] Figure 2 This is a flowchart of the boundary-sensitive region in this invention;

[0065] Figure 3 This is a functional block diagram of an image stabilization system based on pixel displacement compensation in this invention. Detailed Implementation

[0066] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0067] Example 1

[0068] like Figure 1 As shown, an image stabilization method based on pixel displacement compensation includes the following steps:

[0069] S1. Obtain the attitude data and pixel displacement of the virtual gimbal within the motion range, perform attitude sensitivity analysis on the pixel displacement under different attitudes, and identify the key attitude segments for compensating boundary conflicts.

[0070] The method for obtaining the virtual gimbal's attitude data and pixel displacement within the motion range is as follows:

[0071] Preferably, the pitch and roll angles of the drone turntable are obtained as attitude data when the drone flies within the predetermined path movement range, based on the virtual gimbal mounted on the camera module on the drone.

[0072] Extract continuous video frames and the corresponding sampling timestamps from the video stream output by the camera module.

[0073] Obtain the sampling timestamps corresponding to the pitch and roll angles, and perform time alignment processing on the video frames and pitch and roll angles based on the sampling timestamps;

[0074] Perform pixel displacement analysis on consecutive video frames and calculate the global motion vector of consecutive image frames as pixel displacement;

[0075] Preferably, the pixel displacement analysis is performed as follows: In the previous frame and the current frame, find 20-50 feature points respectively. For each feature point in the previous frame, find the point that best matches it in the current frame to form a displacement vector. From all the displacement vectors obtained in the previous step, filter out the incorrect pairs whose directions and distances are significantly different from most vectors. Calculate the average value of all the remaining displacement vectors in the X direction (horizontal) and Y direction (vertical) respectively. The two values ​​obtained are the global motion vector.

[0076] The method for filtering out incorrect pairs is as follows: calculate the mean direction angle μ and standard deviation σ of all displacement vectors, as well as the mean distance d and standard deviation σ of the distance; remove vectors whose direction angle deviates from μ±3σ and whose distance deviates from d±3σ, based on the 3σ principle of normal distribution (99.7% of normal data fall within the range of μ±3σ, effectively removing abnormal pairs).

[0077] The method for performing attitude sensitivity analysis on pixel displacements under different postures to identify key posture segments for compensating boundary conflicts is as follows:

[0078] Decompose the horizontal pixel displacement increment and the vertical pixel displacement increment from the global motion vector;

[0079] Calculate the ratio of the lateral pixel displacement increment to the pitch angle increment to obtain the sensitivity of the lateral pixel displacement to the pitch angle.

[0080] It should be noted that the global motion vector itself is a two-dimensional vector, and its X-axis component is the horizontal pixel displacement increment, and its Y-axis component is the vertical pixel displacement increment.

[0081] Calculate the ratio of the vertical pixel displacement increment to the roll angle increment to obtain the sensitivity of the vertical pixel displacement to the roll angle.

[0082] The calculated sensitivity of the horizontal pixel displacement to the pitch angle and the sensitivity of the vertical pixel displacement to the roll angle are bound to the corresponding pitch angle and roll angle, respectively, to form a set of data points with pitch angle and roll angle as coordinates and sensitivity value as feature.

[0083] A two-dimensional attitude plane composed of pitch angle and roll angle is constructed based on the data point set, and density-based spatial clustering analysis is performed on the sensitivity characteristics of all data points in the two-dimensional attitude plane.

[0084] Spatial clustering analysis was used to divide the entire attitude plane into one low-sensitivity main region and N high-sensitivity outlier regions.

[0085] Identify the range of high-sensitivity outlier regions that are continuously distributed in the pitch-roll plane, and define the range as the key attitude segment where compensation boundary conflicts are likely to occur.

[0086] Understandably, the purpose of constructing key attitude segments is:

[0087] Objective 1: To identify the risk range of compensation and reduce the blindness of compensation operations; through attitude sensitivity analysis, to screen out the key attitude segments that are prone to compensation boundary conflicts from the full attitude range of the virtual gimbal (such as the attitude range corresponding to large pitch and roll), and to reduce the indiscriminate compensation calculations for low-risk and low-sensitivity conventional attitudes.

[0088] Objective 2: To provide a target range for the design of subsequent compensation schemes; the subsequent acquisition of pixel displacement spatial distribution features and the construction of attitude-compensation range mapping relationship are all based on key attitude segments as target intervals. The design of subsequent non-rectangular compensation boundary shapes and dynamic mask prototypes are all designed for feature attitude points within key attitude segments.

[0089] It should be noted that the density-based spatial clustering analysis uses the DBSCAN algorithm. The DBSCAN algorithm can automatically discover clusters of arbitrary shapes and mark data points with a density significantly lower than the surrounding area as outliers. The high-sensitivity outlier region is the continuous region composed of these outliers.

[0090] It should be noted that the neighborhood radius in the DBSCAN algorithm is determined by calculating the inflection point of the statistical distribution of the distances between all data points and their k-th (preferably, k=5) nearest neighbors (as shown in the k-distance graph); the minimum number of points is set according to an empirical percentage (such as 1% to 2%) of the total number of data points; the DBSCAN algorithm discovers clusters of arbitrary shapes and identifies data points with a density significantly lower than that of the surrounding area as outliers, and the high-sensitivity outlier region is the continuous region composed of these outliers.

[0091] S2. Collect the spatial distribution features of pixel displacements corresponding to key posture segments, construct the posture-compensation range, and extract the boundary sensitive areas of dynamic changes of feature posture points by analyzing the compensation capability of the posture-compensation range.

[0092] The method for collecting the spatial distribution features of pixel displacements corresponding to key pose segments and constructing the pose-compensation range is as follows:

[0093] Extract pixel displacement vector data of multiple consecutive frames of images within each key attitude segment (i.e., the pitch-roll range identified in S1);

[0094] Each frame of the image is divided into three concentric band-shaped regions: the central region, the transition region, and the edge region.

[0095] It should be noted that each frame of the image is divided into three concentric band-shaped regions: the central region (radius of 1 / 3 of the image diagonal), the transition region (radius of 1 / 3-2 / 3 of the image diagonal), and the edge region (radius of 2 / 3-1 of the image diagonal). This division is based on the conventional proportions for dividing the image regions and is used to uniformly cover the entire image area and distinguish the motion characteristics of different locations (the central region has more stable motion, while the edge region is more prone to instability).

[0096] Calculate the average magnitude and average orientation angular variance of all pixel displacement vectors within each strip region;

[0097] The instability coefficient is obtained by multiplying the variance of the average modulus and the orientation angle.

[0098] It should be noted that the higher the value of the motion instability coefficient, the more active and inconsistent the motion of the strip region. The motion instability coefficient is used to quantify the degree of instability of pixel motion in the region and distinguish the instability differences of the image center, transition, and edge regions. The value guides the marking of boundary-sensitive regions and provides a basis for high-risk regions for subsequent non-rectangular compensation boundary design and dynamic mask prototype generation, so that the compensation scheme can be specifically adapted to the instability scenario.

[0099] Based on the pitch and roll angles corresponding to key attitude segments, the attitude data is bound to the motion instability coefficient to establish the attitude-compensation range.

[0100] The method for extracting the boundary-sensitive region of dynamic change of feature pose points by analyzing the compensation capability of the pose-compensation range is as follows:

[0101] The preferred method for performing compensation capability analysis is as follows:

[0102] S201. Traverse all key attitude segments, draw the surface where the pixel compensation range changes with pitch and roll angles, and identify the local maxima and local minima of the attitude-compensation range on the surface.

[0103] S202. The two-dimensional attitude point (i.e., the combination of pitch angle and roll angle) corresponding to the local maximum point represents the safe attitude with the strongest compensation capability, and the two-dimensional attitude point corresponding to the local minimum point represents the risk attitude with the weakest compensation capability and the most likely to cause boundary conflict.

[0104] S203. Take the two-dimensional attitude points corresponding to the local minimum points as feature attitude points;

[0105] It should be noted that, in order to balance compensation accuracy and computational efficiency, 1-3 local minima are extracted for each high-sensitivity outlier region. The extraction rules are as follows: if the region area is ≤ 5° (pitch angle) × 5° (roll angle), 1 local minima are extracted; if 5° × 5° < region area ≤ 10° × 10°, 2 local minima are extracted; if the region area is > 10° × 10°, 3 local minima are extracted.

[0106] Calculate the motion instability coefficients of the central region, transition region, and edge region in the image corresponding to each feature pose point. Sum the motion instability coefficients of the three regions and take the average value to obtain the global instability coefficient.

[0107] The motion instability coefficients of the lower edge region and the transition region of the feature attitude point are compared with the global instability coefficient.

[0108] like Figure 2 As shown, if the motion instability coefficient of a certain edge region or transition region is consistently higher than the global instability coefficient, the corresponding region (edge ​​region or transition region) is marked as a boundary sensitive region under the feature attitude point; otherwise, no processing is performed.

[0109] It should be noted that zero, one or more boundary-sensitive regions can be marked under a single feature pose point.

[0110] Example 2

[0111] like Figure 1 As shown, an image stabilization method based on pixel displacement compensation includes the following steps:

[0112] S3. Based on the boundary sensitive region, extract the non-rectangular compensation boundary shape applicable to different postures, and generate a dynamic mask prototype driven by the feature posture point based on the non-rectangular compensation boundary shape, as well as the deformation rule that changes continuously with the feature posture point.

[0113] Among them, the method for extracting non-rectangular compensation boundary shapes applicable to different postures based on boundary-sensitive regions is as follows:

[0114] Based on each feature attitude point output by S2, and the spatial location and motion instability coefficient of all boundary sensitive regions marked under the feature attitude points, the boundary sensitive region is modeled as a repulsive energy field.

[0115] It should be noted that the motion instability coefficient of the boundary sensitive area is quantified as the field strength of the repulsive energy field. The higher the instability coefficient, the greater the risk of conflict in the boundary sensitive area during the compensation process, and the stronger its repulsive energy.

[0116] Obtain the initial rectangular compensation region boundary of the image, discretize the boundary into dense (e.g., discretize into dense uniform control points with a spacing of 5-8 pixels) and uniformly distributed control points, connect the control points to form a deformable elastic polygonal contour, which serves as the initial prototype of the dynamic mask.

[0117] Based on each control point on the elastic polygonal profile, extract the total repulsive force vector of each control point in the repulsive energy field of all boundary sensitive regions.

[0118] It should be noted that the method for extracting the resultant force vector is as follows: traverse each boundary sensitive energy field, calculate the field strength of the energy field after attenuation at this point based on the distance between the control point and the center of the energy field, the direction of the force is the opposite direction from the control point to the center of the energy field, and superimpose the force vectors generated by all energy fields at this point to obtain the total repulsive force vector of the control point;

[0119] The magnitude of the vector is calculated based on the total repulsive force vector at each control point, and the direction is the direction of the total repulsive force vector.

[0120] Based on the total repulsive force vector, iterative deformation and constraint solutions are performed on all control points to extract the non-rectangular compensation boundary shape.

[0121] The method for iterative deformation and constraint solving is as follows:

[0122] S301. Based on the total repulsive force vector at each repulsion point, determine the constraint rules for iterative deformation;

[0123] Preferably, the constraint rules for iterative deformation are determined as follows: the direction of the total repulsive force vector at each control point is taken as the displacement direction of the control point, and the magnitude of the vector is normalized and multiplied by a fixed small step size coefficient to form the displacement step size of the control point; thus forming the constraint rules for iterative deformation, which transforms the repulsive force into the specific displacement of each control point in one iteration.

[0124] Preferably, the microstep size factor is 0.3;

[0125] S302. Calculate the local curvature of the elastic polygonal profile and perform smoothing control on the elastic polygonal profile.

[0126] Preferably, the smoothing control method is as follows: after each iteration of displacement, the change in the angle between each control point and the line segment formed by its two adjacent points is calculated, and this is used as a measure of local curvature; the position of the control point is finely adjusted by the gradient descent method to minimize the sum of the local curvature changes of all control points, thereby achieving smoothing control of the contour.

[0127] S303, repeat 301 and 302 until the control force of all control points converges or the maximum number of iterations is reached, and establish the optimal non-rectangular compensation boundary shape under the current characteristic attitude point;

[0128] Preferably, the method for determining the termination of the iteration is as follows: after each iteration, calculate the sum of the magnitudes of the repulsive force vectors of all control points. If the sum is less than the preset convergence threshold, or the number of iterations reaches the preset maximum value, then terminate the iteration; then the elastic polygonal profile is the stable optimal non-rectangular compensation boundary shape.

[0129] It should be noted that, based on the conventional accuracy requirements for sub-pixel level image processing, the convergence threshold is set to 0.1 pixels to ensure that there is no obvious boundary jitter in human vision, and the maximum number of iterations is set to 10.

[0130] The method for generating dynamic mask prototypes is as follows:

[0131] For each feature attitude point identified by S2, the iterative deformation and constraint solution process from S301 to S303 is executed to calculate and generate an optimal non-rectangular compensation boundary shape that uniquely corresponds to each feature attitude point.

[0132] All feature pose points and their corresponding non-rectangular compensation boundary shapes are integrated and stored as a dynamic mask prototype library;

[0133] For any two-dimensional attitude point not included in the prototype library, the dynamic mask corresponding to the two-dimensional attitude point is calculated by the bilinear interpolation algorithm, and a continuous deformation rule based on the two-dimensional attitude plane is established.

[0134] Preferably, the method of calculating the corresponding dynamic mask by bilinear interpolation algorithm is as follows: in the two-dimensional attitude plane formed by pitch and roll, find and lock the four nearest feature attitude points surrounding the two-dimensional attitude point.

[0135] Obtain the mask shape contours corresponding to the four nearest feature pose points from the dynamic mask prototype library; perform bilinear interpolation calculation on the two-dimensional coordinates of the corresponding control points on the four mask contours to interpolate a new set of control point coordinates.

[0136] Connecting the control point coordinates in sequence yields the dynamic mask shape at any feature pose point, i.e., the dynamic mask prototype.

[0137] Understandably, the purpose of constructing a dynamic mask prototype is:

[0138] Function 1: Provides compensation templates for different high-risk postures, reducing the difficulty of adapting to fixed boundaries; the dynamic mask prototype is an optimal non-rectangular compensation boundary template generated based on feature posture points (high-risk postures with the weakest compensation ability and prone to conflict), which can specifically match the instability characteristics of different postures within key posture segments (such as high instability coefficients in edge regions). Compared with traditional fixed rectangular boundaries, it reduces the problem of insufficient compensation or boundary conflicts under large-angle postures, and provides accurate template basis for subsequent compensation of arbitrary postures;

[0139] Secondly, it facilitates rapid dynamic deformation of the compensation area, ensuring real-time stabilization. The dynamic mask prototype library pre-stores the boundary shapes corresponding to feature attitude points. When dealing with non-feature attitude points, there is no need to re-iterate the boundary calculation; a suitable dynamic mask can be quickly generated simply through bilinear interpolation. This reduces the amount of compensation calculation for each frame of the image during the dynamic flight of the UAV, enabling real-time generation of angle-based compensation areas and meeting the real-time stabilization requirements of UAVs in complex flight scenarios.

[0140] S4. Based on the dynamic mask prototype and deformation rules, adaptive deformation control processing is performed on the compensation area to obtain the angled compensation area of ​​the two-dimensional attitude point dynamic change, and image stabilization processing is performed in the angled compensation area to output the stabilized image sequence.

[0141] Among them, based on the dynamic mask prototype and deformation rules, adaptive deformation control processing is performed on the compensation region to obtain the angled compensation region for the dynamic change of the two-dimensional attitude point as follows:

[0142] Preferably, the horizontal pixel displacement increment and vertical pixel displacement increment of the current video frame are obtained, the two pixel displacement increments are combined to establish a real-time pixel displacement compensation amount, and the modulus of the pixel displacement compensation amount is calculated as the compensation modulus.

[0143] Simultaneously acquire the current attitude data of the drone and extract the pitch and roll angles from the attitude data;

[0144] Based on the current pitch and roll angles as the current two-dimensional attitude points, and input into the bilinear interpolation algorithm of S3, the corresponding non-rectangular compensation boundary shape is matched from the dynamic mask prototype library as the initial compensation boundary.

[0145] Based on the comparison between the compensation modulus and the allowable range of the initial compensation boundary, if the compensation modulus exceeds the maximum value of the allowable range, the elastic polygon control points are adaptively adjusted based on the iterative deformation rule of S3 to obtain the video frame after preliminary adjustment.

[0146] If the compensation modulus is within the tolerance range, the control point iterative adjustment is not initiated. Instead, the initial compensation boundary is directly used, and only subsequent constraint and mapping steps are executed to obtain the preliminarily adjusted video frame.

[0147] The adaptive adjustment method is as follows: the X and Y component directions of the real-time pixel displacement compensation amount are used as the control point adjustment direction to ensure that the compensation area can cover the required compensation range; the adjustment step size is dynamically adapted according to the compensation modulus and the repulsion energy field strength of the boundary sensitive area. The larger the compensation modulus, the larger the base value of the step size. The stronger the repulsion energy field, the higher the step size decay ratio, to avoid excessive deformation triggering new boundary conflicts; the adjustment is iteratively adjusted in stages according to the constraint rules of S3. After each iteration, the boundary smoothness is optimized by local curvature fine-tuning. The iteration termination condition is that the compensation area completely accommodates the real-time pixel displacement compensation amount or reaches the preset maximum number of iterations (3 times), and the preliminary adjusted compensation boundary is output.

[0148] For the initially adjusted video frames, add boundary constraints at the compensation boundaries;

[0149] It should be noted that the method for adding boundary constraints at the compensation boundary is as follows:

[0150] The location and motion instability coefficient of the boundary sensitive area based on S2 markers are used to convert the motion instability coefficient into the repulsive energy field intensity, which applies additional constraints to the elastic polygon control points and reduces the coverage of high-risk boundary sensitive areas by the compensation area.

[0151] Angle-based quantization rules are established based on video frames with added boundary constraints and the pose-compensation range.

[0152] For example, the angle quantification rule is: for every ±5° roll angle, the lateral boundary of the compensation area is deflected inward by ±1°;

[0153] For every ±5° pitch angle, the longitudinal boundary of the compensation area deflects inward by ±1°.

[0154] It should be noted that when the roll angle is positive, the lateral boundary of the compensation area deflects to the left; when the roll angle is negative, the lateral boundary deflects to the right; when the pitch angle is positive, the longitudinal boundary of the compensation area deflects upward; when the pitch angle is negative, the longitudinal boundary deflects downward. This is based on the correlation logic between the UAV's roll angle, pitch angle, and pixel displacement direction, i.e., the roll angle controls the lateral pixel displacement, and the pitch angle controls the longitudinal pixel displacement.

[0155] The real-time two-dimensional attitude point is transformed into the angle parameter of the compensation region boundary by the angle quantization rule, forming an angle-based compensation region that dynamically changes with the two-dimensional attitude point.

[0156] It should be noted that the boundary of the angle compensation region always maintains the smooth characteristics after the iterative optimization in S3, and the jagged edges are reduced by fine-tuning the local curvature.

[0157] The method for performing image stabilization processing in the angle compensation region and outputting the stabilized image sequence is as follows:

[0158] S401. Weight fusion and allocation processing is performed based on the angled compensation region that forms a dynamic change in two-dimensional attitude points.

[0159] Preferably, based on the angled compensation region that forms a dynamic change in two-dimensional attitude points, the global instability coefficient corresponding to the current video frame is extracted, and a preset non-zero supplementary coefficient is obtained;

[0160] Preferably, the non-zero supplementary coefficient is 0.1;

[0161] The non-zero supplement coefficient and the global instability coefficient are summed, and the reciprocal of the sum is used as the fusion weight. The fusion weights of the current frame and the previous frame's stabilized image are then assigned. The lower the global instability coefficient, the higher the fusion weight of the current video frame.

[0162] S402. Based on the pixels within the angled compensation area after weighted fusion allocation, perform position correction processing;

[0163] Preferably, the position correction process is performed as follows: based on the horizontal and vertical pixel displacement increments obtained in S1, the floating-point coordinates of the target pixel in the current frame within the angle compensation area in the original image are calculated. , = Target pixel horizontal coordinate - Horizontal pixel displacement increment =Target pixel vertical coordinate - Vertical pixel displacement increment;

[0164] Pick The integer part is , The integer part is Determine the original pixel points of the four nearest integer coordinates surrounding the target pixel, namely the top left corner point. Top right corner , lower left corner point bottom right corner ;

[0165] definition for decimal part , for For the decimal part, calculate the interpolation weights of the four neighboring pixels according to the rule of "the closer the distance, the greater the weight", and the weight of the top left corner pixel. Weight of the top right corner point Weight of the bottom left corner point Weight of the bottom right corner ;

[0166] Obtain the original pixel values ​​of four neighboring pixels, and then calculate the current frame pixel value after position correction using the formula: ×Top left corner pixel value+ ×Top right corner pixel value+ × Pixel value of bottom left corner + The pixel value of the bottom right corner is summed pixel by pixel to compensate for the offset caused by the increase in horizontal and vertical pixel displacement.

[0167] Call the current frame fusion weight allocated by S401, set the fusion weight of the previous frame stabilized image = 1 - current frame fusion weight; for pixels with the same coordinates in the angle compensation area, calculate pixel by pixel according to the final pixel value after fusion = current frame pixel value after position correction × current frame fusion weight + previous frame stabilized pixel value × previous frame fusion weight, so as to achieve accurate pixel position alignment and smooth connection of inter-frame information.

[0168] S403. Feathering and blending processing is performed on video frames based on position correction processing;

[0169] Preferably, the feathering fusion processing method is as follows: for the edge pixels of the angle compensation area, feathering fusion processing is applied, and the feathering width is set to 3 pixels to reduce the splicing artifacts at the region boundary.

[0170] Based on the S401-403 processing method, the image stabilization result is output in real time for each frame of image processing, and a continuous stabilized image sequence is output.

[0171] Example 3

[0172] Please see Figure 3 As shown, an image stabilization system based on pixel displacement compensation includes the following modules:

[0173] Attitude acquisition module: used to acquire the attitude data and pixel displacement of the virtual gimbal within the motion range, perform attitude sensitivity analysis on the pixel displacement under different attitudes, and identify key attitude segments for compensating for boundary conflicts.

[0174] Compensation Analysis Module: Used to collect the spatial distribution features of pixel displacements corresponding to key pose segments, construct the pose-compensation range, and extract the boundary sensitive areas of dynamic changes of feature pose points by analyzing the compensation capability of the pose-compensation range.

[0175] Rule determination module: Based on the boundary sensitive region, extract the non-rectangular compensation boundary shape applicable to different postures, generate a dynamic mask prototype driven by feature posture points based on the non-rectangular compensation boundary shape, and deformation rules that change continuously with feature posture points.

[0176] Stabilization and reinforcement module: Based on the dynamic mask prototype and deformation rules, adaptive deformation control processing is performed on the compensation area to obtain the angular compensation area of ​​dynamic change of two-dimensional attitude points, and image stabilization processing is performed in the angular compensation area to output the stabilized image sequence.

[0177] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A pixel displacement compensation based image stabilization method, characterized in that: The method comprises the following steps: Obtaining attitude data and pixel displacement of a virtual gimbal in a motion range, performing attitude sensitivity analysis on the pixel displacement under different attitudes, and identifying key attitude segments that cause boundary conflict; Collecting spatial distribution characteristics of the pixel displacement corresponding to the key attitude segments, constructing an attitude-compensation range, and performing compensation capability analysis on the attitude-compensation range to extract a boundary sensitive region in which a feature attitude point dynamically changes; Based on the boundary sensitive region, a non-rectangular compensation boundary form applicable to different attitudes is extracted, a dynamic mask prototype driven by the feature attitude point is generated based on the non-rectangular compensation boundary form, and a deformation rule that continuously changes with the feature attitude point is extracted; Based on the dynamic mask prototype and the deformation rule, adaptive deformation control processing is performed on the compensation region to obtain an angular compensation region in which a two-dimensional attitude point dynamically changes, image stabilization processing is performed on the angular compensation region, and a stabilized image sequence is output; The compensation capability analysis process comprises the following steps: Based on the pitch angle and roll angle corresponding to the key attitude segments, the attitude data is bound with the motion instability coefficient of the strip region to establish an attitude-compensation range; All key attitude segments are traversed to draw a curved surface in which the pixel compensable range changes with the pitch angle and roll angle, and local maximum points and local minimum points of the attitude-compensation range on the curved surface are identified; The two-dimensional attitude point corresponding to the local minimum point is taken as a feature attitude point; The global instability coefficient corresponding to the feature attitude point and the motion instability coefficient of the strip region are obtained; The motion instability coefficient of the strip region and the global instability coefficient are compared and analyzed to determine the boundary sensitive region; The motion instability coefficient of the strip region is obtained by the following steps: Pixel displacement vector data of continuous multiple frames of images in each key attitude segment is extracted; Each frame of image is divided into three concentric strip regions, i.e., a central region, a transition region, and an edge region; The average module length and the average direction angle variance of all pixel displacement vectors in each strip region are calculated; The average module length and the direction angle variance are multiplied to obtain the motion instability coefficient of the strip region; The non-rectangular compensation boundary form is extracted by the following steps: Based on the feature attitude point and the spatial positions of all boundary sensitive regions marked at the feature attitude point and the motion instability coefficient of the strip region, the boundary sensitive region is modeled as a repulsive energy field; An initial rectangular compensation region boundary of the image is obtained, the boundary is discretized into dense and uniformly distributed control points, the control points are connected to form an elastic polygon contour; Based on each control point on the elastic polygon contour, the total repulsive force vector of each control point in all repulsive energy fields of the boundary sensitive regions is extracted; Based on the total repulsive force vector, all control points are iteratively deformed and constrained to solve, and the non-rectangular compensation boundary form is extracted.

2. The image stabilization method based on pixel displacement compensation according to claim 1, characterized in that: The attitude sensitivity analysis is performed in the following manner: The pitch angle and roll angle are extracted from the attitude data to obtain a sensitivity value; A data point set with the pitch angle and roll angle as coordinates and the sensitivity value as a feature is constructed. A two-dimensional attitude plane composed of pitch angle and roll angle is constructed based on a data point set, and a density-based spatial clustering analysis is performed on the sensitivity characteristics of all data points in the two-dimensional attitude plane; Through spatial clustering analysis, the entire attitude plane is divided into a low-sensitivity main region and a high-sensitivity outlier region; The range of the high-sensitivity outlier region that is continuously distributed in the pitch angle-roll angle plane is identified, and the range is defined as a key attitude segment prone to compensation boundary conflict.

3. The image stabilization method based on pixel displacement compensation according to claim 2, characterized in that: The sensitivity value is obtained in the following manner: Continuous video frames of the virtual gimbal are obtained, and pixel displacement analysis is performed to calculate the global motion vector of the continuous image frames as the pixel displacement; The lateral pixel displacement increment and the longitudinal pixel displacement increment are decomposed from the global motion vector; The ratio of the lateral pixel displacement increment to the pitch angle increment is calculated to obtain the sensitivity of the lateral pixel displacement to the pitch angle; The ratio of the longitudinal pixel displacement increment to the roll angle increment is calculated to obtain the sensitivity of the longitudinal pixel displacement to the roll angle; The calculated sensitivity of the lateral pixel displacement to the pitch angle and the sensitivity of the longitudinal pixel displacement to the roll angle are bound with the corresponding pitch angle and roll angle, respectively, to form a data point set with the sensitivity value as the characteristic, with the pitch angle and the roll angle as the coordinates.

4. The image stabilization method based on pixel displacement compensation according to claim 1, characterized in that: The iterative deformation and constraint solving are performed in the following manner: Based on the total repulsion force vector of each repulsion point, the constraint rule of iterative deformation is determined; The local curvature of the elastic polygonal contour is calculated, and smoothing control is performed on the elastic polygonal contour; Based on the constraint rule and the smoothing control, the control force of all control points reaches convergence or reaches the maximum number of iterations, and the optimal non-rectangular compensation boundary form under the current characteristic attitude point is established.

5. The image stabilization method based on pixel shift compensation according to claim 1, characterized in that: The image stabilization processing in the angular compensation region is performed in the following manner: Based on the angular compensation region formed by the dynamic changes of the two-dimensional attitude points, weight fusion allocation processing is performed; Based on the pixels in the angular compensation region after weight fusion allocation, position correction processing is performed; Based on the video frames after position correction processing, feathering fusion processing is performed, and a continuous stabilized image sequence is output.

6. The image stabilization method based on pixel displacement compensation according to claim 5, characterized in that: The angular compensation region is obtained in the following manner: The video frames after preliminary adjustment are obtained, and boundary constraints are added to the compensation boundary; based on the video frames after adding boundary constraints, an angular quantization rule is established in combination with the attitude-compensation range: The real-time two-dimensional attitude points are converted into compensation region boundary angle parameters through the angular quantization rule, forming an angular compensation region that dynamically changes with the two-dimensional attitude points.

7. A pixel shift compensation based image stabilization system for implementing the pixel shift compensation based image stabilization method of any one of claims 1-6, characterized in that: The following modules are included: An attitude acquisition module for acquiring attitude data and pixel displacement of a virtual gimbal in a motion interval, performing attitude sensitivity analysis on pixel displacement under different attitudes, and identifying a key attitude segment prone to compensation boundary conflict; A compensation analysis module for collecting spatial distribution characteristics of pixel displacement corresponding to the key attitude segment, constructing an attitude-compensation range, and extracting a boundary sensitive region that dynamically changes with a characteristic attitude point through compensation ability analysis on the attitude-compensation range; The rule determination module extracts a non-rectangular compensation boundary form suitable for different postures based on the boundary sensitive region, generates a dynamic mask prototype driven by a feature posture point based on the non-rectangular compensation boundary form, and generates a deformation rule continuously changing with the feature posture point; The stability enhancement and reinforcement module performs adaptive deformation control processing on the compensation region based on the dynamic mask prototype and the deformation rule, obtains an angular compensation region dynamically changing with a two-dimensional posture point, performs image stability enhancement processing on the angular compensation region, and outputs a stability enhancement image sequence.

Images