Video Stabilization Method and System for Visual Monitoring of Flapping Wing Flying Robots

Abstract

The present invention relates to a video stabilization method and system for vision monitoring of flapping-wing flying robots. The method includes: generating feature points and feature point trajectories of visual images; performing periodic segmentation on the feature point trajectories, obtaining the video frame rate, and calculating the instantaneous frequency of the feature point trajectories; adjusting the time intervals of the feature points according to the instantaneous frequency, then performing linear sampling to generate the original trajectories; interpolating the above-mentioned original trajectories to obtain interpolated trajectories; averaging the interpolated trajectories to obtain smooth trajectories; adjusting the time intervals of the smooth trajectories and performing linear sampling to obtain the final smooth trajectories; and performing frame-by-frame image correction based on the original trajectories and the final smooth trajectories to obtain the stabilized video. The present invention can process each feature point trajectory separately, is applicable to feature point trajectories of different lengths, is convenient for implementing parallel operations, has a high computing efficiency, and is applicable to real-time online processing.

Description

Technical Field

[0001] The present invention relates to a video stabilization method and system for visual monitoring of flapping-wing flying robots, belonging to the field of UAV vision technology. Background Art

[0002] Different from rotor or fixed-wing UAVs, bionic flapping-wing UAVs achieve flight by imitating the flapping of natural birds. When a camera is installed on a flapping-wing UAV for aerial photography during flight, the scenery in the video will vibrate violently due to the flapping motion of the flapping-wing UAV, which makes the video viewers feel uncomfortable. Therefore, it is necessary to use a video stabilization method to process the video to obtain a stable aerial photography video.

[0003] Existing video stabilization methods mainly smooth the feature point trajectories obtained by the typical Kanade-Lucas-Tomasi (KLT) feature point tracking method for the video, and then use a specific smoothing method to smooth the feature point trajectories. Finally, based on the feature point trajectories before and after smoothing, the typical Structure-Preserving Warping (SPW) method is used to perform frame-by-frame image correction to obtain a stabilized video. The difficulty of the video stabilization method lies in smoothing the feature point trajectories. Since the scenery changes in the aerial photography video, the corresponding feature point trajectories are often scattered and discontinuous, and general smoothing methods such as Gaussian smoothing or moving average smoothing cannot be directly used for processing. Therefore, a specific method is needed to smooth the feature point trajectories of the aerial photography video.

[0004] Existing video stabilization methods, such as subspace method, trajectory enhancement method, epipolar geometry method, etc., all process the feature point trajectories of the aerial photography video by moving a short time sampling interval. The disadvantages of their processing methods are: 1) poor processing ability for feature point trajectories with short lengths; 2) in real-time processing, for two continuously acquired videos, it is necessary to ensure that there is a large overlap between them to achieve smooth connection of feature points, and the calculation is relatively redundant.

[0005] To solve the above problems, it is necessary to propose a new video stabilization method according to the flight characteristics of flapping-wing UAVs. Summary of the Invention

[0006] The present invention provides a video stabilization method and device for visual monitoring of flapping-wing flying robots, aiming to solve at least one of the technical problems existing in the prior art.

[0007] The technical solution of the present invention on the one hand relates to a video stabilization method for visual monitoring of flapping-wing flying robots. The method according to the present invention includes the following steps:

[0008] S10. Obtain the feature points of the robot vision image through the feature point detection method; generate the feature point trajectory of the robot vision video through feature point tracking or feature point matching;

[0009] S20. Extract the feature points according to the acceleration in the Y direction of the feature points, generate the envelope of the feature point trajectory, and perform periodic segmentation on the feature point trajectory according to the peak points of the envelope to form several subsets;

[0010] S30. Based on the above subsets, and obtain the video frame rate, and calculate the instantaneous frequency of the feature point trajectory;

[0011] S40. Set a reference frequency according to the instantaneous frequency, adjust the time interval between feature points according to the reference frequency, and then set a fixed time interval according to the video frame rate, and perform linear sampling on the adjusted feature point trajectory at the fixed time interval to generate an original trajectory;

[0012] S50. Interpolate the above original trajectory to obtain an interpolated trajectory;

[0013] S60. Calculate the average value of the interpolated trajectory to obtain a smoothed trajectory; adjust the time interval of the smoothed trajectory according to the reference frequency, and then perform linear sampling on the adjusted smoothed trajectory at the fixed time interval to obtain a final smoothed trajectory;

[0014] S70. Based on the original trajectory and the final smoothed trajectory, perform frame-by-frame image correction to obtain a stabilized video.

[0015] Furthermore, in step S1:

[0016] Obtain the feature point trajectory through the KLT tracking algorithm.

[0017] Furthermore, step S20 includes:

[0018] S21. The acceleration in the Y direction of the feature points is obtained through the following formula:

[0019]

[0020] where i represents the feature point serial number, (px, py) represents the feature point coordinates, and m represents the number of feature points;

[0021] S22. The envelope of the feature point trajectory uses the upper envelope, and the upper envelope is obtained through the following method:

[0022] When a i y the value of is greater than a i-1 y and a i+1y When it is, it will be recorded as ay i peak ; All ays peak The connection forms the upper envelope of ay.

[0023] Furthermore, in the step S30:

[0024] The instantaneous frequency is obtained through the following calculation:

[0025]

[0026] where r i represents the instantaneous frequency of the trajectory point i, r c represents the video frame rate, and nk represents the total number of trajectory points in the subset to which the trajectory point i belongs.

[0027] Furthermore, the step S40 includes:

[0028] S41. Set the reference frequency to the median of the instantaneous frequency;

[0029] S42. Adjust the time interval between feature points according to the following equation:

[0030]

[0031] where Δt(i) represents the adjusted time interval between the feature point i and the feature point i + 1, r f represents the reference frequency; r i represents the instantaneous frequency of the trajectory point i;

[0032] S43. Set the fixed time interval to 1 / r c ; Linearly sample the above-adjusted feature point trajectory at 1 / r c to generate the original trajectory.

[0033] Furthermore, in the step S50:

[0034] The interpolation trajectory is obtained through the following interpolation formula:

[0035]

[0036] where (k - 1)n f + h < j ≤ kn f + h, kn f + h ≤ m′

[0037] where, E h (j) represents the interpolation trajectory, Q(j) represents the original trajectory; n f represents the number of trajectory points in a single flutter cycle, r c represents the video frame rate, rf denotes the above-mentioned reference frequency, n f = r c / r f ; k represents the jitter cycle number, h represents the phase number, h ∈ [0, nf), j represents the trajectory point number, and j = knf + h.

[0038] Furthermore, in the step S60:

[0039] The final smoothed trajectory is calculated and obtained through the following formula:

[0040]

[0041] where S(j) represents the final smoothed trajectory, and E h (j) represents the interpolated trajectory, n f represents the number of trajectory points in a single flapping cycle, r c represents the video frame rate, rf represents the above-mentioned reference frequency, n f = r c / r f ; k represents the jitter cycle number, h represents the phase number, h ∈ [0, nf), j represents the trajectory point number, and j = knf + h.

[0042] Furthermore, in the step S70:

[0043] Perform frame-by-frame image correction through the CPW algorithm to obtain the stabilized video.

[0044] On the other hand, the technical solution of the present invention also relates to a video stabilization system for visual monitoring of a flapping-wing flying robot, including:

[0045] At least one camera carried by the flapping-wing flying robot;

[0046] A computer device connected to the camera, the computer device includes a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented.

[0047] The beneficial effects of the present invention are as follows:

[0048] The present invention provides a video stabilization method and system for vision monitoring of flapping-wing flying robots. Based on the characteristics of the periodic jitter of flapping-wing UAVs, the original feature point trajectory is divided into a smooth feature point trajectory and a periodic jitter trajectory. Multiple interpolation trajectories are obtained through interval sampling and interpolation, and the average of each interpolation trajectory is calculated to obtain a smooth trajectory, thereby obtaining a stabilized video. During the smoothing process of the feature point trajectory, the present invention can process each feature point trajectory separately, is applicable to feature point trajectories of different lengths, avoids the poor smoothing effect at the beginning and end of the video in traditional video stabilization based on time-domain smoothing, is convenient for parallel computing, has high computing efficiency, and is applicable to real-time online processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a basic flowchart of the method according to the present invention.

[0050] Figure 2 is a schematic diagram of the envelope line of the feature point trajectory in an embodiment of the present invention.

[0051] Figure 3 is a schematic diagram of the smoothing vector of the original frame image in an embodiment of the present invention.

[0052] Figure 4 is a schematic diagram of the CPW image correction effect in an embodiment of the present invention.

[0053] Figure 5 is a schematic diagram of the original image of the robot vision according to an embodiment of the present invention.

[0054] Figure 6 is a schematic diagram of the image of the robot vision after stabilization according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The following will clearly and completely describe the concept, specific structure, and technical effects generated by the present invention in combination with the embodiments and the drawings, so as to fully understand the purpose, solution, and effects of the present invention.

[0056] It should be noted that unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. The singular forms "a", "the", and "said" used herein are also intended to include the plural forms unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this technology belongs. The terms used in the description of this specification are only for describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0057] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, without departing from the scope of this disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language (such as "for example", "such as", etc.) provided herein is only intended to better illustrate the embodiments of the present invention and will not impose a limitation on the scope of the present invention unless otherwise required.

[0058] Referring to Figures 1 to 6 , in some embodiments, the technical solution of the present invention is used in a video stabilization system for visual monitoring of a flapping-wing flying robot, including: at least one camera carried by the flapping-wing flying robot, and a computer device connected to the camera. The computer device includes a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, a video stabilization method for visual monitoring of the flapping-wing flying robot is implemented.

[0059] Referring to Figure 1 , in some embodiments, the video stabilization method for visual monitoring of a flapping-wing flying robot according to the present invention at least includes the following steps:

[0060] S10. Obtain the feature points of the robot visual image through the feature point detection method; generate the feature point trajectory of the robot visual video through feature point tracking or feature point matching;

[0061] S20. Extract the feature points according to the acceleration in the Y direction of the feature points, generate the envelope of the feature point trajectory, and perform periodic segmentation on the feature point trajectory according to the peak points of the envelope to form several subsets;

[0062] S30. Based on the above subsets, and obtain the video frame rate, and calculate the instantaneous frequency of the feature points;

[0063] S40. Set a reference frequency according to the instantaneous frequency, adjust the time interval between the feature points according to the reference frequency, and then set a fixed time interval according to the video frame rate, and perform linear sampling on the adjusted feature point trajectory at the fixed time interval to generate an original trajectory;

[0064] S50. Interpolate the above original trajectory to obtain an interpolated trajectory;

[0065] S60. Take the average value of the interpolated trajectory to obtain a smoothed trajectory; adjust the time interval between the smoothed trajectories according to the reference frequency, and then perform linear sampling on the adjusted smoothed trajectory at the fixed time interval to obtain a final smoothed trajectory;

[0066] S70. Based on the original trajectory and the final smoothed trajectory, perform frame-by-frame image correction to obtain a stabilized video.

[0067] Specific implementation of step S10

[0068] The generation of the feature point trajectory of the robot vision image includes the following steps:

[0069] S11. By using feature point detection methods such as the Harris method, SIFT method, SURF method, FAST method, ORB method, etc., detect the coordinates of several pixels with special properties in the image, thereby generating the feature points of the image. Expressing the information of the entire graphic through the detected several feature points can reduce the amount of information and improve the calculation speed. Among them, the feature points in the image include two pieces of information: the detector and the descriptor. The detector represents the position information and is used to describe the position of the feature point in the image. The descriptor represents the feature information and is used for feature matching and for distinguishing from other feature points.

[0070] S12. By using the feature point tracking method or the feature point matching method, obtain all the pixel coordinates of the same feature point in a video and connect them to form a feature point trajectory. By obtaining multiple feature point trajectories in the video, generate the feature point trajectory. Specifically, the feature point tracking method (such as the optical flow tracking method) is based on the specified coordinates in the previous image. By calculating the optical flow vector, solve the coordinates of the specified coordinate point in the subsequent image. The optical flow vector refers to the vector between the coordinates of the previous image and the coordinates of the subsequent image. The feature point matching method determines the similarity degree of two feature points based on the distance between two descriptors (feature description vectors) between the feature points of the previous image and the feature points of the subsequent image, thereby determining whether the feature points of the front and rear images belong to the same feature point.

[0071] The feature point trajectory in the embodiment of the present invention is obtained by the Kanade-Lucas-Tomasi (KLT) method. KLT is actually composed of the Harris feature point detection method and the optical flow tracking method, and has the characteristics of relatively stable calculation structure, strong ability to resist video jitter, and good comprehensive effect.

[0072] Further, the Harris feature point detection method uses corner detection. Assuming the graphic offset is (u, v), then the E(u, v) generated by translating the image by (u, v) can be expressed as:

[0073]

[0074] Among them, E(u, v) can be obtained through the following formula:

[0075] I(x + u, y + v) = I(x, y) + I x u + I y v + O(u2 , v 2 )

[0076]

[0077] For the movement of the local WeChat, E(u, v) can be approximately expressed as:

[0078]

[0079] Wherein,

[0080]

[0081] E(u, v) is the covariance matrix, which represents the rate of change of intensity in all directions. The two eigenvalues of the covariance give the maximum average intensity change and the average intensity change in the vertical direction. Then: if both of these eigenvalues are low, it is determined that the current corner point is a homogeneous region; if one eigenvalue is high and the other is low, it is determined that the current corner point is on the edge; if both eigenvalues are high, it is determined that the current corner point is at the corner.

[0082] Perform the following threshold processing on the corner response function:

[0083] R = λ1λ2 - k(λ1 + λ2) 2

[0084] Wherein, R is the corner response function, and K is the last parameter of the cn::cornerHarris function in OpenCV. It is set that if R is greater than the threshold threshold, the local maximum value is extracted.

[0085] Furthermore, the optical flow tracking method obtains the optical flow vector through the following calculation:

[0086] Assume that within the m×m area around the target pixel, each pixel has the same optical flow vector V. Then, within a small neighborhood,

[0087] In the above formula, W2(x) is a window weight function, which makes the weighted value at the center of the neighborhood larger than that around it. For the n points X1…Xn in Ω, it is set that:

[0088] Let That is

[0089] 令 That is

[0090] Let That is

[0091] Therefore, the solution of the above equation can be obtained by the least squares method:

[0092] Finally, we get:

[0093] Specific implementation of step S20

[0094] S21. For each feature point trajectory, it can be expressed as The acceleration of the feature point in the Y direction is obtained through the following formula:

[0095]

[0096] where i represents the feature point serial number, (px, py) represents the feature point coordinates, and m represents the number of feature points;

[0097] S22. The envelope of the feature point trajectory includes an upper envelope and a lower envelope. Perform Gaussian smoothing on the acceleration a y The envelope is obtained through the following method:

[0098] When the value of a i y is greater than a i-1 y and a i+1 y , it is considered that a i y is a peak value, denoted as ay i peak ; conversely, when the value of a i y is less than a i-1 y and a i+1 y , it is considered that a i y is a valley value, denoted as ay i valley . Connect all ay peak , regarded as the upper envelope of a y ; connect all ay valley , regarded as the lower envelope of a y . The upper envelope of a y is the line segment L1 shown in the appendix Figure 2 , and the lower envelope of a y is the line segment L2 shown in the appendix Figure 2 .

[0099] It should be noted that for the envelope of the feature point trajectory in the embodiment of the present invention, either the upper envelope or the lower envelope can be adopted. Here, the upper envelope is specifically described. Specifically, in the upper envelope, when the value of ay i peak is greater than ay i-1peak and ay i+1 peak When, then ay i peak is regarded as the peak point of the upper envelope line, and the peak point is as shown in the attached Figure 2 dot D1 shown by the dot. The peak point divides the characteristic point trajectory into several subsets.

[0100] Specific implementation manner of step S30

[0101] The instantaneous frequency is obtained through the following calculation:

[0102]

[0103] where r i represents the instantaneous frequency of the trajectory point i, and the video frame rate is r c , n k represents the total number of trajectory points in the subset to which the trajectory point i belongs.

[0104] Specific implementation manner of step S40

[0105] Given the video frame rate r c , the time interval of the original trajectory point P(i) is 1 / r c . Assume that the time interval between point i and point i + 1 after adjustment is Δt(i). Take the median of the instantaneous frequencies of the trajectory points as the reference frequency r f . For the trajectory points with instantaneous frequency r i greater than r f , amplify Δt(i), and vice versa, as shown in equation (3).

[0106]

[0107] For the characteristic point trajectory P(i) with the adjusted time interval, the time information t(i) of its i-th point is as shown in equation (4-2):

[0108]

[0109] Since the time intervals between the trajectory points in the adjusted characteristic point trajectory P(i) are inconsistent, linearly resample P(i) with the adjusted time interval again at the time interval 1 / r C to obtain the trajectory Q(j) with consistent frequencies. j = 0, 1, 2... m' represents the sequence number of its trajectory points, and m' is the total number of trajectory points after sampling, as shown in equation (4-3):

[0110]

[0111] where [] represents rounding down.

[0112] Specific implementation of step S50

[0113] For Q(j), it can be regarded as the superposition of the smooth feature point trajectory and the periodic jitter feature point trajectory. Therefore, the expression is Equation (5-1):

[0114] Q(kn f +h) = S(kn f +h) + A(k)·J(kn f +h) (5-1)

[0115] Among them, S(knf+h) is the smooth feature point trajectory, and J(knf+h) is the jitter cycle trajectory with frequency rf. nf is the number of points in a single flutter cycle trajectory. Given the video frame rate as rc, nf = rc / rf. In a form similar to a trigonometric function, the trajectory point serial number j is expressed as j = knf+h, k represents the jitter cycle serial number, h represents the phase serial number, and h ∈ [0, nf). In addition, A(k) represents the jitter amplitude function, which is related to the cycle serial number k, meaning that the jitter amplitudes between different jitter cycles are different. Since J(knf+h) is a periodic function with a period of nf, Equations (5-2) and (5-3) hold.

[0116]

[0117] J(k·n f +h) = J(h) (5-3)

[0118] In order to obtain the smooth trajectory S(knf+h), combining Equation (6), Equation (4) is transformed into:[[]]

[0119] S(k·n f +h) = Q(k·n f +h) - A(k)·J(h) (5-4)

[0120] According to Equation (10), assuming h is fixed, the smooth trajectory S(knf+h) intersects with the trajectory Q(knf+h) - A(k)J(h) at h, nf+h, 2nf+h,.... Extract the point set {Q(h), Q(n f +h), Q(2n f +h),...} at the corresponding positions in the original trajectory Q(knf+h). Perform linear interpolation on this point set. Through the following linear interpolation function (5-5), obtain the interpolation trajectory Eh(j) with the same length as the trajectory Q(j).

[0121]

[0122] Among them, (k-1)n f +h < j ≤ kn f +h, knf +h ≤ m′ (5-5)

[0123] It should be noted that the interpolation trajectory in the embodiments of the present invention can be obtained by methods such as linear interpolation, quadratic spline interpolation, and cubic spline interpolation.

[0124] Specific implementation manner of step S60:

[0125] It can be approximately considered that the interpolation trajectory Eh(j) and the smooth trajectory S(j) only differ by A(k)J(h), as shown in equation (5-5).

[0126] S(j) ≈ E h (j) - A(k)·J(h) (6-1)

[0127] Respectively obtain the interpolation trajectory Eh for h ∈ [0, nf), substitute it into equation (6-1) and take the average, and combine with equation (5-2) to obtain equation (6-2):

[0128]

[0129] Perform linear sampling on the smooth trajectory S(j) according to the time information in equations (4-1) and (4-2) to obtain a smooth trajectory M(i) with the same time information as the original trajectory P(i), as shown in equation (6-3):

[0130]

[0131] Specific implementation manner of step S70:

[0132] Based on the original trajectory and the final smooth trajectory, use the typical Content-Preserving Warp (CPW) method to perform frame-by-frame image correction to obtain a stabilized video. Specifically, in each frame of video image, regard the position from the original trajectory of each feature point to the smooth trajectory position as a smooth vector. Based on the stable frame generation algorithm of grid-based warping - the Content-Preserving Warp (CPW) method, according to all the smooth vectors in each frame of video image, deform and distort the image to generate each frame of stabilized image. The specific processing steps are as follows:

[0133] S71. Divide each frame of image into several grids so that the original feature points and their smooth vectors belong to different grids. As shown in the Figure 3 attachment, in the original frame image, the dot D2 is the original feature point, and the straight line segment L3 is the smooth vector.

[0134] S72. Use the smooth vector between the original position and the smooth position of the feature point to deform the grid. Specifically,

[0135]

[0136] Among them, ～ V represents the set of grid vertices after all grid deformations, and E( ～ V) represents the overall cost, and E d ( ～ V) represents the data cost, and E s ( ～ V) represents the structure preservation cost, and E r ( ～ V) represents the regularization cost. The data cost, the structure preservation cost, and the regularization cost can be obtained by existing well-known methods.

[0137] Among them, ～ Most of the corner points in V are shared by four grids. By minimizing the overall cost function in (Equation 7-1 above) using the least squares method, the corner points of the optimal grid position in the distorted frame ～ V* are obtained. Then, for each grid cell, a local projective matrix from the original corner to the optimal distorted corner is estimated, and the projective matrix is used to reconstruct the pixel values in each distorted grid cell to generate the stabilized frame image (as Figure 4 shown).

[0138] For the experimental verification of the video stabilization method and system according to the present invention, the original image of the bionic flapping-wing UAV flight aerial video is as Figure 5 shown. After the original image is processed by the stabilization system, the stabilized image as shown in Figure 6 is output.

[0139] It should be recognized that the method steps in the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or computer instructions stored in a non-transitory computer-readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, for this purpose, the program can run on a dedicated integrated circuit programmed for this purpose.

[0140] In addition, the operations of the processes described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed jointly on one or more processors, by hardware, or by a combination thereof. The computer program includes a plurality of instructions executable by one or more processors.

[0141] Further, the method can be implemented in any type of computing platform operably connected to a suitable one, including but not limited to personal computers, minicomputers, mainframes, workstations, network or distributed computing environments, separate or integrated computer platforms, or communicating with charged particle tools or other imaging devices, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into the computing platform, such as a hard disk, optical read and / or write storage medium, RS1M, ROM, etc., such that it can be read by a programmable computer and can be used to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above-described steps in conjunction with a microprocessor or other data processor, the inventions described herein include these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0142] The computer program can be applied to the input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the transformed data represents physical and tangible objects, including a specific visual depiction of the physical and tangible objects generated on the display.

[0143] As described above, this is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. As long as it achieves the technical effects of the present invention by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation manners can have various different modifications and variations.

Claims

1. A video stabilization method for visual monitoring of a flapping-wing flying robot, characterized in that, The method includes the following steps: S10. Obtain the feature points of the robot vision image through the feature point detection method; generate the feature point trajectory of the robot vision video through feature point tracking or feature point matching; S20. Extract feature points according to the acceleration in the Y direction of the feature points, generate the envelope of the feature point trajectory, and perform periodic segmentation on the feature point trajectory according to the peak points of the envelope to form several subsets; S30. Based on the above subsets, and obtain the video frame rate, and calculate the instantaneous frequency of the feature point trajectory; S40. Set a reference frequency according to the instantaneous frequency, adjust the time interval between feature points according to the reference frequency, then set a fixed time interval according to the video frame rate, and linearly sample the adjusted feature point trajectory at the fixed time interval to generate an original trajectory; S50. Interpolate the above original trajectory to obtain an interpolated trajectory; S60. Calculate the average value of the interpolated trajectory to obtain a smoothed trajectory; adjust the time interval of the smoothed trajectory according to the reference frequency, and then linearly sample the adjusted smoothed trajectory at the fixed time interval to obtain a final smoothed trajectory; S70. Based on the original trajectory and the final smoothed trajectory, perform frame-by-frame image correction to obtain a stabilized video; Among them, the step S40 includes: S41. Set the reference frequency r f to the median of the instantaneous frequencies; S42. The time interval △t(i) between feature points is adjusted according to the following equation: ， Among them, △t(i) represents the adjusted time interval between feature point i and feature point i + 1, and r f represents the reference frequency; r i represents the instantaneous frequency of trajectory point i; the known video frame rate is r C ; S43. Set the time interval of the original trajectory point P(i) to 1 / r C ; At the time interval of 1 / r C Re - perform linear sampling on P(i) after adjusting the time interval to obtain the trajectory Q(j) with frequency uniformization, where j = 0, 1, 2... m' represents the trajectory point serial number, and m' is the total number of trajectory points after sampling; Among them, Q(j) is the superposition of the smoothed feature point trajectory and the periodic jitter feature point trajectory, and it is expressed as follows: ， where, S(kn f +h) is the smoothed feature point trajectory, and J(kn f +h) is the jitter cycle trajectory of frequency r f ; n f is the number of points in a single flutter cycle trajectory, n f = r C / r f ; where the trajectory point serial number j is expressed as j = kn f +h, k represents the jitter cycle serial number, h represents the phase serial number, h ∈ [0, n f ); A(k) represents the jitter amplitude function; Among them, according to J(kn f +h) is a periodic function with a period of n f we can get: ， ， Set h to be fixed, and the smooth feature point trajectory S(kn f +h) intersects with the trajectory Q(kn f +h)-A(k)J(h) at h, nf+h, 2nf+h... to extract the point set at the corresponding positions in the original trajectory Q(kn f +h), perform linear interpolation on this point set to obtain an interpolation trajectory Eh(j) with the same length as the trajectory Q(j). ​ 2. The method according to claim 1, wherein, In the step S10: Obtain the feature point trajectory through the KLT tracking algorithm.

3. The method according to claim 1, wherein, The step S20 includes: S21. The acceleration in the Y direction of the feature points is obtained through the following formula: ， Among them, i represents the serial number of the feature point, (p x , p y ) represents the coordinate of the feature point, and m represents the number of feature points; S22. The envelope of the feature point trajectory includes an upper envelope and a lower envelope. Perform Gaussian smoothing on the acceleration a y to obtain the upper envelope and the lower envelope in the following manner: When a i y is greater than a i-1 y and a i+1 y , a i y reaches its peak, denoted as a i ypeak ; conversely, when a i y is less than a i-1 y and a i+1 y , a i y reaches its trough, denoted as a i yvalley ; Connect all a i ypeak to obtain the upper envelope of a y ; Connect all a i yvalley to obtain the lower envelope of a y .

4. The method according to claim 1, wherein, In the step S30: The instantaneous frequency is obtained through the following calculation: ， where n k represents the total number of trajectory points in the subset to which the trajectory point i belongs.

5. The method according to claim 1, wherein In the step S50: The interpolated trajectory is calculated through the following interpolation formula: ， Among them, E h (j) represents the interpolation trajectory, and Q(j) represents the original trajectory.

6. The method according to claim 5, wherein In the step S60: The final smoothed trajectory is calculated through the following formula: ， Among them, S(j) represents the final smoothed trajectory.

7. The method according to claim 1, wherein In the step S70: Perform frame-by-frame image correction through the CPW algorithm to obtain a stabilized video.

8. A video stabilization system for visual monitoring of a flapping-wing flying robot, characterized in that, It includes: At least one camera carried by the flapping flight robot; A computer device connected to the camera, the computer device includes a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images