A pod drone video anti-shake method and system

Abstract

The present invention discloses a pod drone video anti-shake method and system, including: S1: collecting real-time wind data and video jitter data, and associating the two data according to the time axis; S2: constructing a high-altitude wind compensation model, and predicting the real wind data and wind direction data according to the video jitter data, wind data and wind direction data; S3: constructing a video jitter processing model, and extracting the feature points of the video jitter data; S4: calculating the jitter direction and amplitude according to the feature points of the video jitter data; S5: performing jitter compensation on the video according to the jitter direction and amplitude to obtain optimized video data. The present invention aims to replace the drone wind sensor by predicting the high-altitude wind force and wind speed; at the same time, giving higher weights to important key points, so as to solve the problems of high cost of traditional pod drone video anti-shake methods, failure to consider the influence of high-altitude wind, and poor anti-shake effect caused by the influence of the importance of key points.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a video anti-shake method and system for a pod unmanned aerial vehicle. Background Art

[0002] Pod drones are usually heavier because they are equipped with a pod system. Therefore, when flying and shooting videos, the body shaking caused by high-altitude wind is more obvious. Traditional anti-shake technologies, such as mechanical gimbals and electronic anti-shake, focus on mechanical shaking and the movement of the device itself, and often fail to fully consider the impact of high-altitude wind on the stability of the drone, thus affecting the stability of the final video.

[0003] At present, the main methods to solve the jitter of pod drone videos include mechanical gimbals and electronic image stabilization technology. Mechanical gimbals use brushless motors and rocker arms to physically isolate jitters, which can provide a larger anti-shake range and are suitable for repairing image jitters caused by larger movements. However, mechanical gimbals are usually heavy, take up space, and are expensive. Electronic image stabilization technology achieves the anti-shake effect at the software level by aligning videos of different frames and cutting unstable images. Although it does not add extra weight, it has limited effect when dealing with large jitters. Summary of the invention

[0004] In view of this, the present invention aims to provide a pod drone video stabilization method and system, aiming to solve the problems of high cost, failure to consider the influence of high-altitude wind, and poor stabilization effect caused by the different importance of different key points in traditional pod drone video stabilization methods.

[0005] A pod drone video anti-shake method comprises the following steps:

[0006] S1: Collect real-time wind data and video jitter data, and associate the two data along the time axis;

[0007] S2: Build a high-altitude wind compensation model to predict the real wind data and wind direction data based on the video jitter data, wind data and wind direction data;

[0008] S3: Build a video jitter processing model and extract feature points of video jitter data;

[0009] S4: Calculating the shaking direction and amplitude according to the feature points of the video shaking data;

[0010] S5: Perform jitter compensation on the video according to the jitter direction and amplitude to obtain optimized video data.

[0011] Furthermore, the step S1 collects real-time wind data and video jitter data, and associates the two data according to the time axis, including:

[0012] S11: A wind sensor is installed at the lower end of the pod drone, and the wind sensor collects and records the wind speed data and wind direction data around it in real time;

[0013] S12: Synchronously record the attitude data of the pod drone equipped with a wind sensor, including pitch angle, roll angle and yaw angle;

[0014] S13: Synchronize the wind force data, wind direction data, pod drone attitude data and video jitter data frame by frame according to the time axis;

[0015] By collecting wind data, wind direction data, drone attitude data and video jitter data, and correlating these data in time, a mapping relationship between various factors can be established, which is the basis for subsequent jitter compensation.

[0016] Traditional video stabilization methods usually rely on sensors such as inertial measurement units, accelerometers, and gyroscopes to directly measure the attitude changes of drones for stabilization. These methods focus on the motion characteristics of drones, but do not specifically consider the impact of environmental wind. The stabilization effect is poor when there is high-altitude wind. For pod drones equipped with wind sensors, the data collected by the wind sensors are not the wind force and direction data in the real environment, and there are errors.

[0017] In contrast, by establishing a mapping relationship between wind force and video jitter data and using algorithms to predict the actual wind force and wind direction conditions, the error caused by the drone rotor wind can be reduced and the data quality can be improved.

[0018] Furthermore, the step S2 constructs a high-altitude wind compensation model to predict the wind speed data and wind direction data at the next moment according to the video jitter data, including:

[0019] S21: Use a convolutional neural network to scan the video jitter data at the first moment and the second moment to obtain the video data features at the first moment and the video data features at the second moment, and the calculation method is:

[0020] F1 = MobileNet (I1)

[0021] F2 = MobileNet (I2)

[0022] Among them, F1 and F2 are the video data features at the first moment and the second moment respectively, MobileNet() is the MobileNet network operation, I1 and I2 are the video jitter data at the first moment and the second moment respectively;

[0023] MobileNet is a lightweight deep neural network that uses depthwise separable convolution to reduce computation and model size while still maintaining high accuracy. It is very suitable for application scenarios that require real-time performance and small number of parameters.

[0024] S22: Calculate the difference between the video data feature at the first moment and the video data feature at the second moment to obtain the video change feature at the second moment. The calculation method is:

[0025] ΔF=F2-F1

[0026] Among them, ΔF is the video change feature at the second moment;

[0027] ΔF represents the change in the characteristics of the video data from the first moment to the second moment. The unchanged part will be eliminated in the subtraction operation. The calculation of ΔF can help the model understand the dynamic changes between video frames, which is helpful for the prediction of wind force and direction in the subsequent steps.

[0028] S23: Construct high altitude wind compensation module:

[0029] After splicing the video change features at the second moment with the pod drone attitude data at the second moment, they are used together with the wind force data and wind direction data at the first moment as the input of the gated loop unit to predict the real wind force data and real wind direction data at the second moment. The calculation method is:

[0030]

[0031] in, They are the actual wind force prediction value and the actual wind direction prediction value at the second moment, GRU() is the gated recurrent unit operation, [;] is the splicing operation, P2 is the pod drone attitude data at the second moment, W1 and D1 are the wind force data and wind direction data at the first moment, respectively.

[0032] The above process uses the changing characteristics of video data, the posture of the drone, and the wind force and wind direction data of the previous moment to predict the actual wind force and wind direction at the current moment, helping the model capture the dynamic changes of time series data and improve the accuracy of wind force prediction;

[0033] Compared with traditional wind prediction methods, such as prediction based on physical models, the gated cycle unit used in the present invention does not require setting fixed parameters, but can self-adjust according to actual data, process data in real time and make predictions quickly.

[0034] Furthermore, the step S3 constructs a video jitter processing model to extract feature points of the video jitter data, including:

[0035] S31: Use SIFT algorithm to identify key points in video frames;

[0036] S32: Calculate the weight of each key point. The calculation method is:

[0037] s i =qf i

[0038]

[0039] Among them, s i is the feature vector f of the key point i Similarity with the query vector q, i is the key point number index, a i is the attention weight for each key point, exp() is the exponent of the natural constant, s j is the similarity between the feature vector of another key point and the query vector, and j is the count index;

[0040] Step S32 introduces an attention mechanism to calculate the importance of feature points, providing a basis for subsequent jitter compensation. Traditional methods evenly consider all detected feature points, ignoring the differences in the influence of different feature points on video jitter. Since they do not distinguish the importance of feature points, traditional methods pay too much attention to some feature points with less actual influence, thereby reducing the overall jitter compensation effect. In contrast, step S32 uses the attention mechanism to identify a group of feature points that are most critical to image jitter compensation, ignoring or reducing feature points with lower weights and less contribution to jitter compensation, so that in the subsequent jitter compensation processing, more reference is made to the jitter conditions of these important feature points, thereby improving the overall jitter compensation effect.

[0041] S33: Use the optical flow algorithm to track the movement of feature points in consecutive frames.

[0042] Furthermore, the step S4 calculates the shaking direction and amplitude according to the feature points of the video shaking data, including:

[0043] S41: For each tracked feature point, calculate its motion vector between consecutive frames, the calculation method is:

[0044]

[0045] in, is the motion vector of the tracked feature point moving from the uth frame to the u+1th frame, is the horizontal coordinate of the tracked i-th feature point in the u+1-th frame image, is the ordinate of the tracked i-th feature point in the u+1-th frame image, is the horizontal coordinate of the tracked i-th feature point in the u-th frame image, is the ordinate of the tracked i-th feature point in the u-th frame image, where u is the frame number index;

[0046] S42: Establish a regression network, and determine the shaking direction and shaking amplitude of the picture according to the weights of different key points. The calculation method is:

[0047]

[0048] in, is the jitter direction of the entire image from the uth frame to the u+1th frame, n is the total number of key points, ∥∥ is the modulus of the orientation, and A is the jitter amplitude of the entire image from the uth frame to the u+1th frame;

[0049] The jitter direction and amplitude of the points with high attention weights have a greater impact on the jitter compensation of the entire picture. Step S42 takes into account the actual contribution of different feature points in jitter compensation, and determines the global jitter characteristics through regression network and weighted calculation, providing a clear direction and amplitude reference for subsequent jitter compensation.

[0050] Furthermore, the step S5 performs jitter compensation on the video according to the jitter direction and amplitude to obtain optimized video data, including:

[0051] S51: Construct the video compensation network input vector, the calculation method is:

[0052]

[0053] Among them, X t is the input vector of the video compensation network, [;] is the vector concatenation operation, E t is the shaking direction of the entire image at each moment, A t is the jitter amplitude of the entire image at each moment, is the predicted value of the real wind data at each moment, is the predicted value of the real wind direction data at each moment, P t The pod drone attitude data at each moment;

[0054] S52: construct a video compensation network and output transformation parameters of the transformation matrix. The calculation method is:

[0055] Y t =GRU(X t )

[0056] Y t ′ =MLP(Y t )

[0057] Among them, Y t is the output of the gated recurrent unit at each time, Y t′ is the transformation matrix parameter, MLP() is the fully connected operation;

[0058] Step S51 constructs the input vector X that combines the vector with multi-dimensional information t , comprehensively considering the drone attitude, image jitter direction and high-altitude wind impact; traditional video jitter compensation methods usually only rely on the video data itself, for example, the optical flow method only uses the motion information between video frames; compared with traditional methods, the present invention integrates multi-source data and improves the video compensation network's processing ability for multiple jitter sources, thereby improving the video anti-shake effect of the pod drone in complex environments;

[0059] S53: Use exponential smoothing method to smooth the transformation matrix parameters. The calculation method is:

[0060] S t =β·Y t ′ +(1-β)·S t-1

[0061] Among them, S t is the smoothing value at time t, β is the smoothing constant, S t-1 is the smoothed value at time t-1;

[0062] S54: Input the smoothing value at time t into the transformation matrix and output the compensated video frame. The calculation method is:

[0063] C t =M·S t

[0064] Among them, C t is the compensated frame, and M is the transformation matrix.

[0065] Compared with the prior art, the advantages of the present invention are:

[0066] (1) The present invention establishes a high-altitude wind compensation model to predict the actual wind force and wind direction conditions, thereby reducing the error caused by the UAV rotor wind and improving the anti-shake effect of the UAV video in a high-altitude wind environment. At the same time, the present invention distinguishes the importance of different feature points in the video and dynamically adjusts the shake compensation parameters to improve the anti-shake effect of the video.

[0067] (2) The present invention constructs a high-altitude wind compensation model, realizes real-time prediction of actual wind speed and wind direction, and provides important environmental data support for video jitter compensation. At the same time, the present invention introduces an attention mechanism to assign weights according to the relative importance of feature points, so that key feature points receive more attention in jitter compensation, solving the problem that non-important feature points have a greater impact on the direction and amplitude of video jitter, and improving the compensation accuracy.

[0068] (3) The present invention introduces an exponential smoothing method, and combines a gated recurrent unit and a multi-layer perceptron to dynamically adjust the transformation matrix parameters to adjust the jitter changes between frames in real time, making the video compensation effect more natural.

[0069] (4) In terms of video jitter processing, the traditional pure optical flow algorithm only considers the changes of a single pixel. The present invention combines the idea of ​​feature difference to extract the changes of the video at the feature level, which can comprehensively consider the local and overall pixel information, and has a better effect in extracting video jitter features. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of the process flow of the pod drone video anti-shake method provided by the present invention;

[0071] Figure 2 A schematic diagram of the algorithm flow of the high-altitude wind compensation model provided by the present invention. DETAILED DESCRIPTION

[0072] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0073] Embodiment 1: A method for stabilizing video of a pod drone, such as Figure 1 As shown, the following steps are included:

[0074] S1: Collect real-time wind data and video jitter data, and associate the two data along the time axis;

[0075] S11: Controlling the pod drone equipped with a wind sensor to hover in the working environment, and collecting and recording the surrounding wind speed and wind direction data in real time; the pod drone is equipped with a wind sensor at the lower end, and the wind sensor collects and records the surrounding wind speed and wind direction data in real time; in this way, the wind sensor can more accurately measure the wind speed and wind direction of the surrounding environment, and reduce the influence of the drone rotor wind on its measurement results;

[0076] S12: Synchronously record the attitude data of the pod drone, including pitch angle, roll angle and yaw angle;

[0077] S13: Synchronize the wind data, the pod drone attitude data and the video jitter data frame by frame according to the time axis;

[0078] By collecting wind data, wind direction data, drone attitude data and video jitter data, and correlating these data in time, a mapping relationship between various factors can be established, which is the basis for subsequent jitter compensation.

[0079] Traditional video stabilization methods usually rely on sensors such as inertial measurement units, accelerometers, and gyroscopes to directly measure the attitude changes of drones for stabilization. These methods focus on the motion characteristics of drones, but do not specifically consider the impact of environmental wind. The stabilization effect is poor when there is high-altitude wind. For pod drones equipped with wind sensors, the data collected by the wind sensors are not the wind force and direction data in the real environment, and there are errors.

[0080] In contrast, by establishing a mapping relationship between wind force and video jitter data and using algorithms to predict the actual wind force and wind direction conditions, the error caused by the drone rotor wind can be reduced and the data quality can be improved.

[0081] S2: Construct a high-altitude wind compensation model to predict the wind speed and direction data at the next moment based on the video jitter data, such as Figure 2 As shown;

[0082] S21: Use a convolutional neural network to scan the video jitter data at the first moment and the second moment to obtain the video data features at the first moment and the video data features at the second moment, and the calculation method is:

[0083] F1 = MobileNet (I1)

[0084] F2 = MobileNet (I2)

[0085] Among them, F1 and F2 are the video data features at the first moment and the second moment respectively, MobileNet() is the MobileNet network operation, I1 and I2 are the video jitter data at the first moment and the second moment respectively;

[0086] MobileNet is a lightweight deep neural network that uses depthwise separable convolution to reduce computation and model size while still maintaining high accuracy. It is very suitable for application scenarios that require real-time performance and small number of parameters.

[0087] S22: Calculate the difference between the video data feature at the first moment and the video data feature at the second moment to obtain the video change feature at the second moment. The calculation method is:

[0088] ΔF=F2-F1

[0089] Among them, ΔF is the video change feature at the second moment;

[0090] ΔF represents the change in the characteristics of the video data from the first moment to the second moment. The unchanged part will be eliminated in the subtraction operation. The calculation of ΔF can help the model understand the dynamic changes between video frames, which is helpful for the prediction of wind force and direction in the subsequent steps.

[0091] S23: Construct high altitude wind compensation module:

[0092] After splicing the video change features at the second moment with the pod drone attitude data at the second moment, they are used together with the wind force data and wind direction data at the first moment as the input of the gated loop unit to predict the real wind force data and real wind direction data at the second moment. The calculation method is:

[0093]

[0094] in, They are the actual wind force prediction value and the actual wind direction prediction value at the second moment, GRU() is the gated recurrent unit operation, [;] is the splicing operation, P2 is the pod drone attitude data at the second moment, W1 and D1 are the wind force data and wind direction data at the first moment, respectively.

[0095] The above process uses the changing characteristics of video data, the posture of the drone, and the wind force and wind direction data of the previous moment to predict the actual wind force and wind direction at the current moment, helping the model capture the dynamic changes of time series data and improve the accuracy of wind force prediction;

[0096] Compared with traditional wind prediction methods, such as prediction based on physical models, the gated cycle unit used in the present invention does not require setting fixed parameters, but can self-adjust according to actual data, process data in real time and make predictions quickly.

[0097] For example, a pod drone is performing a filming mission at an altitude of 200 meters. At this time, the wind sensor on the drone measures a wind speed of 10 meters per second and a wind direction of 30 degrees north-east. However, due to the influence of the drone's rotor wind, these data have errors. At the same time, the video recording equipment on the drone also shows some jitter. The drone's pitch angle: 0 degrees; yaw angle: 0 degrees; roll angle: -2 degrees;

[0098] The video jitter data, wind force data, wind direction data, and drone attitude data were input into the high-altitude wind compensation model, and the predicted wind force was 12.1 m / s and the wind direction was 27 degrees north-east;

[0099] In order to verify the accuracy of the high-altitude wind compensation prediction results, more accurate high-altitude wind data was obtained by deploying weather balloons in the sky. The results were: true wind speed: 11.5 m / s; true wind direction: 27 degrees north-east;

[0100] Comparing the model prediction results with the real data measured by meteorological sensors, it can be seen that the predicted wind force differs from the actual wind force by 0.6 m / s, and the predicted wind direction differs from the actual wind direction by 2 degrees, and the error range is acceptable; therefore, the algorithm can effectively offset most of the errors caused by rotor wind and provide relatively accurate high-altitude wind forecasts.

[0101] S3: Build a video jitter processing model and extract feature points of video jitter data;

[0102] S31: Use SIFT algorithm to identify key points in video frames;

[0103] S32: Calculate the weight of each key point. The calculation method is:

[0104]

[0105] Among them, s i is the feature vector f of the key point i Similarity with the query vector q, i is the key point number index, a i is the attention weight for each key point, exp() is the exponent of the natural constant, s j is the similarity between the feature vector of another key point and the query vector, and j is the count index;

[0106] Step S32 introduces an attention mechanism to calculate the importance of feature points, providing a basis for subsequent jitter compensation. Traditional methods evenly consider all detected feature points, ignoring the differences in the influence of different feature points on video jitter. Since they do not distinguish the importance of feature points, traditional methods pay too much attention to some feature points with less actual influence, thereby reducing the overall jitter compensation effect. In contrast, step S32 uses the attention mechanism to identify a group of feature points that are most critical to image jitter compensation, ignoring or reducing feature points with lower weights and less contribution to jitter compensation, so that in the subsequent jitter compensation processing, more reference is made to the jitter conditions of these important feature points, thereby improving the overall jitter compensation effect.

[0107] For example, a pod drone is shooting an aerial video of a street. In this scenario, the SIFT algorithm detects and extracts various feature points, including corners of buildings, street markings, and edges of vehicles. After the attention mechanism is calculated, the boundary points of stationary objects such as corners of houses, manhole covers on the ground, road signs, and intersection corners are calculated as more important feature points with higher weights, stable geometric features, and higher detectability. Objects such as pedestrians and vehicles are usually given lower weights because their positions change quickly and are unstable, reducing their adverse effects on jitter compensation. This differentiated processing allows the jitter compensation algorithm to pay more attention to stable feature points, thereby providing a more accurate and natural anti-shake effect.

[0108] S33: Use the optical flow algorithm to track the movement of feature points in consecutive frames.

[0109] S4: Calculate the shaking direction and amplitude according to the feature points of the video shaking data;

[0110] S41: For each tracked feature point, calculate its motion vector between consecutive frames, the calculation method is:

[0111]

[0112] in, is the motion vector of the tracked feature point moving from the uth frame to the u+1th frame, is the horizontal coordinate of the tracked i-th feature point in the u+1-th frame image, is the ordinate of the tracked i-th feature point in the u+1-th frame image, is the horizontal coordinate of the tracked i-th feature point in the u-th frame image, is the ordinate of the tracked i-th feature point in the u-th frame image, where u is the frame number index;

[0113] S42: Establish a regression network, and determine the shaking direction and shaking amplitude of the picture according to the weights of different key points. The calculation method is:

[0114]

[0115]

[0116] in, is the jitter direction of the entire image from the uth frame to the u+1th frame, n is the total number of key points, ∥∥ is the modulus of the orientation, and A is the jitter amplitude of the entire image from the uth frame to the u+1th frame;

[0117] The jitter direction and amplitude of the point with high attention weight have a greater impact on the jitter compensation of the entire picture. Step S42 considers the actual contribution of different feature points in jitter compensation, determines the global jitter characteristics through regression network and weighted calculation, and provides a clear direction and amplitude reference for subsequent jitter compensation;

[0118] For example, a pod drone is shooting a video, and multiple moving objects appear in the video:

[0119] The corners of buildings and road signs on the ground show a small and relatively consistent movement direction between two consecutive frames, affected by wind or small changes in the drone's posture. When pedestrians are moving, their feature points change direction rapidly and irregularly.

[0120] If the attention mechanism is not introduced, the jitter directions of all feature points will be considered averagely, and the system will mistake the slight movement of pedestrians for screen jitter, resulting in the jitter direction and amplitude being unable to accurately reflect the overall screen stability. After the attention mechanism is introduced, the pedestrian feature points are given smaller weights, the impact of the irregular movement of pedestrians on jitter compensation is suppressed, and the anti-shake processing results more accurately reflect the actual jitter of the video.

[0121] S5: Perform jitter compensation on the video according to the jitter direction and amplitude to obtain optimized video data;

[0122] S51: Construct the video compensation network input vector, the calculation method is:

[0123]

[0124] Among them, X t is the input vector of the video compensation network, [;] is the vector concatenation operation, E t is the shaking direction of the entire image at each moment, A t is the jitter amplitude of the entire image at each moment, is the predicted value of the real wind data at each moment, is the predicted value of the real wind direction data at each moment, P t The pod drone attitude data at each moment;

[0125] S52: construct a video compensation network and output transformation parameters of the transformation matrix. The calculation method is:

[0126] Y t =GRU(X t )

[0127] Y t ′ =MLP(Y t )

[0128] Among them, Y t is the output of the gated recurrent unit at each time, Y t ′ is the transformation matrix parameter, MLP() is the fully connected operation;

[0129] Step S51 constructs the input vector X that combines the vector with multi-dimensional information t , comprehensively considering the drone attitude, image jitter direction and high-altitude wind impact; traditional video jitter compensation methods usually only rely on the video data itself, for example, the optical flow method only uses the motion information between video frames; compared with traditional methods, the present invention integrates multi-source data and improves the video compensation network's processing ability for multiple jitter sources, thereby improving the video anti-shake effect of the pod drone in complex environments;

[0130] S53: Use exponential smoothing method to smooth the transformation matrix parameters. The calculation method is:

[0131] S t =β·Y t ′ +(1-β)·S t-1

[0132] Among them, S t is the smoothing value at time t, β is the smoothing constant, S t-1 is the smoothed value at time t-1;

[0133] S54: Input the smoothing value at time t into the transformation matrix and output the compensated video frame. The calculation method is:

[0134] C t =M·S t

[0135] Among them, C t is the compensated frame, and M is the transformation matrix.

[0136] The present invention is particularly suitable for pod drones in scenes where the wind is strong at high altitudes and the shooting content contains a large number of dynamic objects; in these complex scenes, traditional video stabilization methods often fail to provide ideal jitter compensation effects because they ignore the impact of high-altitude wind on the drone and fail to effectively distinguish the importance of different feature points; the present invention constructs a high-altitude wind compensation model and an attention mechanism, fully considers the impact of wind and the importance of feature points when processing jitter compensation, effectively offsets the impact of wind, and automatically identifies key feature points, ultimately outputting a stable video image.

[0137] Embodiment 2: The present invention also discloses a pod drone video stabilization system, which is used to implement the pod drone video stabilization method, including:

[0138] Data collection and association module: collects real-time wind data and video jitter data, and associates the two types of data according to the time axis;

[0139] High-altitude wind compensation module: Build a high-altitude wind compensation model to predict the real wind data and wind direction data based on video jitter data, wind data and wind direction data;

[0140] Video jitter processing module: build a video jitter processing model and extract feature points of video jitter data;

[0141] Jitter calculation module: calculates the jitter direction and amplitude according to the feature points of the video jitter data;

[0142] Jitter compensation module: performs jitter compensation on the video according to the jitter direction and amplitude to obtain optimized video data.

[0143] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0145] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A pod drone video anti-shake method, characterized in that: The following steps are involved: S1: Collect real-time wind data and video jitter data, and associate the two data according to the time axis; including: S11: A wind sensor is installed at the lower end of the pod drone, and the wind sensor collects and records the wind speed data and wind direction data around it in real time; S12: Synchronously record the attitude data of the pod drone, including pitch angle, roll angle and yaw angle; S13: Synchronizing the wind force data, wind direction data, pod drone attitude data and video jitter data frame by frame according to the time axis; S2: Construct a high-altitude wind compensation model, and predict the real wind data and wind direction data according to the video jitter data, wind data and wind direction data; use a convolutional neural network to scan the video jitter data at the first moment and the second moment, obtain the video data features at the first moment and the video data features at the second moment, and subtract the video data features at the second moment from the video data features at the first moment to obtain the video change features at the second moment; then use the video change features at the second moment, the pod drone attitude data at the second moment, the wind data at the first moment, and the wind direction data at the first moment as the input of the gated recurrent unit to predict the real wind data and the real wind direction data at the second moment; including: S21: Use a convolutional neural network to scan the video jitter data at the first moment and the second moment to obtain the video data features at the first moment and the video data features at the second moment, and the calculation method is: F1 = MobileNet (I1) F2 = MobileNet (I2) Among them, F1 and F2 are the video data features at the first moment and the second moment respectively, MobileNet() is the MobileNet network operation, I1 and I2 are the video jitter data at the first moment and the second moment respectively; S22: Calculate the difference between the video data feature at the first moment and the video data feature at the second moment to obtain the video change feature at the second moment. The calculation method is: ΔF=F2-F1 Among them, ΔF is the video change feature at the second moment; S23: Construct high altitude wind compensation module: After splicing the video change features at the second moment with the pod drone attitude data at the second moment, they are used together with the wind force data and wind direction data at the first moment as the input of the gated loop unit to predict the real wind force data and real wind direction data at the second moment. The calculation method is: in, are the real wind force prediction value and the real wind direction prediction value at the second moment, GRU() is the gated recurrent unit operation, [;] is the splicing operation, P2 is the pod drone attitude data at the second moment, W1 and D1 are the wind force data and wind direction data at the first moment, respectively; S3: Construct a video jitter processing model to extract feature points of video jitter data; including: S31: Use SIFT algorithm to identify key points in video frames; S32: Calculate the weight of each key point. The calculation method is: s i =qf i Among them, s i is the feature vector f of the key point i Similarity with the query vector q, i is the key point number index, a i is the attention weight for each key point, exp() is the exponent of the natural constant, s j is the similarity between the feature vector of another key point and the query vector, and j is the count index; S33: Use the optical flow algorithm to track the motion of feature points in consecutive frames; S4: Calculating the shaking direction and amplitude according to the feature points of the video shaking data; S5: Perform jitter compensation on the video according to the jitter direction and amplitude to obtain optimized video data.

2. The pod drone video anti-shake method according to claim 1, characterized in that: The S4 step includes: S41: For each tracked feature point, calculate its motion vector between consecutive frames, the calculation method is: in, is the motion vector of the tracked feature point moving from the uth frame to the u+1th frame, is the horizontal coordinate of the tracked i-th feature point in the u+1-th frame image, is the ordinate of the tracked i-th feature point in the u+1-th frame image, is the horizontal coordinate of the tracked i-th feature point in the u-th frame image, is the ordinate of the tracked i-th feature point in the u-th frame image, where u is the frame number index; S42: Establish a regression network, and determine the shaking direction and shaking amplitude of the picture according to the weights of different key points. The calculation method is: in, is the jitter direction of the entire image from the uth frame to the u+1th frame, n is the total number of key points, ∥∥ is the modulus of the orientation quantity, and A is the jitter amplitude of the entire image from the uth frame to the u+1th frame.

3. The pod drone video anti-shake method according to claim 1, characterized in that: The S5 step includes constructing a video compensation network input vector based on the jitter direction, jitter amplitude, wind force and wind direction data predicted values ​​at each moment of the entire image, and the pod drone attitude data; the jitter direction at each moment is calculated by the attention mechanism; after the transformation parameters of the transformation matrix output by the video compensation network are output, the transformation matrix parameters are smoothed using an exponential smoothing method, and finally the compensated video frame is output.

4. The pod drone video anti-shake method according to claim 2, characterized in that: The S5 step includes: S51: Construct the video compensation network input vector, the calculation method is: Among them, X t is the input vector of the video compensation network, [;] is the vector concatenation operation, E t is the shaking direction of the entire image at each moment, A t is the jitter amplitude of the entire image at each moment, is the predicted value of the real wind data at each moment, is the predicted value of the real wind direction data at each moment, P t The pod drone attitude data at each moment; S52: construct a video compensation network and output transformation parameters of the transformation matrix. The calculation method is: AND t =GRU(X t ) AND' t =MLP(Y t ) Among them, Y t is the output of the gated recurrent unit at each time, Y′ t is the transformation matrix parameter, MLP() is the fully connected operation; S53: Use exponential smoothing method to smooth the transformation matrix parameters. The calculation method is: S t =β·Y' t +(1-β)·S t-1 Among them, S t is the smoothing value at time t, β is the smoothing constant, S t-1 is the smoothed value at time t-1; S54: Input the smoothing value at time t into the transformation matrix and output the compensated video frame. The calculation method is: C t =M·S t Among them, C t is the compensated frame, and M is the transformation matrix.

5. A pod drone video anti-shake system, characterized in that: include: Data collection and association module: collects real-time wind data and video jitter data, and associates the two types of data according to the time axis; High-altitude wind compensation module: Build a high-altitude wind compensation model to predict the real wind data and wind direction data based on video jitter data, wind data and wind direction data; Video jitter processing module: build a video jitter processing model and extract feature points of video jitter data; Jitter calculation module: calculates the jitter direction and amplitude according to the feature points of the video jitter data; Jitter compensation module: performs jitter compensation on the video according to the jitter direction and amplitude to obtain optimized video data; To realize the pod drone video stabilization method as described in any one of claims 1-4.

Images