Anti-shake method for image transmission video of unmanned aerial vehicle

Abstract

The invention discloses an anti-shake method for an image transmission video of an unmanned aerial vehicle. The anti-shake method comprises the following steps: step 1, data acquisition; step 2, multi-frame optical flow analysis; step 3, feature extraction and jitter judgment; step 4, generating a jitter-free frame by the generative adversarial network; and step 5, post-treatment and synthesis. And step 6, performing real-time feedback and optimization. Through combination of multi-frame optical flow analysis and deep learning feature extraction, different types of jitters can be accurately identified. The accurate jitter judgment not only improves the reliability of detection, but also provides an important basis for subsequent processing, so that the video quality can be effectively guaranteed in practical application no matter in a complex environment or in a rapid movement condition, and the scheme can remarkably improve the video quality on the premise of keeping details and definition. The visual effect of the image is further enhanced through subsequent color correction and sharpening processing, so that the finally output video is smooth and has higher watching value.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) video screen anti-shake, and in particular to an anti-shake method for unmanned aerial vehicle (UAV) image transmission video. Background Art

[0002] Drones, or unmanned aerial vehicles (UAVs), are autonomous flying devices used for real-time image and video acquisition, remote sensing monitoring, transportation and other applications. With the development of technology, drones are increasingly used in aerial photography, agricultural monitoring, environmental monitoring and other fields. However, drones are affected by factors such as wind and vibration during flight, which can easily lead to unstable video images, thus affecting their application effects.

[0003] However, the existing technology still has major deficiencies, such as:

[0004] In the existing technology: During drone shooting, due to external environmental factors such as wind speed, flight path changes and internal factors such as drone vibration, different types of jitters often occur in the video, which will affect the clarity of the video and the viewing experience. Traditional anti-shake technology mostly relies on single frames or simple motion estimation, and it is difficult to accurately judge the type and degree of jitter. After jitter detection, how to efficiently generate clear jitter-free video frames is another technical difficulty. Traditional anti-shake technology often relies on simple interpolation or cutting processing, which cannot effectively restore image details, resulting in a decrease in the final video quality. Summary of the invention

[0005] The purpose of the present invention is to provide an anti-shake method for unmanned aerial vehicle image transmission video to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for anti-shake of drone image transmission video, comprising the following steps:

[0007] Step 1: Data acquisition;

[0008] Step 2: Multi-frame optical flow analysis;

[0009] Step 3: Feature extraction and jitter determination;

[0010] Step 4: Generate jitter-free frames using a generative adversarial network;

[0011] Step 5: Post-processing and synthesis;

[0012] Step 6: Real-time feedback and optimization.

[0013] Preferably, the step 1: data acquisition specifically includes:

[0014] 1.1: Video acquisition: Configure the drone camera to record video at a high frame rate (at least 30fps) to ensure clear image quality;

[0015] 1.2: Sensor synchronization: Use the drone’s gyroscope and accelerometer to simultaneously record position, angular velocity, and acceleration data, ensuring that timestamps are synchronized with video frames;

[0016] 1.3: Data storage: Store video and sensor data to a local SD card or cloud for subsequent processing steps.

[0017] Preferably, the step 2: multi-frame optical flow analysis specifically comprises the following steps:

[0018] 2.1: Frame extraction: extract the current frame, previous frame and previous two frames from the recorded video stream to form a sliding window;

[0019] 2.2: Optical flow calculation: Compare the current frame with the previous frame to generate an optical flow vector, and perform similar calculations on the current frame and the previous two frames to obtain an optical flow vector;

[0020] 2.3: Optical flow weighted average: The weighted average of the calculated optical flow vectors is used to reduce noise interference and obtain the final optical flow vector.

[0021] Preferably, the step 3: feature extraction and jitter determination specifically includes:

[0022] 3.1: Feature extraction: Use the pre-trained convolutional neural network to extract features from the current frame and obtain a feature vector;

[0023] 3.2: Feature fusion: Combine the extracted feature vector with the previously calculated optical flow vector, gyroscope and accelerometer data to form a comprehensive feature vector;

[0024] 3.3: Jitter judgment: Input the comprehensive feature vector into the trained classifier to judge the jitter type.

[0025] Preferably, the step 4: generating a jitter-free frame by a generative adversarial network specifically includes:

[0026] 4.1: GAN model training: Collect a large number of jittery video frames and corresponding jitter-free frames, build a training set, and train the GAN model so that it can generate jitter-free frames from jittery frames;

[0027] 4.2: Real-time frame input: The current frame judged to have jitter in step 3 is sent to the GAN model for processing;

[0028] 4.3: Shake-free frame generation: Generate clear shake-free images through the GAN model as input for subsequent steps.

[0029] Preferably, the step 5: post-processing and synthesis specifically includes:

[0030] 5.1: Color correction: Perform color analysis on the generated image and use the color mapping algorithm to adjust the color of the generated image to keep it consistent with the original video;

[0031] 5.2: Sharpening: Use image sharpening algorithms to improve image clarity and enhance details;

[0032] 5.3: Time smoothing: The continuously generated jitter-free frames are smoothed using a cubic spline interpolation algorithm to reduce flickering;

[0033] 5.4: Video synthesis: Synthesize the processed jitter-free frames into a complete video file in chronological order.

[0034] Preferably, the step 6: real-time feedback and optimization specifically includes:

[0035] 6.1: Collect feedback: Collect feedback on the generated videos during user use and record effect evaluation information;

[0036] 6.2: Model retraining: Retrain the GAN model and classifier based on the collected feedback data to improve its performance and adaptability;

[0037] 6.3: Algorithm Update: Regularly review and update algorithms to ensure they remain consistent with the latest technological developments and user needs.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. This technical solution effectively solves the problem of high-precision detection and classification of drone video jitter. By combining multi-frame optical flow analysis and deep learning feature extraction, it can accurately identify different types of jitter. This precise jitter judgment not only improves the reliability of detection, but also provides an important basis for subsequent processing, so that in practical applications, whether in complex environments or fast-moving situations, video quality can be effectively guaranteed;

[0040] 2. By using a generative adversarial network (GAN) to generate jitter-free frames in real time, the solution can significantly improve video quality while maintaining details and clarity. Subsequent color correction and sharpening processing further enhances the visual effect of the image, making the final output video not only smooth but also more watchable. This technological advancement enables drones to achieve professional-level image quality when shooting videos; BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 For the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] See also Figure 1 , the present invention provides a technical solution:

[0044] 1. A method for stabilizing a video transmitted by a drone, comprising the following steps:

[0045] Step 1: Data acquisition;

[0046] Step 2: Multi-frame optical flow analysis;

[0047] Step 3: Feature extraction and jitter judgment;

[0048] Step 4: Generate jitter-free frames using a generative adversarial network;

[0049] Step 5: Post-processing and synthesis;

[0050] Step 6: Real-time feedback and optimization.

[0051] Step 1: Data acquisition, including:

[0052] 1.1: Video acquisition: Configure the drone camera to record video at a high frame rate (at least 30fps) to ensure clear image quality;

[0053] 1.2: Sensor synchronization: Use the drone’s gyroscope and accelerometer to simultaneously record position, angular velocity, and acceleration data, ensuring that timestamps are synchronized with video frames;

[0054] 1.3: Data storage: Store video and sensor data to a local SD card or cloud for subsequent processing steps.

[0055] First, the video acquisition link needs to ensure that the drone camera is configured with a high frame rate (at least 30fps) to record high-quality video streams. By adjusting the camera's resolution and frame rate settings, ensure that the image quality remains clear under different lighting conditions. In addition, it is recommended to use wide dynamic range (WDR) technology to deal with the contrast problem of strong light and shadows, thereby improving the overall image quality. Secondly, in terms of sensor synchronization, the dynamic state of the drone, including position, angular velocity and acceleration, needs to be recorded in real time. To this end, the gyroscope and accelerometer should be synchronized with the video frame through a high-precision timestamp to ensure that each frame of video corresponds to accurate sensor data. This can be achieved by writing a control program and using the drone's flight control system to ensure the consistency and accuracy of data acquisition. Finally, during the data storage process, the video and sensor data are written to the local SD card and cloud storage at the same time for subsequent processing. Using a high-performance SD card can increase the write speed and ensure that data is not lost. At the same time, the data is backed up to the cloud regularly for subsequent analysis and processing. This series of data acquisition steps lays a solid foundation for subsequent video processing and analysis, ensuring the accuracy and integrity of the data, thereby effectively supporting subsequent key links such as jitter detection, image generation and quality improvement. With high-quality data input, the final anti-shake effect will be more significant, improving the overall quality and user experience of drone video.

[0056] Step 2: Multi-frame optical flow analysis, including the following steps:

[0057] 2.1: Frame extraction: extract the current frame, previous frame and previous two frames from the recorded video stream to form a sliding window;

[0058] 2.2: Optical flow calculation: Compare the current frame with the previous frame to generate an optical flow vector, and perform similar calculations on the current frame and the previous two frames to obtain an optical flow vector;

[0059] 2.3: Optical flow weighted average: The weighted average of the calculated optical flow vectors is used to reduce noise interference and obtain the final optical flow vector.

[0060] First, frame extraction is performed. The current frame, the previous frame, and the previous two frames are selected from the recorded video stream to form a sliding window. This process can be implemented through a video processing tool library (such as OpenCV) to ensure the efficiency and accuracy of the operation. The extracted frames need to undergo certain preprocessing, such as denoising and resizing, so that the subsequent optical flow calculation is more stable. Then, the optical flow calculation is performed, the current frame is compared with the previous frame, and the optical flow vector is generated using an optical flow algorithm (such as the Lucas-Kanade optical flow method). At the same time, the current frame and the previous two frames are similarly calculated to obtain the corresponding optical flow vector. This process uses the principle of image brightness invariance to estimate the motion of the image content by tracking the motion of feature points, providing important information for jitter detection. Finally, the optical flow weighted average is performed. By performing weighted calculations on multiple optical flow vectors, noise interference is effectively reduced to obtain the final optical flow vector. This process can ensure the accuracy and reliability of the calculation results by setting appropriate weight coefficients and emphasizing more stable frame data.

[0061] This step provides key motion information for subsequent jitter detection and classification, ensuring that the type and degree of jitter in the video can be accurately identified. Through accurate optical flow analysis, we can effectively improve our understanding of jitter problems, thus providing a solid foundation for subsequent image processing and quality improvement.

[0062] Step 3: Feature extraction and jitter determination, including:

[0063] 3.1: Feature extraction: Use the pre-trained convolutional neural network to extract features from the current frame and obtain a feature vector;

[0064] 3.2: Feature fusion: Combine the extracted feature vector with the previously calculated optical flow vector, gyroscope and accelerometer data to form a comprehensive feature vector;

[0065] 3.3: Jitter judgment: Input the comprehensive feature vector into the trained classifier to judge the jitter type.

[0066] In the specific implementation scheme of feature extraction and jitter judgment, feature extraction is first performed, and the current frame is processed using a pre-trained convolutional neural network (CNN). This process can be done by using a deep learning framework (such as TensorFlow or PyTorch) to load a pre-trained model, input the current frame into the network, and obtain a high-dimensional feature vector. This feature vector can effectively capture key information in the image, such as texture, edges, and shapes, and provide rich visual features for subsequent analysis;

[0067] Next, feature fusion is performed to combine the extracted feature vector with the previously calculated optical flow vector and the sensor data from the gyroscope and accelerometer. A comprehensive feature vector is formed by concatenation or weighting, so that the fused data covers the image content, motion information, and flight status. This process is crucial to improving the accuracy of jitter judgment, because the comprehensive features can more comprehensively reflect the dynamic situation in the video;

[0068] Finally, the comprehensive feature vector is input into a pre-trained classifier for jitter determination. The classifier can use a support vector machine (SVM), random forest, or deep learning model to identify the type of jitter in the video and its severity by analyzing the feature vector. Such a judgment output provides a clear direction for subsequent image processing and quality improvement measures;

[0069] This series of steps ensures that when analyzing video jitter, image features, motion data, and flight status can be comprehensively considered, thereby achieving high-accuracy jitter judgment, laying a solid foundation for subsequent image restoration and quality enhancement. Accurate judgment can effectively guide subsequent processing strategies and improve the overall quality of the video and viewing experience.

[0070] Step 4: Generate a jitter-free frame using a generative adversarial network, including:

[0071] 4.1: GAN model training: Collect a large number of jittery video frames and corresponding jitter-free frames, build a training set, and train the GAN model so that it can generate jitter-free frames from jittery frames;

[0072] 4.2: Real-time frame input: The current frame judged to have jitter in step 3 is sent to the GAN model for processing;

[0073] 4.3: Shake-free frame generation: Generate clear shake-free images through the GAN model as input for subsequent steps.

[0074] First, the GAN model is trained. A large number of video frames containing jitter and the corresponding jitter-free frames are collected to construct a training set. This process needs to ensure the diversity and representativeness of the samples so that the model can learn jitter features of different types and degrees. The GAN model is trained with the training set so that it can generate clear jitter-free frames from jittery input frames. During the training process, the generator and discriminator are continuously optimized through the adversarial mechanism to improve the generation ability and discrimination accuracy of the model. Real-time frame input is performed, and the current frame judged as having jitter in step 3 is sent to the trained GAN model for processing. This process needs to ensure that the format of the input data is consistent with that during training so that the model can correctly parse and process the data. Through efficient data stream processing, real-time frame input can be achieved to meet the real-time requirements of the application scenario. The GAN model generates jitter-free frames and outputs a clear jitter-free image through the generator as the input for subsequent steps. This generation process not only improves the visual quality of the video, but also provides a more stable foundation for subsequent image enhancement and processing;

[0075] This ensures that in the process of processing jittery videos, deep learning technology can be effectively used to generate high-quality jitter-free frames. Through the adversarial learning mechanism of GAN, the video quality is significantly improved, providing users with a better viewing experience. This generation method provides strong support for further post-processing and application, and fully demonstrates the potential of modern deep learning in the field of image processing.

[0076] Step 5: Post-processing and synthesis, including:

[0077] 5.1: Color correction: Perform color analysis on the generated image and use the color mapping algorithm to adjust the color of the generated image to keep it consistent with the original video;

[0078] 5.2: Sharpening: Use image sharpening algorithms to improve image clarity and enhance details;

[0079] 5.3: Time smoothing: The continuously generated jitter-free frames are smoothed using a cubic spline interpolation algorithm to reduce flickering;

[0080] 5.4: Video synthesis: Synthesize the processed jitter-free frames into a complete video file in chronological order.

[0081] In the specific implementation scheme of post-processing and synthesis, color correction is first performed to perform color analysis on the generated jitter-free image. By using a color mapping algorithm, such as histogram equalization or gamma correction, the color of the generated image is adjusted to be consistent with the original video. This step ensures that the generated image visually matches the color style of the original video, enhancing the consistency of the user experience, and sharpening is performed, and an image sharpening algorithm is applied to improve the clarity of the image and enhance the details. This can be achieved through techniques such as Laplacian operators or high-pass filtering, the purpose of which is to make the edges in the image more obvious, thereby improving the overall visual effect and compensating for the blur that may occur during the generation process. Subsequently, time smoothing is performed, and a cubic spline interpolation algorithm is applied to the continuously generated jitter-free frames to reduce the flickering between frames. This processing helps to smooth the transition, ensure the smoothness of the video playback, and enhance the viewing experience. By optimizing the temporal association between frames, the visual discomfort can be effectively reduced. Finally, video synthesis is performed, and the processed jitter-free frames are synthesized into a complete video file in chronological order. This process can be efficiently synthesized using video editing tool libraries (such as FFmpeg) to generate the final jitter-free video output for user viewing or further analysis;

[0082] This step ensures that the generated jitter-free frames are finely post-processed, so that the final synthesized video meets high standards in color, clarity, and smoothness. Through effective post-processing and synthesis, the video quality is improved, the user experience is enhanced, and a high-quality visual output is provided for subsequent applications. This comprehensive processing method makes the application of technology more mature and provides a reliable solution for video processing in actual scenarios.

[0083] Step 6: Real-time feedback and optimization, including:

[0084] 6.1: Collect feedback: Collect feedback on the generated videos during user use and record effect evaluation information;

[0085] 6.2: Model retraining: Retrain the GAN model and classifier based on the collected feedback data to improve its performance and adaptability;

[0086] 6.3: Algorithm Update: Regularly review and update algorithms to ensure they remain consistent with the latest technological developments and user needs.

[0087] First, we collect feedback. After users use the generated videos, we systematically collect user feedback on the video quality through questionnaires, user evaluations, and usage data analysis. These feedback information includes user evaluations on the clarity, color accuracy, and smoothness of the generated videos, which helps us understand the actual experience and needs of users. Next, we retrain the model. Based on the collected feedback data, we retrain the GAN model and classifier in a targeted manner. This process not only improves the performance of the model, but also enhances its adaptability to different user needs and scenarios. Through continuous optimization and learning, the model can better cope with new jitter types and video features, thereby improving the overall quality of video generation. Finally, we update the algorithm. Regularly review and update the algorithm to ensure that it is consistent with the latest technological developments and user needs. This includes introducing new technologies, improving existing algorithms, and adjusting according to industry trends. Such an update mechanism keeps the system at the forefront of technology and can quickly respond to changing user needs. Through this series of steps, it is ensured that the system can be gradually optimized and improved in actual applications, continuously improving the quality of video generation and user satisfaction. The real-time feedback and optimization mechanism not only enhances the adaptability of the model, but also provides a basis for continuous improvement, forming a virtuous circle and further promoting the advancement of technology and the deepening of its application. This flexible optimization strategy enables the system to remain competitive in a constantly changing environment, thereby creating a better user experience for users.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for stabilizing video transmitted by a drone, characterized in that: The steps include: Step 1: Data acquisition; Step 2: Multi-frame optical flow analysis; Step 3: Feature extraction and jitter determination; Step 4: Generate jitter-free frames using a generative adversarial network; Step 5: Post-processing and synthesis; Step 6: Real-time feedback and optimization.

2. The anti-shake method for drone image transmission video according to claim 1, characterized in that: The step 1: data acquisition, specifically includes: 1.1: Video acquisition: Configure the drone camera to record video at a high frame rate to ensure clear image quality; 1.2: Sensor synchronization: Use the drone’s gyroscope and accelerometer to simultaneously record position, angular velocity, and acceleration data, ensuring that timestamps are synchronized with video frames; 1.3: Data storage: Store video and sensor data to a local SD card or cloud for subsequent processing steps.

3. The anti-shake method for drone image transmission video according to claim 2, characterized in that: The step 2: multi-frame optical flow analysis specifically includes the following steps: 2.1: Frame extraction: extract the current frame, previous frame and previous two frames from the recorded video stream to form a sliding window; 2.2: Optical flow calculation: Compare the current frame with the previous frame to generate an optical flow vector, and perform similar calculations on the current frame and the previous two frames to obtain an optical flow vector; 2.3: Optical flow weighted average: The weighted average of the calculated optical flow vectors is used to reduce noise interference and obtain the final optical flow vector.

4. The anti-shake method for drone image transmission video according to claim 3, characterized in that: The step 3: feature extraction and jitter determination specifically includes: 3.1: Feature extraction: Use the pre-trained convolutional neural network to extract features from the current frame and obtain a feature vector; 3.2: Feature fusion: Combine the extracted feature vector with the previously calculated optical flow vector, gyroscope and accelerometer data to form a comprehensive feature vector; 3.3: Jitter judgment: Input the comprehensive feature vector into the trained classifier to judge the jitter type.

5. The anti-shake method for drone image transmission video according to claim 4, characterized in that: The step 4: generating a jitter-free frame using a generative adversarial network specifically includes: 4.1: GAN model training: Collect a large number of jittery video frames and corresponding jitter-free frames, build a training set, and train the GAN model so that it can generate jitter-free frames from jittery frames; 4.2: Real-time frame input: The current frame judged to have jitter in step 3 is sent to the GAN model for processing; 4.3: Shake-free frame generation: Generate clear shake-free images through the GAN model as input for subsequent steps.

6. The anti-shake method for drone image transmission video according to claim 1, characterized in that: The step 5: post-processing and synthesis, specifically includes: 5.1: Color correction: Perform color analysis on the generated image and use the color mapping algorithm to adjust the color of the generated image to keep it consistent with the original video; 5.2: Sharpening: Use image sharpening algorithms to improve image clarity and enhance details; 5.3: Time smoothing: The continuously generated jitter-free frames are smoothed using a cubic spline interpolation algorithm to reduce flickering; 5.4: Video synthesis: Synthesize the processed jitter-free frames into a complete video file in chronological order.

7. The anti-shake method for drone image transmission video according to claim 1, characterized in that: The step 6: real-time feedback and optimization, specifically includes: 6.1: Collect feedback: Collect feedback on the generated videos during user use and record effect evaluation information; 6.2: Model retraining: Retrain the GAN model and classifier based on the collected feedback data to improve its performance and adaptability; 6.3: Algorithm Update: Regularly review and update algorithms to ensure they remain consistent with the latest technological developments and user needs.

Images