Method and system for detecting blurred frames and restoring blurred images of UAV original video

Abstract

The present invention discloses a method and system for detecting blurred frames and restoring blurred images of original videos of unmanned aerial vehicles, aiming to solve the imaging blur problem that occurs when unmanned aerial vehicles collect images in a high dynamic environment. The present invention improves the video frame rate through video frame interpolation technology, generates blurred images based on camera response function simulation, constructs a scoring mechanism integrating multiple algorithms to evaluate and screen blurred images, constructs and trains three deep learning networks for online blurred image classification, online blurred image restoration, and offline blurred image restoration to meet the image processing requirements in different scenarios, and deploys the algorithm model to the edge computing platform to realize instant and efficient processing of unmanned aerial vehicle video data. The present invention significantly improves the quality of unmanned aerial vehicle video data, provides excellent preprocessing means for the collection and in-depth analysis of video data, and has important practical value and broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the field of image data processing and artificial intelligence, and relates to image processing technology based on deep learning technology, and in particular to a method and system for detecting blurred frames and restoring blurred images of original videos of unmanned aerial vehicles. Background Art

[0002] In recent years, with the rapid development of drone technology and image processing technology, drones have been widely used in many fields such as security monitoring, environmental monitoring, disaster response, etc. Because drones can provide unique perspectives and their excellent maneuverability, they have become an important means of obtaining real-time information.

[0003] However, when performing complex tasks, drones often need to operate in highly dynamic environments and are easily affected by factors such as strong winds, body vibrations, and high-speed movement, which can cause blurred video data and seriously affect image quality and the effectiveness of subsequent analysis.

[0004] At present, the blur processing of drone videos mainly relies on post-processing technology, such as image sharpening and deconvolution algorithms. Although there have been many attempts, the effect is still limited when processing blurred images in high-dynamic scenes. Although the image deblurring technology based on the attention mechanism has improved to a certain extent, it is difficult to ensure processing efficiency while ensuring processing quality. It usually takes a long time to process and is not suitable for real-time scene applications.

[0005] In addition, most existing deblurring technologies need to be performed offline, that is, data is transmitted to the ground station before processing, which not only delays information acquisition but also increases the data processing burden. For scenarios that require rapid response (such as emergency rescue, safety monitoring, etc.), such offline processing methods often cannot meet actual application needs.

[0006] Therefore, there is an urgent need to develop an efficient and real-time deblurring technology, especially an instant video image processing solution suitable for drones in complex flight conditions. This technology should be able to effectively cope with the challenges of high-dynamic environments, significantly improve the quality of drone video data, ensure that clear and reliable image data can be provided in real time, and meet the high standards of diverse application scenarios. Summary of the invention

[0007] In view of the above problems, the present invention provides a method and system for detecting blur frames and restoring blur images of UAV raw videos. The scheme mainly utilizes an online blur image classification network, an online blur image restoration network and an offline blur image restoration network to perform efficient preprocessing on the captured raw video on the UAV airborne platform.

[0008] Specifically, if Figure 2As shown, the present invention first performs scaling processing on the original image collected by the drone, and then inputs it into the online fuzzy image classification network to identify and distinguish fuzzy frames and clear frames in real time. For the identified clear frames, the system directly outputs them as clear images; for the fuzzy frames, they are input into the online fuzzy image restoration network for deep restoration processing, and the processed images will also be scaled and fuzzy frame judgment. If the image is judged to be clear at this time, it will be output as the final clear image. For those frames that are still blurry after online fuzzy image restoration, the system will record their status, and after the video acquisition is completed, these fuzzy frames that are difficult to restore online are further deep restored through the offline fuzzy image restoration network to ensure the high definition of the final video output. Through the above method and device, the present invention ensures the real-time acquisition and processing of drone video data, and significantly improves the quality of image data. The application of this set of technologies not only provides an efficient and high-quality processing method for the acquisition of drone video data, but also provides good preprocessing support for subsequent video depth analysis and semantic interpretation tasks, thereby significantly expanding the practicality and effectiveness of drones in various practical application scenarios.

[0009] A method for detecting blurred frames of original video of a drone and restoring blurred images, the method comprising the following steps:

[0010] Step 1: Video data collection and frame rate optimization: Collect high-definition videos shot by drones, increase the frame rate of the videos through video frame interpolation technology, and generate high-frame rate videos;

[0011] Step 2: Generate blurry-clear image pairs: Generate blurry frames using high frame rate video clips, use the middle frames of the clips as clear frames, and construct blurry-clear image pairs;

[0012] Step 3: Construction and division of data sets: Build a comprehensive multi-algorithm scoring system for scoring, and divide the data sets into training sets, validation sets, and test sets;

[0013] Step 4: Model construction, training and testing: construct an online fuzzy image classification network, an online fuzzy image restoration network and an offline fuzzy image restoration network, and train and test the models respectively;

[0014] Step 5: Deployment and application of the model: Deploy the model on the edge computing platform, input the original image data, and perform fuzzy image classification and restoration processing.

[0015] As a further improvement of the above solution, the present invention provides a standardized process for drone video acquisition. In step 1, by acquiring videos of different conditions including different regions, weather conditions, light intensity, flight altitude, and imaging equipment used, it is ensured that the acquired high-definition videos can fully cover a variety of acquisition conditions. This is intended to optimize the diversity and representativeness of the data set and provide a solid foundation for subsequent image processing.

[0016] As a further improvement of the above scheme, the present invention provides a high frame rate video generation method. In step 1, the frame rate of the video is improved by video frame interpolation technology based on optical flow estimation and deep learning model, and the frame rate of the original video is increased tenfold, thereby significantly enhancing the smoothness and clarity of the video, and providing better quality input data for the detection and restoration of blurred images. The optical flow estimation adopts an improved PWC-Net network, and a residual learning module is added to the feature extraction stage of each layer of the pyramid layer, and a channel attention mechanism is added to each upsampling stage and downsampling stage; the depth estimation adopts an improved Hourglass network, and a channel attention mechanism is added to the feature extraction stage of each downsampling layer, and the upsampling method is modified to use transposed convolution and bilinear interpolation at the same time.

[0017] As a further improvement of the above scheme, the present invention provides a method for generating blurry-clear image pairs. In step 2, by analyzing the video clips after the frame rate is increased and using the simulated camera response function to process five consecutive frames of images and the interpolated intermediate frames, the technical solution can effectively generate a high-quality blurry image and its corresponding clear image (i.e., the selected intermediate frame is used as a clear reference). This method provides accurate data support for the training and verification of subsequent image restoration technology.

[0018] As a further improvement of the above scheme, the present invention provides an automatic image blur discrimination mechanism. In step three, by constructing a comprehensive multi-algorithm scoring system, the image blur degree is accurately evaluated using indicators such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), variance of Laplace operator, and variance of Fourier transform. This mechanism quantifies the blur degree of the image and can more accurately identify whether the image is blurred, significantly reducing the workload of manual annotation, and can provide a data basis for continuous learning of the model.

[0019] As a further improvement of the above scheme, the present invention provides an online fuzzy image classification network, the network design focuses on efficient computing speed and low resource occupation. In step 4, Figure 3As shown in the figure, the core of the network is the use of an efficient and lightweight image classification convolutional block architecture. During the training phase, the convolutional block first performs 3×3 convolution and batch normalization, 1×1 convolution and batch normalization, and a separate batch normalization operation on the input feature map, then merges the results of these operations and processes them through activation functions to enhance the nonlinear expression ability. It is worth noting that in order to further improve the computational efficiency of the network, the design only retains the weights obtained by the 3×3 convolution during the training phase during the retraining and inference phase of the model, and uses this to re-parameterize the model. This method significantly reduces the computational load of model inference. For the constructed online fuzzy image classification network, the input image will be processed through a sequence of 5 groups of convolutional layers consisting of two image classification lightweight convolutional blocks. In this sequence, the step size of the first group of convolutions is set to 1, while in the subsequent groups, the step size of the first convolution of each group is 2. This step is designed to effectively extract and reduce the spatial dimension of the feature map to reduce the complexity of subsequent processing. After the last set of convolutions, the network flattens the feature map and processes it through a three-layer fully connected network, ultimately outputting a binary classification result — that is, determining whether the image is blurry or clear.

[0020] As a further improvement of the above solution, the present invention provides an online blurred image restoration network, which is specially designed to achieve low-latency, quasi-real-time image restoration processing while ensuring the image restoration effect. Specifically, in step 4, Figure 4 As shown in Figure 1, this network adopts an encoder-decoder architecture and is optimized for online image restoration tasks. The key component of this network is a lightweight convolution block designed specifically for image restoration. Unlike lightweight convolution blocks used for image classification, the lightweight convolution block for image restoration deliberately removes the batch normalization operation to meet the specific needs of image restoration. During the training phase, the convolution block simultaneously performs 3×3 convolution, 1×1 convolution, and skip connection operations on the input feature map, then merges the results of these operations, and then processes them through activation functions to increase nonlinear expression capabilities. To optimize computational efficiency, during the retraining and inference phases of the model, the convolution block only retains the 3×3 convolution weights obtained during training, and uses these weights for re-parameterized training, which significantly reduces the computational burden during inference applications. The operation process of the online blurred image restoration network starts with three consecutive convolutions, followed by four downsampling and convolution operations and four upsampling and convolution operations, each of which contains three lightweight convolution blocks. Each downsampling operation is accompanied by a jump connection, which merges the feature maps that match the size of the upsampling stage to maintain the integrity of the information. Three, two, and one additional convolution blocks are performed at the first three jump connections to further refine the feature information. Before the output of the model, the feature map is processed by the last two convolution blocks to obtain the final restored image.

[0021] As a further improvement of the above solution, the present invention provides an offline blurred image restoration network, which is specially designed to process extremely blurred images to achieve a better restoration effect. Although this network takes a long time in the inference process, it has better quality in image quality restoration. Specifically, in step 4, if Figure 5 As shown in the figure, the network is based on the architecture of the online blurred image restoration network. The offline blurred image restoration network is built by adding local-global-local (LGL) attention mechanism modules after each downsampling, after each upsampling, at the jump connection, and at the beginning and end of the network. The local-global-local attention mechanism module consists of the following four submodules: patch embedding, feedforward network, local attention, and global attention. This comprehensive module uses local attention to enhance the capture of image edge features in the downsampling stage of the network, uses global attention to optimize the ability to restore edge details in the upsampling stage, helps to further highlight the criticality of edge information when merging codec features at the jump connection of the network, and performs preprocessing and postprocessing at the beginning and end of the network to enhance the edge information of the original image and improve edge clarity in the final output.

[0022] The present invention provides a UAV original video fuzzy frame detection and fuzzy image restoration system, comprising:

[0023] Processor for performing video data acquisition, frame rate optimization, blur and clear image pair generation, data set construction and partitioning, model construction, model training, model testing, model deployment and application;

[0024] A memory for storing programs, including but not limited to a video frame interpolation algorithm, a blurry and clear image pair generation algorithm, a multi-algorithm scoring system, an online blurry image classification network, an online blurry image restoration network, and an offline blurry image restoration network construction and training algorithm;

[0025] When the program is executed by the processor, the above-mentioned method for detecting blurred frames and restoring blurred images of original drone videos can be implemented.

[0026] The present invention also provides a storage medium storing instructions executable by a processor, and when the processor executes the instructions executable by the processor, the method for detecting blurred frames and restoring blurred images of original videos of unmanned aerial vehicles is executed.

[0027] The beneficial effects of the present invention are as follows:

[0028] 1. The present invention can immediately identify and process fuzzy frames during the data acquisition phase, effectively reducing the amount of data required to be stored and transmitted. This is particularly critical for application scenarios such as real-time monitoring and rapid response, as it not only reduces the reliance on network bandwidth, but also significantly reduces data storage costs.

[0029] 2. The present invention can identify and distinguish blurry frames and clear frames in video data in real time through an online fuzzy image classification network. This process not only effectively reduces the demand for storage resources, but also speeds up the entire video processing process, so that clear images can be quickly output, greatly enhancing the real-time performance and overall efficiency of data processing.

[0030] 3. The present invention combines online and offline blurred image restoration networks, greatly improving the effect and efficiency of blurred image restoration processing. The online restoration network quickly processes slightly blurred images, while the offline restoration network is responsible for those heavily blurred images that are difficult for the online network to process, ensuring that all blurred images can be effectively restored, thereby ensuring the clarity and quality of the output video.

[0031] 4. The present invention, with its efficient and high-quality video preprocessing capabilities, greatly expands the applicability of drones in a variety of practical application scenarios, especially under conditions where high-quality video data is required. Whether it is a video acquisition task in the fields of environmental monitoring, disaster assessment, emergency rescue, etc., or as a data source for analysis tasks such as machine learning training and image recognition, the high-quality input data provided by the present invention can significantly improve the accuracy and reliability of the final processing results, and enhance the value of drones as an efficient data acquisition tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a basic flow chart of the method for detecting blurred frames and restoring blurred images of original video of a UAV according to the present invention.

[0033] Figure 2 The present invention is a flowchart of the reasoning application of the method for detecting blurred frames and restoring blurred images of the original video of the unmanned aerial vehicle.

[0034] Figure 3 Schematic diagram of online fuzzy image classification network for the fuzzy frame detection and fuzzy image restoration processing method of the UAV original video of the present invention.

[0035] Figure 4 The figure is a schematic diagram of an online blurred image restoration network of the blurred frame detection and blurred image restoration processing method of the original video of the UAV of the present invention.

[0036] Figure 5 The figure is a schematic diagram of an offline blurred image restoration network of the blurred frame detection and blurred image restoration processing method of the UAV original video of the present invention.

[0037] Figure 6 The present invention is a schematic diagram of a terminal device for detecting blurred frames and restoring blurred images of original videos of drones.

[0038] Figure 7 Schematic diagram of test results of the method for detecting blurred frames in original video of a drone according to the present invention.

[0039] Figure 8 Schematic diagram of test results of the online blurred image restoration method for the original video of the UAV of the present invention.

[0040] Fig. 9 Schematic diagram of test results of the offline blurred image restoration method of the original video of the drone of the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in combination with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 1-6 As shown, the present invention provides a method for detecting blurred frames of original video of a drone and restoring blurred images, the method comprising the following steps:

[0043] Step 1: Collect high-definition video taken by the drone, increase the frame rate of the video through video frame interpolation technology, and generate high-frame rate video.

[0044] Step 1.1, HD video acquisition: Use low-mobility drones to acquire HD videos with a resolution of 1920×1080 and a frame rate of 30 frames per second under various environmental conditions (including different regions, weather conditions, light intensity, flight altitude, and imaging equipment used). Save these videos to the local hard disk for subsequent preprocessing steps.

[0045] Step 1.2: Improve the video frame rate:

[0046] First, for two given consecutive video images I0(x) and I1(x), where x∈[1,H]×[1,W] defines the two-dimensional spatial coordinates on the image plane, and H and W are the height and width of the image, respectively. The two input frames are first preprocessed to adjust the size and color space to meet the needs of subsequent processing.

[0047] Next, optical flow and depth estimation are performed. Optical flow estimation uses the improved PWC-Net network, adds a residual learning module to the feature extraction stage of each layer of the pyramid layer, and adds a channel attention mechanism to each upsampling stage and downsampling stage to estimate the bidirectional optical flow F from two frames of input images. 0→1 and F 1→0, the optical flow vector represents the movement of each pixel in the image from I0 to I1 (and vice versa). The depth estimation uses an improved Hourglass network, adds a channel attention mechanism in each layer of downsampling feature extraction stage, and modifies the upsampling method to use transposed convolution and bilinear interpolation at the same time, respectively, to estimate the depth of I0 and I1, and generate depth maps D0 and D1. The depth map provides the relative distance information of each pixel in the image to the camera.

[0048] Third, based on the optical flow and depth information obtained in the previous step, the depth-aware flow projection technique is used to estimate the target intermediate frame I t The optical flow F of (x) t→0 and F t→1 In this process, the weight of the optical flow vector is adjusted according to the depth information to favor objects with closer sampling distance. The depth weight is calculated as follows:

[0049]

[0050] Wherein, D0 is a depth map, which provides the relative distance from the camera to each pixel in the first frame image I0. The depth value D0(y) of each pixel describes the distance of the point in the three-dimensional space relative to the observation point (camera); y is the two-dimensional spatial coordinate of a pixel in the image I0, which is used to traverse all pixels or a specific set of pixels in the image and calculate the weight and optical flow vector; ò is a very small positive number, which is 10 in the present invention. -6 , used to ensure stability. Then, combined with the optical flow vector adjusted by the depth weight, the optical flow of the intermediate frame is synthesized according to the following formula:

[0051]

[0052] Among them, F t→0 (x) is the target intermediate frame I t (x) is the optical flow from the initial frame I0; t is a temporal factor between 0 and 1, indicating the temporal position of the target intermediate frame relative to the two input frames I0 and I1; F 0→1 (y) represents the optical flow vector from I0 to I1 at position y; w0(y) is the depth weighting factor at position y, calculated based on the depth map D0; S(x) is the set of pixel contribution information for estimating the optical flow at position x.

[0053] Afterwards, the input frame, depth map and context features are spatially deformed using the calculated intermediate frame optical flow, using adaptive sampling techniques to synthesize the target intermediate frame image. In particular, the frame synthesis network comprehensively considers the deformed frame, depth information and optical flow information, and generates the final high-quality interpolated frame through the learned residual information.

[0054] Step 1.3, generation of high frame rate video: Repeat the above steps to process the entire video sequence until the entire video sequence is processed, thereby completing the overall improvement of the video frame rate. Through the technical solution of the present invention, the video frame rate can be effectively improved, and even if there are fast-moving or occluded scenes in the video, a clear and coherent high frame rate video can be generated, which significantly improves the smoothness and visual experience of video playback and provides high-quality data support for subsequent blurred image generation.

[0055] Step 2: Use high frame rate video clips to generate blurry frames, use the middle frames of the clips as clear frames, and construct blurry-clear image pairs.

[0056] This step focuses on using the video clips with increased frame rates to generate blurry frames, and uses the middle frames in the clips as the corresponding clear frames to construct blurry-clear image pairs. This process focuses on simulating the physical phenomenon of blurry images in real environments due to changes in light received by the camera sensor during exposure, and is closer to the real physical principle of blur generation.

[0057] Specifically, the motion blurred image is generated because the camera sensor receives light during the exposure process, and each time it accumulates sharp image stimuli, which causes blur. Therefore, in the process of generating blurred images, the present invention adopts a simulation process of the camera response function (CRF). CRF is a nonlinear function used to convert the integral signal during the exposure process into the final pixel value. By simulating the accumulation of signals in video frames shot continuously at high speed, an image close to the real blur effect can be generated. The accumulation process of blurred images can be described by the following model:

[0058]

[0059] Where T and S(t) represent the exposure time and sensor signal of the image at time t, respectively. Similarly, M and S[i] are the number of sampling frames during the exposure time and the captured i-th frame signal, respectively. g is the mapping of the signal S(t) to the observed image The CRF is such that or Through this image superposition method, a large number of images shot by drones in high-maneuverability scenes are generated.

[0060] In practical applications, by selecting the original clear frame I n To I n+4 and its inter-frame interpolation image to generate the blurred frame, and the original clear frame I n+2 The clear image corresponding to the blurred image is used as the clear image, thereby completing the construction process of the blurred-clear image pair.

[0061] Step 3: Build a comprehensive multi-algorithm scoring system for scoring, and divide the set into training set, validation set and test set.

[0062] Specifically, a core link of the present invention is to use a comprehensive multi-algorithm scoring system to accurately classify the generated drone high-mobility scene images. Considering that in the actual flight process, the drone is not always in a high-speed motion state, it is sometimes in a relatively static state or a low-speed motion state, so not all frames are blurred. To this end, the present invention proposes a set of fuzzy frame classification methods to ensure the accuracy and availability of data.

[0063] The present invention uses two performance indicators commonly used in the field of blurred image restoration, namely Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), to evaluate image quality, and uses the variance of Laplace values ​​and the variance of Fourier spectrum, which are commonly used indicators for judging image blur, to calculate the blur of the image.

[0064] PSNR stands for Peak Signal-to-Noise Ratio, which is a measure of image quality. It evaluates the degree of image distortion by comparing the signal-to-noise ratio between the original image and the compressed or processed image. The higher the PSNR value, the smaller the image quality loss. Its calculation formula is as follows:

[0065]

[0066] Wherein, MAX represents the maximum possible value of an image pixel (eg, for an 8-bit image, MAX=255), and MSE represents Mean Squared Error.

[0067] SSIM stands for structural similarity index, which is an indicator used to measure image similarity. It takes into account the three dimensions of brightness, contrast and structure. It is more representative of the image quality perceived by the human eye. A high SSIM value means that the image quality is close to the original image. Its calculation formula is as follows:

[0068]

[0069] Among them, μ x and μ y Represent the mean of image x and y respectively, and Represents the variance of images x and y, σ xy represents the covariance of images x and y, and C1 and C2 are constants used for stability.

[0070] The variance of the Laplace operator of an image can represent the distribution of image details, thereby indirectly reflecting the clarity and detail richness of the image. The larger the variance of the Laplace value, the richer the image details and the higher the clarity. The calculation formula for the variance of the Laplace value is as follows:

[0071]

[0072] Among them, f(i) represents the pixel value in the image, μ represents the average grayscale value of the image, and N represents the total number of pixels in the image.

[0073] The variance of the Fourier spectrum is an indicator used to measure the spectral characteristics of an image. The variance of the Fourier spectrum of an image can reflect the spectral distribution of the image, thereby indirectly reflecting the blurriness of the image. The larger the variance of the Fourier spectrum, the wider the spectral distribution of the image and the clearer the image. The calculation formula of the variance of the Fourier spectrum is as follows:

[0074]

[0075] Among them, F(u,v) represents the two-dimensional Fourier transform of the image, μ represents the average value of the Fourier spectrum, and M and N represent the width and height of the image respectively.

[0076] The present invention combines the above indicators and sets a series of threshold judgment criteria to classify the clarity of images. Specifically, if an image meets at least one of the set "strong conditions" and at least two "weak conditions", it is considered a clear image; otherwise, it is judged as a blurred image. The specific thresholds of these conditions are based on extensive experimental data and analysis, ensuring the accuracy and reliability of the classification process.

[0077] Finally, through the method of the present invention, the processed images are divided into a training set, a validation set, and a test set in a ratio of 6:2:2, providing high-quality data support for fuzzy classification and restoration.

[0078] Step 4: Build an online blurred image classification network, an online blurred image restoration network, and an offline blurred image restoration network, and train and test the models respectively.

[0079] Specifically, the present invention provides an online fuzzy image classification network, the network design focuses on efficient computing speed and low resource occupation. In step 4, Figure 3As shown in the figure, the core of the network is the use of an efficient and lightweight image classification convolutional block architecture. During the training phase, the convolutional block first performs 3×3 convolution and batch normalization, 1×1 convolution and batch normalization, and a separate batch normalization operation on the input feature map, then merges the results of these operations and processes them through activation functions to enhance the nonlinear expression ability. It is worth noting that in order to further improve the computational efficiency of the network, the design only retains the weights obtained by the 3×3 convolution during the training phase during the retraining and inference phase of the model, and uses this to re-parameterize the model. This method significantly reduces the computational load of model inference. For the constructed online fuzzy image classification network, the input image will be processed through a sequence of 5 groups of convolutional layers consisting of two image classification lightweight convolutional blocks. In this sequence, the step size of the first group of convolutions is set to 1, while in the subsequent groups, the step size of the first convolution of each group is 2. This step is designed to effectively extract and reduce the spatial dimension of the feature map to reduce the complexity of subsequent processing. After the last set of convolutions, the network flattens the feature map and processes it through a three-layer fully connected network, ultimately outputting a binary classification result — that is, determining whether the image is blurry or clear.

[0080] Specifically, the present invention provides an online fuzzy image restoration network, which is specially designed to achieve low-latency, quasi-real-time image restoration processing while ensuring the image restoration effect. Specifically, in step 4, if Figure 4As shown, specifically, the present invention provides an online blurred image restoration network, which is specially designed to achieve low-latency, quasi-real-time image restoration processing while ensuring the image restoration effect. The network adopts an encoder-decoder architecture and is optimized for online image restoration tasks. The key component of the network is a lightweight convolution block designed specifically for image restoration. Unlike the lightweight convolution block used for image classification, the lightweight convolution block for image restoration deliberately removes the batch normalization operation to meet the specific needs of image restoration. During the training phase, the convolution block simultaneously performs 3×3 convolution, 1×1 convolution and jump connection operations on the input feature map, and then merges the results of these operations, and then processes them through activation functions to increase nonlinear expression capabilities. In order to optimize computational efficiency, the convolution block only retains the 3×3 convolution weights obtained during training during the retraining and inference phases of the model, and uses these weights for re-parameterized training, which significantly reduces the computational burden during inference applications. The operation process of the online blurred image restoration network starts with three consecutive convolutions, followed by four downsampling and convolution operations and four upsampling and convolution operations. Each convolution operation contains three lightweight convolution blocks. Each downsampling operation is accompanied by a jump connection. The jump connection merges the feature maps that match the size of the upsampling stage to maintain the integrity of the information. The first three jump connections will be processed by three, two, and one additional convolution blocks respectively to further refine the feature information. Before the output of the model, the feature map is processed by the last two convolution blocks to obtain the final restored image.

[0081] Specifically, the present invention provides an offline blurred image restoration network, which is specially designed to process extremely blurred images to achieve a better restoration effect. Although this network takes a long time in the inference process, it has excellent performance in image quality restoration. Specifically, in step 4, if Figure 5 As shown in the figure, the network is based on the architecture of the online blurred image restoration network. The offline blurred image restoration network is built by adding local-global-local (LGL) attention mechanism modules after each downsampling, after each upsampling, at the jump connection, and at the beginning and end of the network. The local-global-local attention mechanism module consists of the following four submodules: patch embedding, feedforward network, local attention, and global attention. This comprehensive module uses local attention to enhance the capture of image edge features in the downsampling stage of the network, uses global attention to optimize the ability to restore edge details in the upsampling stage, helps to further highlight the criticality of edge information when merging codec features at the jump connection of the network, and performs preprocessing and postprocessing at the beginning and end of the network to enhance the edge information of the original image and improve edge clarity in the final output.

[0082] Specifically, in order to ensure that the network model proposed in the present invention achieves optimal performance, a series of refined training and testing strategies are adopted. These strategies aim to comprehensively optimize model parameters and improve the accuracy and efficiency of the model for classification and restoration of blurred images. During the training process, all input data first undergoes a preprocessing step, including standardization and data enhancement (such as rotation, scaling, cropping, etc.) to improve the robustness and generalization ability of the model. At the same time, a suitable loss function is selected according to the specific tasks of different networks. For the classification network, the cross entropy loss function is used; while the restoration network uses contrastive learning loss and optimization function to accurately measure the difference between the restored image and the original image. In addition, to prevent overfitting, the L2 regularization strategy is introduced, and the early stopping technique is applied to the validation set. When the model performance has not been significantly improved for multiple consecutive training cycles, the training is terminated in advance.

[0083] It should be further explained that the restoration network is trained using contrastive learning loss and optimization function. Contrastive learning loss is to enhance the performance of the model by making the deblurred image generated by the model close to its clear real image and making the deblurred image far from its blurred input image. Its calculation formula is as follows:

[0084]

[0085] Among them, S represents the clear real image, R represents the deblurred image, X represents the blurred input image, ψ represents the hidden features extracted from the pre-trained VGG-19 network, and ||·||1 represents the L1 norm.

[0086] The optimization function combines Charbonnier loss, edge loss, and the above-mentioned contrastive learning loss, and its calculation formula is as follows:

[0087] L=L char +λ1L edge +λ2L con

[0088] Among them, L char and L edge are Charbonnier loss and edge loss respectively, and λ1 and λ2 are weight coefficients that adjust the contribution of these loss terms.

[0089] Specifically, during the testing process, after each stage of model training, the model is evaluated on an independent test set that has not participated in the training to ensure that the model has good generalization ability. The blurry image classification network is evaluated using multiple performance indicators such as accuracy, recall, and F1 score; at the same time, the blurry image restoration network uses PSNR and SSIM indicators to quantitatively analyze the image quality.

[0090] Step 5: Deploy the model on the edge computing platform, input the original image data, and perform fuzzy image classification and restoration.

[0091] Specifically, in the present invention, an edge computing platform with high-performance computing unit support is used to meet the computing needs of deep learning models. The platform is configured with storage, memory, and processor as the hardware foundation, equipped with an operating system and an inference accelerator designed specifically for edge computing.

[0092] Specifically, the model deployment process includes transferring the optimized model from the cloud server to the edge computing device, using a secure transmission protocol to ensure data security during the transmission process. On the edge device, the model is encapsulated and deployed through an environment container to ensure the consistency and repeatability of the deployment process. At the same time, data encryption and access control measures are taken during the deployment process to enhance data security and privacy protection.

[0093] The model of this embodiment is deployed on the drone video acquisition system to process blurred images in the video stream in real time. Through the application of the model, the accuracy and efficiency of video analysis are improved, especially in high-mobility scenes or bad weather conditions, effectively improving the performance of target detection and recognition. In addition, the need to transmit large amounts of data to the cloud is reduced, reducing bandwidth usage and data processing delays.

[0094] like Figure 6 , which is a schematic diagram of a terminal device for detecting blurred frames and restoring blurred images of original videos of drones according to the present invention.

[0095] like Figure 7 FIG. 1 is a schematic diagram showing the test results of the method for detecting blurred frames of original video of a UAV according to the present invention.

[0096] like Figure 8 , which is a schematic diagram of the test results of the online blurred image restoration method of the original video of the drone of the present invention.

[0097] like Fig. 9 , which is a schematic diagram of the test results of the offline blurred image restoration method of the original video of the drone of the present invention.

[0098] The present invention provides a UAV original video fuzzy frame detection and fuzzy image restoration system, comprising:

[0099] Processor for performing video data acquisition, frame rate optimization, blur and clear image pair generation, data set construction and partitioning, model construction, model training, model testing, model deployment and application;

[0100] A memory for storing programs, including but not limited to a video frame interpolation algorithm, a blurry and clear image pair generation algorithm, a multi-algorithm scoring system, an online blurry image classification network, an online blurry image restoration network, and an offline blurry image restoration network construction and training algorithm;

[0101] When the program is executed by the processor, the above-mentioned method for detecting blurred frames and restoring blurred images of original drone videos can be implemented.

[0102] An embodiment of the present invention further provides a storage medium storing instructions executable by a processor. When the processor executes the instructions executable by the processor, the method for detecting blurred frames and restoring blurred images of original videos of unmanned aerial vehicles is executed.

[0103] It can also be seen that the contents in the above method embodiments are all applicable to the present storage medium embodiments, and the functions and beneficial effects achieved are the same as those in the method embodiments.

[0104] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0105] The logic and / or steps represented in the embodiments or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses.

[0106] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "this embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0108] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for detecting blurred frames and restoring blurred images of drone original videos, characterized in that: The method comprises the following steps: Step 1: Video data collection and frame rate optimization: Collect high-definition videos shot by drones, increase the frame rate of the videos through video frame interpolation technology, and generate high-frame rate videos; The video frame interpolation technology includes optical flow estimation and depth estimation, and optical flow adjustment based on depth information to synthesize intermediate frame images; The optical flow estimation uses the PWC-Net network, which adds a residual learning module to the feature extraction stage of each pyramid layer and a channel attention mechanism to each upsampling and downsampling stage. The depth estimation uses the Hourglass network, which adds a channel attention mechanism to the feature extraction stage of each downsampling layer and modifies the upsampling method to use transposed convolution and bilinear interpolation at the same time. According to the optical flow and depth information obtained in the previous step, the depth-aware flow projection technique is used to estimate the optical flow of the target intermediate frame. In this process, the weight of the optical flow vector is adjusted according to the depth information to favor objects with a closer sampling distance; the depth weight is calculated as follows: Where D0 is a depth map, which provides the relative distance from the observation point to each pixel in the first frame image. The depth value D0(y) of each pixel describes the distance of the point in the three-dimensional space relative to the observation point; y is the two-dimensional spatial coordinate of a pixel in the first frame image, which is used to traverse all pixels or a specific set of pixels in the image and calculate weights and optical flow vectors; ò is a positive number used to ensure stability; Next, the optical flow vector adjusted by the depth weight is combined to synthesize the optical flow of the intermediate frame according to the following formula: Among them, F t→0 (x) is the optical flow from the target intermediate frame to the initial frame; t is a time factor between 0 and 1, indicating the temporal position of the target intermediate frame relative to the two input frames; F 0→1 (y) represents the optical flow vector from the first frame image to the second frame image at position y; w0(y) is the depth weighting factor at position y, calculated based on the depth map D0; S(x) is the set of pixel contribution information for estimating the optical flow at position x; Afterwards, the calculated intermediate frame optical flow is used to spatially deform the input frame, depth map, and context features using adaptive sampling technology to synthesize the target intermediate frame image; Repeat the above steps to process the entire video sequence until the entire video sequence is processed; Step 2: Generate blurry-clear image pairs: Generate blurry frames using high frame rate video clips, use the middle frames of the clips as clear frames, and construct blurry-clear image pairs; Step 3: Construction and division of data sets: Build a comprehensive multi-algorithm scoring system for scoring, and divide the data sets into training sets, validation sets, and test sets; Step 4: Model construction, training and testing: construct an online fuzzy image classification network, an online fuzzy image restoration network and an offline fuzzy image restoration network, and train and test the models respectively; The online blurred image restoration network is used to perform instant restoration processing on slightly blurred images. The network is characterized by adopting an encoder-decoder architecture and designing an image restoration lightweight convolution block to meet the needs of online image restoration. During the training phase, the image restoration lightweight convolution block simultaneously performs 3×3 convolution, 1×1 convolution, and skip connection operations on the input feature map, then merges the results of these operations, and then processes them through the activation function to increase the nonlinear expression capability; during the retraining and inference phases of the model, the convolution block only retains the 3×3 convolution weights obtained during training, and uses these weights for re-parameterization training; The operation process of the online blurred image restoration network starts with three consecutive convolutions, followed by four downsampling and convolution operations and four upsampling and convolution operations. Each convolution operation contains three lightweight convolution blocks. Each downsampling operation is accompanied by a jump connection, which merges the feature maps that match the size of the upsampling stage. The first three jump connections will perform three, two, and one additional convolution block processing respectively. Before the output of the model, the feature map is processed by the last two convolution blocks to obtain the final restored image. Step 5: Model deployment and application: Deploy the model on the edge computing platform, input the original image data, and perform fuzzy image classification and restoration.

2. The processing method according to claim 1, characterized in that: In step 2, the blurred-clear image pair is generated by simulating the camera response function, using the accumulation of frame signals of video clips shot continuously at high speed to generate an image close to the real blurred effect, and using the middle frame of the clip as the clear image corresponding to the blurred image.

3. The processing method according to claim 1, characterized in that: In step three, the multi-algorithm scoring system includes using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), variance of Laplace value and variance of Fourier spectrum as scoring indicators.

4. The processing method according to claim 1, characterized in that: In step four, the online blurred image classification network is used to identify and classify blurred frames and clear frames in the video in real time. The network is characterized by using lightweight convolution blocks for feature extraction and being optimized by a deep learning algorithm.

5. The processing method according to claim 1, characterized in that: In step 4, the offline blurred image restoration network is used to process severely blurred images, and the network is characterized by adding a local-global-local attention mechanism module at key nodes.

6. The processing method according to claim 5, characterized in that: The offline blurred image restoration network is based on the architecture of the online blurred image restoration network. The offline blurred image restoration network is constructed by adding local-global-local attention mechanism modules after each downsampling, after each upsampling, at the jump connection, and at the beginning and end of the network. The local-global-local attention mechanism module consists of the following four sub-modules: patch embedding, feedforward network, local attention, and global attention.

7. A UAV original video blur frame detection and blur image restoration system, characterized in that: include: Processor for performing video data acquisition, frame rate optimization, blur and clear image pair generation, data set construction and partitioning, model construction, model training, model testing, model deployment and application; A memory for storing programs, including but not limited to a video frame interpolation algorithm, a blurry and clear image pair generation algorithm, a multi-algorithm scoring system, an online blurry image classification network, an online blurry image restoration network, and an offline blurry image restoration network construction and training algorithm; When the program is executed by the processor, the method for detecting blurred frames and restoring blurred images of original videos of unmanned aerial vehicles as described in any one of claims 1 to 6 can be implemented.

Images