A method and system for real-time deblurring of a blurred video

Abstract

The application discloses a fuzzy video real-time deblurring method and system, and the method comprises the following steps: detecting feature points through multi-scale FAST detection and light-weight MLP detection and dynamic weight distribution, combining MobileNet-FPN network and light-weight MLP dynamic weight to extract robust features of feature points, using Transform to model global matching relationship of adjacent two frame feature points, and using Bayesian adaptive RANSAC algorithm to realize efficient false matching pair elimination, and finally calculating an affine transformation matrix and inversely mapping to compensate for the jitter. The method can significantly improve the image stabilization accuracy in a severe jitter scene while ensuring the real-time performance of a mobile terminal.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and video processing, and particularly relates to a fuzzy video real-time deblurring method and system. BACKGROUND

[0002] The wide application of unmanned aerial vehicles (UAVs) in environmental monitoring, disaster reconnaissance and other fields puts forward very high requirements for real-time acquisition of stable and clear video data. However, the videos collected by UAVs generally have picture shift caused by wind disturbance, rotation caused by response delay of a gimbal or tilting of a body, zooming caused by change of viewing angle due to rapid ascending and descending, and shearing caused by asymmetric motion or lens distortion. Affine transformation can effectively remove video jitter by modeling and compensating for complex motion such as translation, rotation, scaling and shearing between video frames.

[0003] In the prior art, traditional feature matching methods (such as SIFT and ORB) rely on manually designed features, have poor robustness to fuzzy and low-texture scenes, and have high error rate in matching. Depth methods based on optical flow (such as PWC-Net and RAFT) have high computational complexity, are difficult to process high-resolution videos in real time, and are sensitive to severe blurring. End-to-end deblurring networks (such as DeepStab) have large model parameter quantity, rely on a large amount of training data, and have insufficient generalization ability. Moreover, existing solutions often use a fixed-weight multi-scale feature fusion strategy (such as direct splicing or average pooling), which ignores the contribution difference of features of different scales in different scenes. SUMMARY

[0004] The present application provides a fuzzy video real-time deblurring method and system, which is used to solve the technical problem that existing solutions often use a fixed-weight multi-scale feature fusion strategy, which ignores the contribution difference of features of different scales in different scenes.

[0005] In a first aspect, the present application provides a fuzzy video real-time deblurring method, comprising:

[0006] obtaining a gray image corresponding to each original frame image in a fuzzy video, and scaling each gray image into a target gray image of a preset number of layers, and calculating the blurriness, contrast and texture complexity of each target gray image corresponding to the same gray image;

[0007] detecting the number of feature points of each target gray image based on a FAST algorithm, and determining the average response intensity of a target gray image according to the response intensity of each feature point in the same target gray image;

[0008] The blur degree, contrast, texture complexity, feature point number and average response intensity of each target gray image corresponding to the same gray image are input into a preset first multi-layer perception machine, and the first multi-layer perception machine outputs the first weight proportion of each target gray image corresponding to the same gray image;

[0009] According to the feature point extraction in each target gray image according to each first weight proportion, at least one target feature point is obtained, and the feature point coordinates of all target feature points are mapped into a two-dimensional coordinate system containing the gray image, so as to obtain a target feature point set of the gray image.

[0010] Each gray image and the target feature point set corresponding to the gray image are input into a preset MobileNet-FPN network, the MobileNet-FPN network outputs feature maps of different levels, and the blur degree, contrast, texture complexity, feature point number and average response intensity of the feature maps of different levels are input into a preset second multi-layer perception machine, the second multi-layer perception machine outputs the second weight proportion of each feature map corresponding to the same gray image, and determines the local enhanced feature of each target feature point in the target feature point set according to each second weight proportion.

[0011] The coordinates and local enhanced features of each target feature point in the gray images of the adjacent two frames are input, the coordinates are sinusoidally encoded, and then the local enhanced features are spliced to obtain an input feature matrix of a pre-trained transformer network, and the cross-frame matching relationship between the target feature points of the adjacent two frames is constructed through self-attention and cross-attention.

[0012] According to each cross-frame matching relationship, an improved adaptive RANSAC algorithm is used to remove the mismatched points, and according to the feature points after removing the mismatched points, the affine transformation parameters between the adjacent frames are calculated, and finally the video jitter compensation is completed.

[0013] In a second aspect, the present application provides a fuzzy video real-time de-jitter system, comprising:

[0014] The acquisition module is configured to acquire gray images corresponding to each original frame image in the fuzzy video, and scale each gray image into a target gray image of a preset number of layers, and calculate the blur degree, contrast and texture complexity of each target gray image corresponding to the same gray image.

[0015] The determination module is configured to detect the feature point number of the feature points in each target gray image based on the FAST algorithm, and determine the average response intensity of the target gray image according to the response intensity of each feature point in the same target gray image.

[0016] The first output module is configured to input the blur, the contrast, the texture complexity, the number of feature points and the average response intensity of each target gray image corresponding to the same gray image into a preset first multi-layer perception machine, and the first multi-layer perception machine outputs the first weight proportion of each target gray image corresponding to the same gray image.

[0017] The mapping module is configured to extract feature points in each target gray image according to the first weight proportions to obtain at least one target feature point, and map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the gray image to obtain a target feature point set of the gray image.

[0018] The second output module is configured to input each gray image and the target feature point set corresponding to the gray image into a preset MobileNet-FPN network, the MobileNet-FPN network outputs feature maps of different levels, and inputs the blur, the contrast, the texture complexity, the number of feature points and the average response intensity of the feature maps of different levels into a preset second multi-layer perception machine, the second multi-layer perception machine outputs the second weight proportion of each feature map corresponding to the same gray image, and determines the local enhanced features of each target feature point in the target feature point set according to the second weight proportions.

[0019] The construction module is configured to input the coordinates and the local enhanced features of each target feature point in the gray images of the adjacent two frames, perform sinusoidal coding on the coordinates, and then splice the local enhanced features to obtain an input feature matrix of a pre-trained transformer network, and construct the cross-frame matching relationship between the target feature points of the adjacent two frames through self-attention and cross-attention.

[0020] The de-bouncing module is configured to remove the mismatched points by using an improved adaptive RANSAC algorithm according to each cross-frame matching relationship, calculate the affine transformation parameters between the adjacent frames according to the feature points after removing the mismatched points, and finally complete the video jitter compensation.

[0021] In a third aspect, an electronic device is provided, which includes at least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method for real-time de-bouncing of a blurred video according to any of the embodiments of the present application.

[0022] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, and the program instructions are executed by a processor to enable the processor to perform the steps of the method for real-time de-bouncing of a blurred video according to any of the embodiments of the present application.

[0023] The fuzzy video real-time de-shake method and system of the present application first pre-processes the fuzzy video original frame, converts it into a gray image and scales it into a multi-scale target gray image, and simultaneously calculates the fuzziness, contrast and texture complexity of each target gray image; the number of feature points of each image is detected based on the FAST algorithm, the average response intensity is determined according to the response intensity of the feature points, and these indicators are input into the first multi-layer perception machine to obtain the weight of each target gray image, and the target feature points are extracted and mapped to the original coordinate system according to the weight, forming a feature point set. Then the gray image and the feature point set are input into the MobileNet-FPN network, and a multi-scale feature map is output, and then the indicators such as the fuzziness and contrast of the feature map are input into the second MLP to obtain the weight of each feature map, and the features are extracted and weighted through bilinear interpolation to obtain the local enhanced features of the feature points. Subsequently, the coordinates of the feature points of adjacent frames are sinusoidally encoded, and the local enhanced features are spliced and input into the Transformer, and the cross-attention and cross-attention are used to build the cross-frame matching relationship to obtain an initial matching pair set. Finally, the improved adaptive RANSAC algorithm is used to remove the mismatched points, and the affine transformation parameters of adjacent frames are calculated to complete the shake compensation. This method solves the defects of existing fixed weights through dynamic weight distribution, combines lightweight network and efficient matching strategy, and significantly improves the stabilization accuracy in severe shaking scenes while ensuring the real-time performance of mobile terminals, overcoming the problems of poor robustness of traditional methods and complex calculation of deep methods. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A flowchart of a fuzzy video real-time de-shake method provided by an embodiment of the present application is provided.

[0026] Figure 2 A whole architecture diagram of a fuzzy video real-time de-shake method of a specific embodiment provided by an embodiment of the present application is provided.

[0027] Figure 3 A structure schematic diagram of the first multi-layer perception machine of a specific embodiment provided by an embodiment of the present application is provided.

[0028] Figure 4 A structure schematic diagram of the second multi-layer perception machine of a specific embodiment provided by an embodiment of the present application is provided.

[0029] Figure 5A weighted sampling diagram of a specific embodiment of the MobileNet-FPN network provided by an embodiment of the present application;

[0030] Figure 6 A flowchart of a specific embodiment of the improved adaptive RANSAC algorithm provided by an embodiment of the present application;

[0031] Figure 7 A structural block diagram of a fuzzy video real-time deblurring system provided by an embodiment of the present application;

[0032] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0034] Please refer to Figure 1 , which shows a flowchart of a fuzzy video real-time deblurring method of the present application.

[0035] As shown in Figure 1 , the fuzzy video real-time deblurring method specifically includes the following steps:

[0036] Step S101, gray images corresponding to each original frame image in a fuzzy video are acquired, each gray image is scaled into a target gray image of a preset number of layers, and the blurriness, contrast and texture complexity of each target gray image corresponding to the same gray image are calculated.

[0037] Step S102, the number of feature points of feature points in each target gray image is detected based on the FAST algorithm, and the average response intensity of the target gray image is determined according to the response intensity of each feature point in the same target gray image.

[0038] Step S103, the blurriness, contrast, texture complexity, number of feature points and average response intensity of the same target gray image are input into a preset first multilayer perception machine, and the first multilayer perception machine outputs a first weight ratio of each target gray image corresponding to the same gray image.

[0039] Step S104: Extract feature points from each target grayscale image according to each first weight ratio to obtain at least one target feature point, and map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image.

[0040] Step S105: Input each grayscale image and the target feature point set corresponding to each grayscale image into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels. Input the blurriness, contrast, texture complexity, number of feature points and average response intensity of the feature maps of different levels into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image. Determine the local enhancement features of each target feature point in the target feature point set according to each second weight ratio.

[0041] In this step, each grayscale image and the target feature point set corresponding to each grayscale image are input into the MobileNet network, and the MobileNet network outputs feature maps at different levels.

[0042] Based on the FPN structure, multi-scale features are fused through top-down and lateral connections to generate pyramid features corresponding to feature maps at different levels, thus obtaining FPN feature maps at different levels.

[0043] The coordinates of the target feature points in the grayscale image are mapped onto the FPN feature maps of different levels according to the downsampling rate of different FPN levels in the FPN structure, thus obtaining floating-point coordinates;

[0044] The pyramid level, blur, contrast, number of feature points, and average response intensity of the original grayscale image containing the feature point set are input into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image.

[0045] Local features are extracted using bilinear interpolation at floating-point coordinates, and then interpolated using built-in functions of deep learning frameworks to obtain the interpolated features. And based on the second weight ratio, the features Weighted summation is performed to obtain the local features of each feature point, wherein the expression of the local features is:

[0046] ,

[0047] In the formula, The local features are obtained by weighted summation. , , are weights obtained through the second MLP, 、 、 are features extracted by bilinear interpolation, is the number of batches of images processed simultaneously, is the feature dimension obtained by bilinear interpolation, is the number of feature points extracted by frame image processing.

[0048] In step S106, the coordinates of each target feature point in the gray-scale images of the adjacent two frames and the local enhanced features are input, the coordinates are sinusoidally encoded, and then the local enhanced features are spliced to obtain an input feature matrix of a pre-trained transformer network, and a cross-frame matching relationship between the target feature points of the adjacent two frames is constructed through self-attention and cross-attention.

[0049] In step S107, according to each cross-frame matching relationship, an improved adaptive RANSAC algorithm is used to remove the mismatched points, and according to the feature points after removing the mismatched points, the affine transformation parameters between the adjacent frames are calculated, and finally the video jitter compensation is completed.

[0050] In this step, the cross-frame matching relationship is obtained, wherein each pair of indicates that the i-th feature point of the first frame matches the j-th feature point of the second frame. According to the cross-frame matching relationship, the matching point coordinates are extracted to obtain an initial matching pair set , wherein

[0051] is the number of matching pairs in the initial matching pair set, is the coordinate of the i-th feature point in the initial matching pair set, is the coordinate of the i-th matching feature point. According to the improved adaptive RANSAC algorithm, the inliers are screened to remove the mismatched point pairs.

[0052] According to the feature points after removing the mismatched points, the affine transformation parameters between the adjacent frames are calculated, and finally the video jitter compensation is completed.

[0053] According to the feature points after removing the mismatched points, the affine transformation parameters between the adjacent frames are calculated, and finally the video jitter compensation is completed.

[0054] It should be noted that the screening of inliers to remove mismatched point pairs according to the improved adaptive RANSAC algorithm includes:

[0055] initialization distribution parameters , , optimal model , best inlier number and iteration number are 35. ​

[0056] 3 pairs of points are selected from the matching pairs of points without replacement, and an affine transformation matrix is constructed according to the 3 pairs of points , the affine transformation matrix is solved by the equation group , the expression of the equation group is:

[0057] ,

[0058] In the formula, is the horizontal coordinate of the first matching feature point, is the vertical coordinate of the first matching feature point, is the horizontal coordinate of the third matching feature point, is the vertical coordinate of the first matching feature point, is the horizontal scaling, is the horizontal tilt coefficient, is the horizontal translation coefficient, is the vertical tilt coefficient, is the vertical scaling coefficient, is the vertical translation coefficient;

[0059] converted into a matrix form , and solved by the least square method , and the re-projection error is calculated for each matching pair, and the expression of the re-projection error is:

[0060] ,

[0061] In the formula, is the re-projection error of the initial matching feature point pair, is the horizontal coordinate of the i th matching feature point, is the horizontal coordinate of the i th feature point in the initial matching pair point set, is the vertical coordinate of the i th feature point in the initial matching pair point set, is the vertical coordinate of the i th matching feature point;

[0062] The point pair with the re-projection error less than the threshold value is regarded as an inlier, and the number of inliers of the current model is counted , if , the optimal model is updated: , ;

[0063] if , the and are not updated, and the distribution parameters of are updated: and , and the inlier ratio expectation is calculated ;

[0064] recompute the total number of iterations , for confidence, if the current number of iterations , terminate the loop, otherwise continue the loop.

[0065] In summary, the method of the present application first pre-processes the blurred video original frame, converts it to a gray scale image and scales it to a multi-scale target gray scale image, while calculating the blur, contrast and texture complexity of each target gray scale image; based on the FAST algorithm, the number of feature points of each image is detected, and the average response intensity is determined according to the response intensity of the feature points, and these indicators are input into the first multi-layer perception machine to obtain the weight of each target gray scale image, and the target feature points are extracted and mapped to the original coordinate system according to the weight, forming a feature point set. Then the gray scale image and the feature point set are input into the MobileNet-FPN network, and a multi-scale feature map is output, and then the blur, contrast and other indicators of the feature map are input into the second MLP to obtain the weight of each feature map, and the feature points are extracted and weighted through bilinear interpolation to obtain the local enhanced features of the feature points. Subsequently, the coordinates of the feature points of adjacent frames are sinusoidally encoded, and the local enhanced features are spliced and input into the Transformer, and the cross-attention and cross-attention are used to build the cross-frame matching relationship to obtain the initial matching pair set. Finally, the improved adaptive RANSAC algorithm is used to remove the mismatched points, and the affine transformation parameters of the adjacent frames are calculated to complete the jitter compensation. This method solves the defects of existing fixed weights through dynamic weight distribution, combines lightweight networks and efficient matching strategies, and significantly improves the stabilization accuracy in severe jitter scenes while ensuring real-time performance on mobile devices, overcoming the problems of poor robustness of traditional methods and complex calculation of deep methods.

[0066] In one specific embodiment, a blurred video real-time jitter reduction method based on multi-scale feature enhancement and global matching is used to realize the jitter reduction of unmanned aerial vehicle video images, and the running framework diagram is as shown in Figure 2 , including the following steps:

[0067] Step 1, read the video signal data and pre-process.

[0068] The running system realizes the camera video call. When initialized, the device index (such as 0 to call the default camera) is passed in to create a video capture object. The function is called in a loop to read the video data frame by frame, and the state identifier (succes s is ) and the current frame BGR three-channel image are returned. The function is used to pass in the parameter identifier (such as Acquiring frame rate) Acquiring video properties (frame rate, resolution). After processing, call release() to release resources and close the window. This step provides raw data input for subsequent processing and ensures efficient parsing of real-time streams or files. Uniformly store the read data in the specified folder, and then the system acquires the local path of all pictures in the folder.

[0069] To improve the robustness of the de-bouncing algorithm, the original frame needs to be pre-processed. By reading the pictures in the folder, first, adopt Perform Gaussian filtering with a kernel of size (5, 5) and a standard deviation of 1.5. Perform weighted average blur on the image to suppress high-frequency noise and small disturbances, outputting a smoothed three-channel image. Then, call the function to convert the BGR image to a single-channel grayscale image with the parameter, reducing the data dimension and enhancing feature contrast. Return a grayscale image .

[0070] Step two, multi-scale pyramid FAST detection and dynamic weight fusion.

[0071] Input the grayscale image , the improved FAST algorithm will first scale the image into five layers of different sizes, constructing a pyramid structure to detect feature points at multiple scales and enhance scale invariance.

[0072] For each layer of image, the improved FAST algorithm calculates the image gradient variance, image grayscale standard deviation, and LBP (Local Binary Patterns) based entropy value method to obtain the image blur, contrast, and texture complexity ) of each layer respectively. The formula is as follows:

[0073] Blur ( ): , where, is the image gradient, calculated by , is the image width and height.

[0074] Contrast ( ): , is the image mean.

[0075] The FAST (Features from Accelerated Segment Test) algorithm is used to detect corners. The FAST algorithm detects corners as follows: A candidate pixel is selected, and a circular neighborhood (a circle with a radius of 3 containing 16 pixels) is chosen around that pixel. The brightness values ​​of the selected pixel are compared with the 16 pixels on the circumference. If at least 12 consecutive pixels have a brightness difference greater than 25 from the center pixel, then the pixel is considered a corner (i.e., a feature point).

[0076] After checking the floor The number of feature points is obtained after considering all feature points. The feature points are sorted according to their response intensity, and the average response intensity is calculated. The specific steps are as follows: Construct a small window around the selected feature points and calculate the image gradient of that window (using...). Operator). Through The formula for calculating the response intensity of a feature point in corner detection is as follows: ,in It is the autocorrelation matrix of the image gradient. It is a constant (taken as 0.05). and Represent matrices respectively The determinant and trace, This represents the response intensity of a feature point; the larger the value, the more significant the feature point.

[0077] After obtaining the image blur, contrast, texture complexity, number of feature points, and average response intensity of each layer, these are input into the lightweight network MLP1 (first multilayer perceptron) to dynamically generate the fusion weights for each pyramid level. The structure of MLP1 (…) Figure 3 As shown): Input layer: 5-dimensional vector (blurriness, contrast, texture complexity, number of feature points, average response intensity). Hidden layer: 2 fully connected layers, 8 nodes each, using ReLU activation. Output layer: 5-dimensional vector (corresponding to a five-layer pyramid), with weights normalized using Softmax. The formula is as follows: , ,in, This is the weight matrix for the second layer. This is the weight matrix for the first layer. For input, For the output of the k-th layer, This is the output of the Lth layer. This is the bias vector for the first layer. This is the bias vector for the second layer;

[0078] Then the weight of each target layer is fused with the feature points in the pyramid layer to map back to the original image coordinates: the pyramid layers The detected feature point coordinates are local coordinates Map to coordinates in the original image It can be calculated by the following formula: , , where , is the width and height of the layer. Finally, output the feature point set information (coordinates, level, average response value, blur, contrast).

[0079] Step three, MobileNetV3-large-FPN bilinear interpolation and weighted sampling.

[0080] Input the original image and feature point set information into the feature extraction module (as shown in Figure 5 ). First, use MobileNetV3-large (a lightweight network based on depth separable convolution) as the backbone, output feature maps containing multiple levels (C3, C4, C5), corresponding to different spatial resolutions and semantic levels: C3: shallow features, high resolution (128x128), capturing details and textures. C4: mid-level features, medium resolution (64x64), balancing details and semantics. C5: deep features, low resolution (32x32), encoding global context. Then combine the FPN structure to fuse multi-scale features through top-down and lateral connections to generate unified pyramid features (P3, P4, P5). Starting from C5, through upsampling and lateral connection with the next layer (C4) (1x1 convolution to align the number of channels). Repeat layer by layer to generate P5, P4, P3. Output: P3: 128x128x256 (fusing C3 details and high-level semantics). P4: 64x64x256. P5: 32x32x256.

[0081] Then map the coordinates in the original image to the FPN feature maps of different levels according to the down-sampling rate of different FPN levels to get floating point coordinates , input the pyramid level where the feature points are located and the blur, contrast, average response value, and feature point number of the level to the lightweight network MLP2 as Figure 4 (input layer 5D, hidden layer 2 fully connected, 8 nodes per layer , output layer 3D normalized) dynamically allocate weights for P3-P5 , , where is the pyramid level, BS is the blur, Ct is the contrast, is the number of feature points, The average response intensity.

[0082] Then in floating point coordinates Bilinear interpolation is used to extract local features, and the floating-point coordinates are normalized to the range [-1, 1]. , ,in and For the first The width and height of the layer feature map. Then use the built-in functions of the deep learning framework ( of Interpolation is performed to obtain the interpolated features. Finally, based on the weights generated by MLP2, the features from the three levels are summed using a weighted average: This allows us to obtain the local features of each feature point.

[0083] Step 4: Coordinate encoding and global matching with Transformer.

[0084] (1) Input frame 1 feature point set, the point set contains There are 1 point, each containing coordinates. +Eigenvectors (The feature vector dimension is 256), and the tensor of its feature point set is ( (2D coordinate vector + 256-dimensional feature vector). (The coordinates...) The x and y values ​​are normalized to [0,1] (divided by the image width and height respectively), and then the normalized coordinates are... A 256-dimensional encoded vector is generated using sinusoidal encoding. The formula is as follows:

[0085] , ,

[0086] (Similarly, for coordinates) Sine coding, For dimensional indexing, ∈[0,127]);

[0087] Finally, the positional encoding is added to the feature vector, using the following formula: This fusion method, while keeping the feature dimensions unchanged, deeply couples geometric position information into the feature representation, laying the foundation for subsequent global matching and obtaining a 256-dimensional feature input for the self-attention encoder.

[0088] (2) The features encoded from the feature point coordinates of frame 1 are input into the self-attention encoder to model the contextual relationships between feature points within a single frame. The self-attention encoder is formed by stacking four identical modules. Each module consists of a multi-head self-attention layer, residual connections + layer normalization, and a feedforward network. Among them:

[0089] Multi-head self-attention layer Number of heads 8, dimension per head = 256 / 8 = 32. , Q, K, V represent matrix, matrix , matrix. N is the number of feature points of frame 1.

[0090] Feedforward network : output by a linear layer (256→1024) through activation, and then output by a linear layer (1024→256), and finally residual connection: , output 256-dimensional enhanced features, and similarly obtain the enhanced features of frame 2.

[0091] (3) After obtaining the enhanced features of the adjacent two frames, input the matrix of frame 1 and the , matrix of frame 2 into the cross-attention decoder to model the cross-frame matching relationship between the two frames. The principle is: for each feature point in frame 1, the cross-attention calculates the correlation weight of all feature points in frame 2, and then the frame 2 features are weighted and aggregated based on these weights. This process is essentially to let each feature point in frame 1 find the most similar feature point in frame 2. The cross-attention decoder is also formed by stacking four identical modules, and each module is also composed of a multi-head cross-attention layer, a residual connection+layer normalization, and a feedforward network. Among them:

[0092] Multi-head cross-attention layer Number of heads 8, dimension per head 32, from frame 1, , from frame 2, the formula is the same as , but N is the number of feature points of frame 1, and M is the number of feature points of frame 2.

[0093] Residual connection+layer normalization: ;

[0094] The feedforward network is the same as the encoder, and finally outputs the 256-dimensional matching perception features of frame 1.

[0095] (4) Input the matching perception features of frame 1 output by the cross-attention decoder and the enhanced features of frame 2 output by the self-attention encoder into the matching head to generate the matching probability matrix of the two frames of feature points;

[0096] First, the features of the two frames are normalized to unit length, making the similarity calculation (dot product) equivalent to the cosine similarity, and then the similarity matrix is calculated. The formula is as follows: , , , is used to represent the first feature point of frame 1 and the first feature point of frame 2, N is the number of feature points of frame 1, and M is the number of feature points of frame 2.

[0097] Finally, the bidirectional nearest neighbor method is used to filter the Top-1 point pairs, and a high-confidence matching list is output is the number of valid matching pairs.

[0098] Step five, Bayesian optimization RANSAC motion estimation

[0099] Input the matching list , where each pair represents the matching of the first feature point of frame 1 and the first feature point of frame 2. Since only three matching pairs are needed to calculate the affine transformation matrix, in the improved , 3 pairs of points are selected each time to construct the affine transformation matrix, and the affine transformation matrix with the most inliers is obtained. After Kalman filter smoothing processing of the matrix parameters, the current frame (frame 2) is aligned to the reference frame (frame 1), and the jitter is eliminated.

[0100] According to the matching list, the matching point coordinates are extracted to obtain the initial matching pair set : . Then, the improved is used to filter out false matching point pairs (outliers) such as Figure 6 , to ensure the robustness of the affine matrix. Filter inliers: first initialize distribution parameters =1, =1, and the optimal model , the best inlier number =0, and the iteration number is 35. Then, 3 pairs of points are selected without replacement from the matching point pairs, and an affine transformation matrix : is constructed using the 3 pairs of points. The solution is obtained by the following equation set : ,

[0101] Convert to matrix form , and solve by least squares method (SVD decomposition). ​

[0102] Then calculate the re-projection error for each matched pair: .

[0103] The point pairs with error less than a threshold of 3 pixels are regarded as inliers, and the number of inliers of the current model is counted , if , the optimal model is updated: , , if , the optimal model is not updated , .

[0104] The distribution parameters of are updated: , , and the expectation of the inlier ratio is calculated: . The iteration number is recalculated: ( = 0.99), and if the current iteration number , the loop of the above steps is terminated.

[0105] Finally, the matrix with the most inliers is retained, and the Kalman filter smoothing process is applied to the parameters of the matrix. Then, the affine transformation matrix is applied to transform the image of frame 2 to the coordinate system of frame 1. Then, the main steps two to five are repeated. To suppress the accumulated drift caused by the small error of each affine transformation, the reference frame of the next frame is set to the first 30 frames of the current frame after processing 30 frames of images, thereby achieving smooth video frame output.

[0106] Training strategy and loss function

[0107] The neural network in the algorithm is trained in stages. First, in stage one, the image pair (shaking frame and stable frame) and its affine transformation matrix true value are input, the ImageNet pre-training weight of MobileNetV3-large is fixed, and the multi-scale FAST dynamic weight MLP1 and feature descriptor MLP2 are trained. The loss function is:

[0108] Feature point distribution loss : , is the minimum distance threshold, which is 10 pixels, is the number of feature points.

[0109] Descriptor comparison loss : ,

[0110] In the formula, , , are the anchor point, positive sample, and negative sample features, respectively, The interval hyperparameter (1.0) is set.

[0111] In the second stage, the feature point coordinates and descriptors extracted in the first stage are input, the feature extraction network weight is fixed, and the Transformer encoder-decoder is trained to model the global matching relationship. The loss function is the matching probability cross-entropy loss : , : the true value matching pair set, : the matching probability output by the Transformer.

[0112] The final stage is an end-to-end joint fine-tuning, inputting the original image pair and , and jointly optimizing all module weights. The total loss function is: .

[0113] Where the affine matrix regression loss : , is the feature point coordinates, is the predicted affine matrix.

[0114] Please refer to Figure 7 , which shows the structure block diagram of a fuzzy video real-time deblurring system of the present application.

[0115] As shown in Figure 7 , the fuzzy video real-time deblurring system 200 includes an acquisition module 210, a determination module 220, a first output module 230, a mapping module 240, a second output module 250, a construction module 260, and a deblurring module 270.

[0116] Among them, the acquisition module 210 is configured to acquire gray images corresponding to each original frame image in the fuzzy video, and scale each gray image into a target gray image of a preset number of layers, and calculate the blurriness, contrast, and texture complexity of each target gray image corresponding to the same gray image;

[0117] The determination module 220 is configured to detect the number of feature points in each target gray image based on the FAST algorithm, and determine the average response intensity of the target gray image according to the response intensity of each feature point in the same target gray image;

[0118] The first output module 230 is configured to input the blurriness, contrast, texture complexity, number of feature points, and average response intensity of the same target gray image into a preset first multilayer perception machine, and the first multilayer perception machine outputs the first weight proportion of each target gray image corresponding to the same gray image;

[0119] The mapping module 240 is configured to perform feature point extraction in each target gray scale image according to the respective first weight ratio, to obtain at least one target feature point, and to map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the gray scale image, to obtain a target feature point set of the gray scale image;

[0120] The second output module 250 is configured to input each gray scale image and the target feature point set corresponding to the gray scale image into a preset MobileNet-FPN network, the MobileNet-FPN network outputs feature maps of different levels, and inputs the blur degree, contrast, texture complexity, feature point number and average response intensity of the feature maps of different levels into a preset second multilayer perception machine, the second multilayer perception machine outputs a second weight ratio of each feature map corresponding to the same gray scale image, and determines the local enhanced feature of each target feature point in the target feature point set according to the respective second weight ratio;

[0121] The construction module 260 is configured to input the coordinates and local enhanced features of each target feature point in the gray scale images of adjacent two frames, perform sinusoidal coding on the coordinates, and then splice the local enhanced features to obtain an input feature matrix of a pre-trained transformer network, and construct a cross-frame matching relationship between the target feature points of adjacent two frames through self-attention and cross-attention;

[0122] The de-bouncing module 270 is configured to remove the mismatched points by using an improved adaptive RANSAC algorithm according to the respective cross-frame matching relationship, and calculate the affine transformation parameters between adjacent frames according to the feature points after removing the mismatched points, to finally complete the video jitter compensation.

[0123] It should be understood that, Figure 7 The modules described in the above Figure 1 The modules described in the above Figure 7 The modules described in the above

[0124] In some other embodiments, the present application also provides a computer readable storage medium having a computer program stored thereon, the program instructions being executed by a processor to cause the processor to perform the fuzzy video real-time de-bouncing method in any of the above method embodiments.

[0125] As an implementation manner, the computer readable storage medium of the present application stores computer executable instructions, and the computer executable instructions are configured to:

[0126] obtain a plurality of gray images corresponding to the plurality of original frame images in the blurred video, and scale each of the plurality of gray images into a target gray image of a preset number of layers, and calculate a blur degree, a contrast, and a texture complexity of each target gray image corresponding to the same gray image;

[0127] detect a number of feature points in each target gray image based on a FAST algorithm, and determine an average response intensity of the target gray image according to a response intensity of each feature point in the same target gray image;

[0128] input the blur degree, the contrast, the texture complexity, the number of feature points, and the average response intensity of the same target gray image into a preset first multi-layer perception machine, and the first multi-layer perception machine outputs a first weight ratio of each target gray image corresponding to the same gray image;

[0129] extract feature points in each target gray image according to each first weight ratio to obtain at least one target feature point, and map feature point coordinates of all target feature points to a two-dimensional coordinate system containing the gray image to obtain a target feature point set of the gray image;

[0130] input each gray image and a target feature point set corresponding to the gray image into a preset MobileNet-FPN network, the MobileNet-FPN network outputs feature maps of different levels, and inputs blur degrees, contrasts, texture complexities, numbers of feature points, and average response intensities of the feature maps of different levels into a preset second multi-layer perception machine, the second multi-layer perception machine outputs a second weight ratio of each feature map corresponding to the same gray image, and determines a local enhanced feature of each target feature point in the target feature point set according to each second weight ratio;

[0131] input coordinates and local enhanced features of each target feature point in adjacent two frame gray images, perform sine coding on the coordinates, and then splice the local enhanced features to obtain an input feature matrix of a pre-trained transformer network, and build a cross-frame matching relationship between the target feature points of the adjacent two frames through self-attention and cross-attention;

[0132] According to each cross-frame matching relationship, an improved adaptive RANSAC algorithm is used to remove the mismatched points, and according to the feature points after removing the mismatched points, affine transformation parameters between adjacent frames are calculated, and finally video jitter compensation is completed.

[0133] Computer-readable storage media may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required for at least one function; the data storage area may store data created based on the use of the real-time blur video stabilization system, etc. Furthermore, the computer-readable storage medium may include high-speed random access memory, and may also include memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include memory remotely configured relative to a processor, which can be connected to the real-time blur video stabilization system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 8 As shown, the device includes a processor 310 and a memory 320. The electronic device may also include an input device 330 and an output device 340. The processor 310, memory 320, input device 330, and output device 340 can be connected via a bus or other means. Figure 8 Taking a bus connection as an example, the memory 320 is the computer-readable storage medium described above. The processor 310 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 320, thereby implementing the real-time blur video slewing method described in the above embodiment. The input device 330 can receive input digital or character information and generate key signal inputs related to user settings and function control of the real-time blur video slewing system. The output device 340 may include a display screen or other display device.

[0135] The aforementioned electronic device can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0136] In one implementation, the above-described electronic device is applied in a real-time blurry video stabilization system for a client, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0137] Obtain grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast, and texture complexity of each target grayscale image corresponding to the same grayscale image.

[0138] The number of feature points of each target gray image is detected based on the FAST algorithm, and the average response intensity of the target gray image is determined according to the response intensity of each feature point in the same target gray image.

[0139] The blur, contrast, texture complexity, feature point number and average response intensity of the same target gray image are input into a preset first multi-layer perception machine, and the first multi-layer perception machine outputs a first weight ratio of each target gray image corresponding to the same gray image.

[0140] According to each first weight ratio, feature point extraction is performed in each target gray image to obtain at least one target feature point, and the feature point coordinates of all target feature points are mapped to a two-dimensional coordinate system containing the gray image to obtain a target feature point set of the gray image.

[0141] Each gray image and the target feature point set corresponding to the gray image are input into a preset MobileNet-FPN network, the MobileNet-FPN network outputs feature maps of different levels, and the blur, contrast, texture complexity, feature point number and average response intensity of the feature maps of different levels are input into a preset second multi-layer perception machine, the second multi-layer perception machine outputs a second weight ratio of each feature map corresponding to the same gray image, and determines a local enhanced feature of each target feature point in the target feature point set according to each second weight ratio.

[0142] The coordinates and local enhanced features of each target feature point in adjacent two frames of gray images are input, the coordinates are sinusoidally encoded, and then spliced with the local enhanced features to obtain an input feature matrix of a pre-trained transformer network, and a cross-frame matching relationship between the target feature points of adjacent two frames is constructed through self-attention and cross-attention.

[0143] According to each cross-frame matching relationship, an improved adaptive RANSAC algorithm is used to remove the mismatched points, and according to the feature points after removing the mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally the video jitter compensation is completed.

[0144] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of the various embodiments or some parts of the embodiments.

[0145] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A method for real-time deblurring of a blurred video, characterized in that, The method comprises the following steps: obtaining a plurality of gray images corresponding to each original frame image in the blurred video, and scaling each gray image into a target gray image of a preset number of layers, and calculating the blur, contrast and texture complexity of each target gray image corresponding to the same gray image; detecting the number of feature points in each target gray image based on the FAST algorithm, and determining the average response intensity of the target gray image according to the response intensity of each feature point in the same target gray image; inputting the blur, contrast, texture complexity, number of feature points and average response intensity of the same target gray image into a preset first multi-layer perception machine, and outputting a first weight ratio of each target gray image corresponding to the same gray image by the first multi-layer perception machine; extracting feature points in each target gray image according to each first weight ratio to obtain at least one target feature point, and mapping the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the gray image to obtain a target feature point set of the gray image; inputting each gray image and the target feature point set corresponding to the gray image into a preset MobileNet-FPN network, outputting feature maps of different levels by the MobileNet-FPN network, inputting the blur, contrast, texture complexity, number of feature points and average response intensity of the feature maps of different levels into a preset second multi-layer perception machine, outputting a second weight ratio of each feature map corresponding to the same gray image by the second multi-layer perception machine, and determining a local enhanced feature of each target feature point in the target feature point set according to each second weight ratio; inputting the coordinates and local enhanced features of each target feature point in the gray images of adjacent two frames, performing sine coding on the coordinates, and then splicing the local enhanced features to obtain an input feature matrix of a pre-trained transformer network, and constructing a cross-frame matching relationship between the target feature points of adjacent two frames by self-attention and cross-attention; According to each cross-frame matching relationship, the improved adaptive RANSAC algorithm is used to remove the mismatched points, and the affine transformation parameters between adjacent frames are calculated according to the feature points after removing the mismatched points, and finally the video jitter compensation is completed.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining a plurality of gray images corresponding to each original frame image in the blurred video, and scaling each gray image into a target gray image of a preset number of layers, and calculating the blur, contrast and texture complexity of each target gray image corresponding to the same gray image; detecting the number of feature points in each target gray image based on the FAST algorithm, and determining the average response intensity of the target gray image according to the response intensity of each feature point in the same target gray image; inputting the blur, contrast, texture complexity, number of feature points and average response intensity of the same target gray image into a preset first multi-layer perception machine, and outputting a first weight ratio of each target gray image corresponding to the same gray image by the first multi-layer perception machine; extracting feature points in each target gray image according to each first weight ratio to obtain at least one target feature point, and mapping the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the gray image to obtain a target feature point set of the gray image; inputting each gray image and the target feature point set corresponding to the gray image into a preset MobileNet-FPN network, outputting feature maps of different levels by the MobileNet-FPN network, inputting the blur, contrast, texture complexity, number of feature points and average response intensity of the feature maps of different levels into a preset second multi-layer perception machine, outputting a second weight ratio of each feature map corresponding to the same gray image by the second multi-layer perception machine, and determining a local enhanced feature of each target feature point in the target feature point set according to each second weight ratio; inputting the coordinates and local enhanced features of each target feature point in the gray images of adjacent two frames, performing sine coding on the coordinates, and then splicing the local enhanced features to obtain an input feature matrix of a pre-trained transformer network, and constructing a cross-frame matching relationship between the target feature points of adjacent two frames by self-attention and cross-attention; According to each cross-frame matching relationship, the improved adaptive RANSAC algorithm is used to remove the mismatched points, and the affine transformation parameters between adjacent frames are calculated according to the feature points after removing the mismatched points, and finally the video jitter compensation is completed. input the respective gray-scale images and the target feature point sets corresponding to the respective gray-scale images into a MobileNet network, and the MobileNet network outputs feature maps of different levels; generate pyramid features corresponding to the feature maps of different levels according to the FPN structure by fusing multi-scale features through top-down and horizontal connection, and obtain FPN feature maps of different levels; map the coordinates of the target feature points in the gray-scale image to the FPN feature maps of different levels according to the down-sampling rates of different FPN levels in the FPN structure, and obtain floating-point coordinates; input the blur, contrast, texture complexity, feature point number and average response intensity of the FPN feature maps of different levels into a preset second multi-layer perception, and the second multi-layer perception outputs second weight proportions of the respective feature maps corresponding to the same gray-scale image; The local feature is extracted at the floating point coordinate using bilinear interpolation, and an interpolation function of a deep learning framework is used to obtain the interpolated feature The local features of the feature points are obtained by weighting and summing the features according to the second weight proportion, and an expression of the local feature is: ， wherein, are weighted sum of local features, , , are weights obtained by the second MLP, , , are features extracted by bilinear interpolation, is the number of batches of images processed simultaneously, is the feature dimension obtained by bilinear interpolation, is the number of feature points extracted by frame image processing.

3. The method of claim 1, wherein the method further comprises: the improved adaptive RANSAC algorithm is used to remove the false matching points according to the respective cross-frame matching relationships, affine transformation parameters between adjacent frames are calculated according to the feature points after removing the false matching points, and finally the video jitter compensation is completed, which includes: obtaining a cross-frame matching relationship, wherein each pair of indicates that a first feature point of the first frame is matched with a second feature point of the second frame.​​ According to the cross-frame matching relationship, matching point coordinates are extracted to obtain an initial matching pair set wherein, is the number of matching pairs in the initial matching pair set, is the coordinate of the i-th feature point in the initial matching pair point set, is the coordinate of the i-th matching feature point; screening inliers to remove false matching point pairs according to the improved adaptive RANSAC algorithm; calculating affine transformation parameters between adjacent frames according to the feature points after removing the false matching points, and finally completing the video jitter compensation.

4. The method of claim 3, wherein, the improved adaptive RANSAC algorithm is used to remove the false matching points according to the respective cross-frame matching relationships, affine transformation parameters between adjacent frames are calculated according to the feature points after removing the false matching points, and finally the video jitter compensation is completed, which includes: Initialization Distribution parameters , , optimal model , optimal number of interior points and the number of iterations is 35; 3 pairs of points are selected without replacement from the matched pairs of points, and an affine transformation matrix is constructed according to the 3 pairs of points The affine transformation matrix is solved by an equation set An expression of the equation set is ， wherein is a horizontal coordinate of the first matching feature point, is a vertical coordinate of the first matching feature point, is a horizontal coordinate of the third matching feature point, is a vertical coordinate of the first matching feature point, is a horizontal scaling, is a horizontal skew coefficient, is a horizontal translation coefficient, is a vertical skew coefficient, is a vertical scaling coefficient, is a vertical translation coefficient; Transformed into matrix form and solved by least squares and for each matched pair, the re-projection error is calculated, the expression for which is: ， wherein, is the re-projection error of the initial matched feature point pair, is the horizontal coordinate of the i-th matched feature point, is the horizontal coordinate of the i-th feature point in the initial matched pair point set, is the vertical coordinate of the i-th feature point in the initial matched pair point set, is the vertical coordinate of the i-th matched feature point. If the re-projection error of a point is less than a threshold, the point is considered as an inlier, and the number of inliers of the current model is counted If , the optimal model is updated: , ; If , then do not update and , update the distribution parameters: and and compute the interior point proportion expectation ; Recalculate total iteration count , For confidence, if the current iteration count , terminate loop, else continue loop.

5. A system for real-time deblurring of a blurred video, characterized by including: an acquisition module configured to acquire gray-scale images corresponding to respective original frame images in a blurred video, scale each gray-scale image into a target gray-scale image of a preset number of layers, and calculate the blur, contrast and texture complexity of each target gray-scale image corresponding to the same gray-scale image; a determination module configured to detect the feature point number of feature points in each target gray-scale image based on a FAST algorithm, and determine the average response intensity of a target gray-scale image according to the response intensity of each feature point in the same target gray-scale image; a first output module configured to input the blur, contrast, texture complexity, feature point number and average response intensity of the same target gray-scale image into a preset first multi-layer perception, and the first multi-layer perception outputs first weight proportions of the respective target gray-scale images corresponding to the same gray-scale image; a mapping module configured to extract feature points in each target gray-scale image according to the respective first weight proportions, obtain at least one target feature point, and map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the gray-scale image, to obtain a target feature point set of the gray-scale image; The second output module is configured to input each gray-scale image and a target feature point set corresponding to the gray-scale image into a preset MobileNet-FPN network, the MobileNet-FPN network outputs feature maps of different levels, and inputs the blur degree, contrast, texture complexity, feature point number and average response intensity of the feature maps of different levels into a preset second multi-layer perception machine, the second multi-layer perception machine outputs a second weight ratio of each feature map corresponding to the same gray-scale image, and determines a local enhanced feature of each target feature point in the target feature point set according to each second weight ratio; The construction module is configured to input the coordinates and the local enhanced feature of each target feature point in the gray-scale images of the adjacent two frames, perform sine coding on the coordinates, splice the local enhanced feature to obtain an input feature matrix of a pre-trained transformer network, and construct a cross-frame matching relationship between the target feature points of the adjacent two frames through self-attention and cross-attention. The de-bouncing module is configured to remove the mismatched points by using an improved adaptive RANSAC algorithm according to each cross-frame matching relationship, calculate the affine transformation parameters between the adjacent frames according to the feature points after removing the mismatched points, and finally complete the video jitter compensation.

6. An electronic device, comprising: Comprise: At least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1 to 4.

Images