Fuzzy video real-time jitter removal method and system

Abstract

The invention discloses a fuzzy video real-time jitter removing method and system, and the method comprises the steps: distributing feature points through multi-scale FAST detection, lightweight MLP detection and dynamic weight, combining a MobileNet-FPN network and the lightweight MLP dynamic weight to extract the robust features of the feature points, modeling the global matching relation of the feature points of two adjacent frames through Transform, and carrying out the real-time jitter removing of a fuzzy video. And a Bayesian adaptive RANSAC algorithm is adopted to realize high-efficiency mismatching pair elimination, and finally an affine transformation matrix is solved and reverse mapping is performed to compensate jitter. According to the method, while the real-time performance of the mobile terminal is ensured, the image stabilization precision in a violent shaking scene is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video processing technology, and particularly relates to a method and system for real-time shaking removal of blurry videos. Background Technology

[0002] The widespread application of drones in environmental monitoring, disaster reconnaissance, and other fields places extremely high demands on the real-time acquisition of stable and clear video data. However, videos acquired by drones commonly suffer from image shifts caused by wind disturbances, rotations due to gimbal response delays or drone tilt, scaling due to rapid ascent and descent, and cropping due to asymmetrical motion or lens distortion. Affine transformation can effectively remove video jitter by modeling and compensating for complex motions such as translation, rotation, scaling, and cropping between video frames.

[0003] In existing technologies, traditional feature matching methods (such as SIFT and ORB) rely on manually designed features, have poor robustness to blurred and low-texture scenes, and have a high matching error rate; optical flow-based deep methods (such as PWC-Net and RAFT) have high computational complexity, are difficult to process high-resolution videos in real time, and are sensitive to severe blur; end-to-end stabilization networks (such as DeepStab) have a large number of model parameters, rely on a large amount of training data, and have insufficient generalization ability. Moreover, existing solutions often use fixed-weight multi-scale feature fusion strategies (such as direct concatenation or average pooling), ignoring the differences in the contribution of features at different scales under different scenes. Summary of the Invention

[0004] This invention provides a real-time shaking removal method and system for blurry videos, which addresses the technical problem that existing solutions often employ a multi-scale feature fusion strategy with fixed weights, ignoring the differences in the contribution of features at different scales under different scenarios.

[0005] In a first aspect, the present invention provides a real-time shaking removal method for blurry videos, comprising: Obtain grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast, and texture complexity of each target grayscale image corresponding to the same grayscale image. The number of feature points in each target grayscale image is detected based on the FAST algorithm, and the average response intensity of the target grayscale image is determined based on the response intensity of each feature point in the same target grayscale image. The blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image are input into a preset first multilayer perceptron, and the first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image. Feature points are extracted from each target grayscale image according to each first weight ratio to obtain at least one target feature point, and the feature point coordinates of all target feature points are mapped to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image. Each grayscale image and the target feature point set corresponding to each grayscale image are input into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels. The blurriness, contrast, texture complexity, number of feature points, and average response intensity of the feature maps of different levels are input into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image. Based on each second weight ratio, the local enhancement features of each target feature point in the target feature point set are determined. The coordinates of each target feature point and local enhancement features in the grayscale images of two adjacent frames are input. The coordinates are sinusoidally encoded and then concatenated with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. The cross-frame matching relationship between target feature points in two adjacent frames is constructed through self-attention and cross-attention. Based on the cross-frame matching relationships, an improved adaptive RANSAC algorithm is used to remove mismatched points. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally, video jitter compensation is completed.

[0006] Secondly, the present invention provides a real-time shaking removal system for blurry videos, comprising: The acquisition module is configured to acquire grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast and texture complexity of each target grayscale image corresponding to the same grayscale image. The determination module is configured to detect the number of feature points in each target grayscale image based on the FAST algorithm, and determine the average response intensity of the target grayscale image based on the response intensity of each feature point in the same target grayscale image; The first output module is configured to input the blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image into a preset first multilayer perceptron, and the first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image. The mapping module is configured to extract feature points in each target grayscale image according to each first weight ratio, obtain at least one target feature point, and map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image. The second output module is configured to input each grayscale image and the target feature point set corresponding to each grayscale image into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels and inputs the blurriness, contrast, texture complexity, number of feature points and average response intensity of the feature maps of different levels into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image and determines the local enhancement features of each target feature point in the target feature point set according to each second weight ratio. The module is configured to take as input the coordinates of each target feature point and local enhancement features in the grayscale images of two adjacent frames. The coordinates are sinusoidally encoded and then concatenated with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. The cross-frame matching relationship between target feature points in two adjacent frames is constructed through self-attention and cross-attention. The anti-shake module is configured to use an improved adaptive RANSAC algorithm to remove mismatched points based on the cross-frame matching relationship. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated to finally complete the video shake compensation.

[0007] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the real-time blur video stabilization method according to any embodiment of the present invention.

[0008] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the steps of the real-time blur video stabilization method according to any embodiment of the present invention.

[0009] The real-time jitter reduction method and system for blurred video in this application first preprocesses the original blurred video frames, converting them into grayscale images and scaling them into multi-scale target grayscale images. Simultaneously, the blurriness, contrast, and texture complexity of each target grayscale image are calculated. Based on the FAST algorithm, the number of feature points in each image is detected, and the average response intensity is determined according to the response intensity of the feature points. These indicators are input into a first multilayer perceptron to obtain the weights of each target grayscale image. Based on this, target feature points are extracted and mapped to the original coordinate system to form a feature point set. Next, the grayscale images and feature point sets are input into a MobileNet-FPN network, which outputs multi-scale feature maps. The blurriness, contrast, and other indicators of the feature maps are then input into a second MLP to obtain the weights of each feature map. Features are extracted and weighted using bilinear interpolation to obtain local enhancement features for the feature points. Subsequently, the coordinates of feature points in adjacent frames are sinusoidally encoded, concatenated with the local enhancement features, and input into a Transformer. Self-attention and cross-attention are used to construct cross-frame matching relationships to obtain an initial set of matching pairs. Finally, an improved adaptive RANSAC algorithm is used to remove mismatched points, and the affine transformation parameters of adjacent frames are calculated to complete jitter compensation. This method addresses the shortcomings of existing fixed weights through dynamic weight allocation. By combining a lightweight network with an efficient matching strategy, it significantly improves image stabilization accuracy in scenarios with severe shaking while ensuring real-time performance on mobile devices. This overcomes the problems of poor robustness of traditional methods and computational complexity of deep learning methods. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart of a real-time shaking removal method for blurry video provided in an embodiment of the present invention; Figure 2 An overall architecture diagram of a real-time blur video diffusing method according to a specific embodiment of the present invention is provided. Figure 3 A schematic diagram of the structure of a first multilayer sensor according to a specific embodiment of the present invention; Figure 4 A schematic diagram of the structure of a second multilayer sensor according to a specific embodiment of the present invention; Figure 5 A weighted sampling diagram of a MobileNet-FPN network according to a specific embodiment of the present invention; Figure 6A flowchart of an improved adaptive RANSAC algorithm provided for a specific embodiment of the present invention; Figure 7 This is a structural block diagram of a real-time shaking removal system for blurred video provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Please see Figure 1 The diagram shows a flowchart of a real-time shaking removal method for blurry videos according to this application.

[0014] like Figure 1 As shown, the real-time shaking removal method for blurry videos specifically includes the following steps: Step S101: Obtain grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast, and texture complexity of each target grayscale image corresponding to the same grayscale image.

[0015] Step S102: Detect the number of feature points in each target grayscale image based on the FAST algorithm, and determine the average response intensity of the target grayscale image based on the response intensity of each feature point in the same target grayscale image.

[0016] Step S103: The blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image are input into a preset first multilayer perceptron. The first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image.

[0017] Step S104: Extract feature points from each target grayscale image according to each first weight ratio to obtain at least one target feature point, and map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image.

[0018] Step S105: Input each grayscale image and the target feature point set corresponding to each grayscale image into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels. Input the blurriness, contrast, texture complexity, number of feature points and average response intensity of the feature maps of different levels into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image. Determine the local enhancement features of each target feature point in the target feature point set according to each second weight ratio.

[0019] In this step, each grayscale image and the target feature point set corresponding to each grayscale image are input into the MobileNet network, and the MobileNet network outputs feature maps at different levels. Based on the FPN structure, multi-scale features are fused through top-down and lateral connections to generate pyramid features corresponding to feature maps at different levels, thus obtaining FPN feature maps at different levels. The coordinates of the target feature points in the grayscale image are mapped onto the FPN feature maps of different levels according to the downsampling rate of different FPN levels in the FPN structure, thus obtaining floating-point coordinates; The pyramid level, blur, contrast, number of feature points, and average response intensity of the original grayscale image containing the feature point set are input into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image. Local features are extracted using bilinear interpolation at floating-point coordinates, and then interpolated using built-in functions of deep learning frameworks to obtain the interpolated features. And based on the second weight ratio, the features Weighted summation is performed to obtain the local features of each feature point, wherein the expression of the local features is: , In the formula, The local features are obtained by weighted summation. , , All weights are obtained through the second MLP. , , All features were extracted using bilinear interpolation. The number of images processed in a batch at the same time. The feature dimension obtained by bilinear interpolation. The number of feature points extracted from the frame image processing.

[0020] Step S106: Input the coordinates of each target feature point and local enhancement features in the grayscale images of two adjacent frames, perform sine encoding on the coordinates, and then concatenate them with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. Construct the cross-frame matching relationship between target feature points in two adjacent frames through self-attention and cross-attention.

[0021] Step S107: Based on the cross-frame matching relationship, the improved adaptive RANSAC algorithm is used to remove mismatched points. Based on the feature points after removing the mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally the video jitter compensation is completed.

[0022] In this step, cross-frame matching relationships are obtained, where each pair in the cross-frame matching relationship... Indicates the first frame The feature point and the second frame Matching of feature points; Extract the coordinates of matching points based on cross-frame matching relationships to obtain the initial set of matching pairs. ,in, The initial set of matching pairs contains the number of matching pairs. Let i be the coordinates of the i-th feature point in the initial matching point set. Let i be the coordinates of the i-th matching feature point; The improved adaptive RANSAC algorithm is used to filter in-internal points and remove mismatched point pairs. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally, video jitter compensation is completed.

[0023] It should be noted that the process of filtering in-internal points and removing mismatched point pairs according to the improved adaptive RANSAC algorithm includes: initialization Distribution parameters , Optimal Model Optimal number of interior points And the number of iterations is 35; Select 3 pairs of points without replacement from the matched point pairs, and construct an affine transformation matrix based on the 3 pairs of points. Solve the affine transformation matrix using a system of equations. The system of equations is expressed as follows: , In the formula, The x-coordinate of the first matching feature point. The y-coordinate of the first matching feature point. The x-coordinate of the third matching feature point. The y-coordinate of the first matching feature point. For horizontal scaling, The horizontal tilt coefficient, The horizontal translation coefficient is... The vertical tilt coefficient. This is the vertical scaling factor. This is the vertical translation coefficient; Convert to matrix form And solve using the least squares method And for each matching pair, the reprojection error is calculated. The expression for calculating the reprojection error is: , In the formula, To represent the reprojection error of the initially matched feature point pairs, Let x be the x-coordinate of the i-th matching feature point. Let x be the x-coordinate of the i-th feature point in the initial matching point set. Let be the ordinate of the i-th feature point in the initial matching point set. Let y be the ordinate of the i-th matching feature point; Point pairs whose reprojection error is less than a threshold are considered interior points, and the number of interior points in the current model is counted. ,like Then update the optimal model: , ; like If not, then do not update. and and update Distribution parameters: and And calculate the expected proportion of interior points. ; Recalculate the total number of iterations , The confidence level is the number of iterations in the current iteration. If the condition is met, the loop terminates; otherwise, the loop continues.

[0024] In summary, the method of this application first preprocesses the original blurred video frames, converting them into grayscale images and scaling them into multi-scale target grayscale images. Simultaneously, it calculates the blurriness, contrast, and texture complexity of each target grayscale image. Based on the FAST algorithm, it detects the number of feature points in each image and determines the average response intensity based on the feature point response intensity. These indices are input into a first multilayer perceptron to obtain the weights of each target grayscale image. Based on this, target feature points are extracted and mapped to the original coordinate system to form a feature point set. Next, the grayscale images and feature point sets are input into a MobileNet-FPN network, outputting multi-scale feature maps. The blurriness, contrast, and other indices of the feature maps are then input into a second MLP to obtain the weights of each feature map. Features are extracted and weighted using bilinear interpolation to obtain local enhancement features for the feature points. Subsequently, the coordinates of feature points in adjacent frames are sinusoidally encoded, concatenated with the local enhancement features, and input into a Transformer. Self-attention and cross-attention are used to construct cross-frame matching relationships to obtain an initial set of matching pairs. Finally, an improved adaptive RANSAC algorithm is used to remove mismatched points, and the affine transformation parameters of adjacent frames are calculated to complete jitter compensation. This method addresses the shortcomings of existing fixed weights through dynamic weight allocation. By combining a lightweight network with an efficient matching strategy, it significantly improves image stabilization accuracy in scenarios with severe shaking while ensuring real-time performance on mobile devices. This overcomes the problems of poor robustness of traditional methods and computational complexity of deep learning methods.

[0025] In one specific embodiment, a real-time shaking removal method for blurred videos based on multi-scale feature enhancement and global matching is used to shake drone video images. The operational framework diagram is as follows: Figure 2 As shown, it includes the following steps: Step 1: Read and preprocess the video signal data.

[0026] The operating system passes through Implement camera video capture. During initialization, a video capture object is created by passing the device index (e.g., 0 calls the default camera). This is achieved through a loop. The function reads video data frame by frame and returns a status indicator. (successful for) (and the current frame's BGR three-channel image.) Function input parameter identifiers (such as...) Get the frame rate and video attributes (frame rate, resolution). After processing, call `release()` to release resources and close the window. This step provides raw data input for subsequent processing and ensures efficient parsing of real-time streams or files. All read data is stored in a designated folder, and the system then retrieves the local paths of all images within that folder.

[0027] To improve the robustness of the de-jitter algorithm, preprocessing of the original frames is necessary. This is achieved by first reading images from the folder and then... Perform Gaussian filtering using a kernel of size (5,5) and standard deviation. The image is subjected to weighted average blurring to suppress high-frequency noise and minor perturbations, resulting in a smoothed three-channel image. Then, the following steps are called: Function, with The parameters convert the BGR image to a single-channel grayscale image, reducing data dimensionality and enhancing feature contrast. It returns a grayscale image. .

[0028] Step 2: Multi-scale pyramid FAST detection and dynamic weight fusion.

[0029] Input grayscale image The improved FAST algorithm first scales the image into five layers of different sizes to build a pyramid structure, so as to detect feature points at multiple scales and enhance scale invariance.

[0030] For each image layer, the improved FAST algorithm calculates the image gradient variance, the standard deviation of the image grayscale values, and the entropy method based on LBP (Local Binary Patterns) to obtain the blur, contrast, and texture complexity of each image layer. The formula is as follows: Ambiguity ( ): ,in, Image gradient, through Operator computation, , This represents the image width and height.

[0031] Contrast ( ): , This represents the image mean.

[0032] The FAST (Features from Accelerated Segment Test) algorithm is used to detect corners. The FAST algorithm detects corners as follows: A candidate pixel is selected, and a circular neighborhood (a circle with a radius of 3 containing 16 pixels) is chosen around that pixel. The brightness values ​​of the selected pixel are compared with the 16 pixels on the circumference. If at least 12 consecutive pixels have a brightness difference greater than 25 from the center pixel, then the pixel is considered a corner (i.e., a feature point).

[0033] After checking the floor The number of feature points is obtained after considering all feature points. The feature points are sorted according to their response intensity, and the average response intensity is calculated. The specific steps are as follows: Construct a small window around the selected feature points and calculate the image gradient of that window (using...). Operator). Through The formula for calculating the response intensity of a feature point in corner detection is as follows: ,in It is the autocorrelation matrix of the image gradient. It is a constant (taken as 0.05). and Represent matrices respectively Determinant and trace, This represents the response intensity of a feature point; the larger the value, the more significant the feature point.

[0034] After obtaining the image blur, contrast, texture complexity, number of feature points, and average response intensity of each layer, these are input into the lightweight network MLP1 (first multilayer perceptron) to dynamically generate the fusion weights for each pyramid level. The structure of MLP1 (…) Figure 3 As shown): Input layer: 5-dimensional vector (blurriness, contrast, texture complexity, number of feature points, average response intensity). Hidden layer: 2 fully connected layers, 8 nodes each, using ReLU activation. Output layer: 5-dimensional vector (corresponding to a five-layer pyramid), with weights normalized using Softmax. The formula is as follows: , ,in, This is the weight matrix for the second layer. This is the weight matrix for the first layer. For input, This is the output of the k-th layer. This is the output of layer L. This is the bias vector for the first layer. This is the bias vector for the second layer; Then, the positions of the weighted feature points of each target layer in the pyramid layer are mapped back to the original image coordinates: each layer of the pyramid The detected feature point coordinates are local coordinates. Mapped to coordinates in the original image It can be calculated using the following formula: , ,in , For the first The width and height of the layer are calculated. Finally, the feature point set information (coordinates, layer, average response value, ambiguity, contrast) is output.

[0035] Step 3: MobileNetV3-large-FPN bilinear interpolation and weighted sampling.

[0036] Input the original image and feature point set information into the feature extraction module (e.g.) Figure 5 (As shown). First, MobileNetV3-large (a lightweight network based on depthwise separable convolutions) is used as the backbone, outputting feature maps with multiple layers (C3, C4, C5), corresponding to different spatial resolutions and semantic levels: C3: shallow features, high resolution (128x128), capturing detailed textures. C4: mid-level features, medium resolution (64x64), balancing details and semantics. C5: deep features, low resolution (32x32), encoding global context. Then, the FPN structure is combined to fuse multi-scale features through top-down and lateral connections, generating unified pyramid features (P3, P4, P5). Starting from C5, it is upsampled and laterally connected to the next layer (C4) (1x1 convolution aligns the number of channels). This process is repeated layer by layer to generate P5, P4, and P3. Output: P3: 128x128x256 (fusing C3 details and high-level semantics). P4: 64x64x256. P5: 32x32x256.

[0037] Then, the coordinates in the original image are mapped onto the FPN feature maps of different levels according to the downsampling rate of different FPN levels to obtain floating-point coordinates. The pyramid level of the feature point, along with the ambiguity, contrast, average response value, and number of feature points at that level, are input into the lightweight network MLP2. Figure 4 (Input layer is 5-dimensional, hidden layers are 2 fully connected layers, each layer has 8 nodes) ), output layer 3D ( (Normalized) Dynamically allocate weights for P3-P5 , ,in, This represents the pyramid hierarchy, where BS stands for blurriness and Ct for contrast. For the number of feature points, The average response intensity.

[0038] Then in floating point coordinates Bilinear interpolation is used to extract local features, and the floating-point coordinates are normalized to the range [-1, 1]. , ,in and For the first The width and height of the layer feature map. Then use the built-in functions of the deep learning framework ( of Interpolation is performed to obtain the interpolated features. Finally, based on the weights generated by MLP2, the features from the three levels are summed using a weighted average: This allows us to obtain the local features of each feature point.

[0039] Step 4: Coordinate encoding and global matching with Transformer.

[0040] (1) Input frame 1 feature point set, the point set contains There are 1 point, each containing coordinates. +Eigenvectors (The feature vector dimension is 256), and the tensor of its feature point set is ( (2D coordinate vector + 256-dimensional feature vector). (The coordinates...) The x and y values ​​are normalized to [0,1] (divided by the image width and height respectively), and then the normalized coordinates are... A 256-dimensional encoded vector is generated using sinusoidal encoding. The formula is as follows: , , (Similarly, for coordinates) Sine coding, For dimensional indexing, ∈[0,127]); Finally, the positional encoding is added to the feature vector, using the following formula: This fusion method, while keeping the feature dimensions unchanged, deeply couples geometric position information into the feature representation, laying the foundation for subsequent global matching and obtaining a 256-dimensional feature input for the self-attention encoder.

[0041] (2) The features encoded from the feature point coordinates of frame 1 are input into the self-attention encoder to model the contextual relationships between feature points within a single frame. The self-attention encoder is formed by stacking four identical modules. Each module consists of a multi-head self-attention layer, residual connections + layer normalization, and a feedforward network. Among them: Multi-head self-attention layer ( ) number of heads There are 8 heads, each with dimensions of 8. =256 / 8=32. , They represent Matrix, matrix , Matrix. N is the number of feature points in frame 1.

[0042] Feedforward network ( ): Passed through a linear layer (256→1024) Activate, then output through a linear layer (1024→256), and finally perform residual connections: The 256-dimensional enhanced features are output, and the enhanced features of frame 2 are obtained similarly.

[0043] (3) After obtaining the enhancement features of two adjacent frames, frame 1's Matrix and Frame 2 , The matrix input is fed into the cross-attention decoder to model the cross-frame matching relationship between feature points in two frames. The principle is as follows: for each feature point in frame 1, cross-attention calculates its relevance weights to all feature points in frame 2, and then performs weighted aggregation of the features in frame 2 based on these weights. Essentially, this process allows each feature point in frame 1 to find its most similar feature point in frame 2. The cross-attention decoder is also formed by stacking four identical modules, each module consisting of a multi-head cross-attention layer, residual connections + layer normalization, and a feedforward network. Specifically: Multi-head cross attention layer ( ) number of heads There are 8 heads, each with a dimension of 32. From frame 1, , From frame 2, the formula is the same. ,but N is the number of feature points in frame 1, and M is the number of feature points in frame 2.

[0044] Then perform residual connections and layer normalization: ; The feedforward network, like the encoder, ultimately outputs 256-dimensional frame 1 matching sensing features.

[0045] (4) Input the frame 1 matching perceptual features output by the cross-attention decoder and the frame 2 enhancement features output by the self-attention encoder into the matching head to generate a matching probability matrix of feature points in the two frames; First, the features of these two frames are normalized and scaled to a unit length, making the similarity calculation (dot product) equivalent to the cosine similarity. Then, the similarity matrix is ​​calculated. The formula is as follows: , , , To represent the first frame of frame 1 The feature point and the first feature point in frame 2 There are N feature points, where N is the number of feature points in frame 1 and M is the number of feature points in frame 2.

[0046] Finally, the bidirectional nearest neighbor method is used to filter the top-1 pairs of points, and a high-confidence matching list is output. ( (For the number of valid matching logs).

[0047] Step 5: Bayesian optimization of RANSAC motion estimation Enter matching list Each pair Indicates the first frame of frame 1 The feature point and the first feature point in frame 2 For feature point matching, since only three matching pairs are needed to calculate the affine transformation matrix, the improved method... In the process of filtering inliers and eliminating mismatched point pairs, three pairs of points are selected each time to construct an affine transformation matrix. After obtaining the affine transformation matrix with the most inliers, the matrix parameters are smoothed by Kalman filtering. Finally, the current frame (frame 2) is aligned to the reference frame (frame 1) to eliminate jitter.

[0048] Extract the coordinates of the matching points from the matching list to obtain the initial set of matching pairs. : Then use the improved version. Filtering interior points to remove mismatched point pairs (outliers), such as... Figure 6 This ensures the robustness of the affine matrix. Filtering interior points: First initialize Distribution parameters =1、 =1, and the optimal model Optimal number of interior points =0, the number of iterations is 35. Then, without replacement, 3 pairs of points are selected from the matched point pairs, and the affine transformation matrix is ​​constructed using these 3 pairs of points. : Solve using the following system of equations : , Convert to matrix form And solve it using the least squares method (SVD decomposition). .

[0049] Then, the reprojection error is calculated for each matching pair: .

[0050] Point pairs with an error less than a threshold of 3 pixels are considered inliers, and the number of inliers in the current model is counted. ,like Then update the optimal model: , ,like Then do not update , .

[0051] Update Distribution parameters: , Then calculate the expected proportion of interior points: Recalculate the number of iterations: ( =0.99), if the current iteration number Then, terminate the above steps in the loop.

[0052] Finally, the matrix with the most interior points is retained, and the parameters of the matrix are smoothed by Kalman filtering. Then, the affine transformation matrix is ​​applied to transform the image of frame 2 to the coordinate system of frame 1. Then, the main steps two to five are repeated. In order to suppress the superposition drift caused by the small error of each affine transformation, after processing 30 frames of images, the reference frame of the next frame is set to the 30th frame before the current frame, so as to achieve smooth video frame output.

[0053] Training strategy and loss function The neural network in the algorithm is trained in stages. In stage one, the input image pairs (jittery frames and stable frames) and the true values ​​of their affine transformation matrices are first taken into account. We fix the ImageNet pre-trained weights of MobileNetV3-large and train the multi-scale FAST dynamic weights MLP1 and feature descriptors MLP2. The loss function is: Feature point distribution loss : , The minimum spacing threshold is 10 pixels. This represents the number of feature points.

[0054] Descriptor contrast loss : , In the formula, , , These represent anchor points, positive samples, and negative sample features, respectively. The interval hyperparameter is 1.0.

[0055] In stage two, the feature point coordinates and descriptors extracted in stage one are input as input. The weights of the feature extraction network are fixed, and a Transformer encoder-decoder is trained to model the global matching relationship. The loss function is the matching probability cross-entropy loss. : , : Set of truth-matching pairs : The matching probability output by Transformer.

[0056] The final stage is end-to-end joint fine-tuning, inputting the original image pair and... The weights of all modules are jointly optimized. Total loss function: .

[0057] Among them, the affine matrix regression loss : , The coordinates of the feature point, is the predicted affine matrix.

[0058] Please see Figure 7The diagram shows a structural block diagram of a real-time blur removal system for video according to this application.

[0059] like Figure 7 As shown, the real-time blurry video stunning system 200 includes an acquisition module 210, a determination module 220, a first output module 230, a mapping module 240, a second output module 250, a construction module 260, and a stunning module 270.

[0060] The acquisition module 210 is configured to acquire grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast and texture complexity of each target grayscale image corresponding to the same grayscale image. The determination module 220 is configured to detect the number of feature points in each target grayscale image based on the FAST algorithm, and determine the average response intensity of the target grayscale image based on the response intensity of each feature point in the same target grayscale image. The first output module 230 is configured to input the blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image into a preset first multilayer perceptron, and the first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image. The mapping module 240 is configured to extract feature points in each target grayscale image according to each first weight ratio, obtain at least one target feature point, and map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image. The second output module 250 is configured to input each grayscale image and the target feature point set corresponding to each grayscale image into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels and inputs the blurriness, contrast, texture complexity, number of feature points and average response intensity of the feature maps of different levels into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image and determines the local enhancement features of each target feature point in the target feature point set according to each second weight ratio. The module 260 is configured to take as input the coordinates of each target feature point in the grayscale images of two adjacent frames and the local enhancement features, perform sine encoding on the coordinates, and then concatenate them with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. The cross-frame matching relationship between target feature points in two adjacent frames is constructed through self-attention and cross-attention. The de-shaking module 270 is configured to use an improved adaptive RANSAC algorithm to remove mismatched points based on the cross-frame matching relationship, and calculate the affine transformation parameters between adjacent frames based on the feature points after removing the mismatched points, and finally complete the video shake compensation.

[0061] It should be understood that Figure 7 The modules and references described in the document Figure 1 The steps described in the text correspond to those in the method described above. Therefore, the operations, features, and corresponding technical effects described above also apply to the method described in the text. Figure 7 The various modules in the document will not be described in detail here.

[0062] In other embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the real-time blur video slewing method in any of the above method embodiments. In one embodiment, the computer-readable storage medium of the present invention stores computer-executable instructions, which are configured as follows: Obtain grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast, and texture complexity of each target grayscale image corresponding to the same grayscale image. The number of feature points in each target grayscale image is detected based on the FAST algorithm, and the average response intensity of the target grayscale image is determined based on the response intensity of each feature point in the same target grayscale image. The blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image are input into a preset first multilayer perceptron, and the first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image. Feature points are extracted from each target grayscale image according to each first weight ratio to obtain at least one target feature point, and the feature point coordinates of all target feature points are mapped to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image. Each grayscale image and the target feature point set corresponding to each grayscale image are input into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels. The blurriness, contrast, texture complexity, number of feature points, and average response intensity of the feature maps of different levels are input into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image. Based on each second weight ratio, the local enhancement features of each target feature point in the target feature point set are determined. The coordinates of each target feature point and local enhancement features in the grayscale images of two adjacent frames are input. The coordinates are sinusoidally encoded and then concatenated with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. The cross-frame matching relationship between target feature points in two adjacent frames is constructed through self-attention and cross-attention. Based on the cross-frame matching relationships, an improved adaptive RANSAC algorithm is used to remove mismatched points. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally, video jitter compensation is completed.

[0063] Computer-readable storage media may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required for at least one function; the data storage area may store data created based on the use of the real-time blur video stabilization system, etc. Furthermore, the computer-readable storage medium may include high-speed random access memory, and may also include memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include memory remotely configured relative to a processor, which can be connected to the real-time blur video stabilization system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0064] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 8 As shown, the device includes a processor 310 and a memory 320. The electronic device may also include an input device 330 and an output device 340. The processor 310, memory 320, input device 330, and output device 340 can be connected via a bus or other means. Figure 8 Taking a bus connection as an example, the memory 320 is the computer-readable storage medium described above. The processor 310 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 320, thereby implementing the real-time blur video slewing method described in the above embodiment. The input device 330 can receive input digital or character information and generate key signal inputs related to user settings and function control of the real-time blur video slewing system. The output device 340 may include a display screen or other display device.

[0065] The aforementioned electronic device can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0066] In one implementation, the above-described electronic device is applied in a real-time blurry video stabilization system for a client, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Obtain grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast, and texture complexity of each target grayscale image corresponding to the same grayscale image. The number of feature points in each target grayscale image is detected based on the FAST algorithm, and the average response intensity of the target grayscale image is determined based on the response intensity of each feature point in the same target grayscale image. The blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image are input into a preset first multilayer perceptron, and the first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image. Feature points are extracted from each target grayscale image according to each first weight ratio to obtain at least one target feature point, and the feature point coordinates of all target feature points are mapped to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image. Each grayscale image and the target feature point set corresponding to each grayscale image are input into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels. The blurriness, contrast, texture complexity, number of feature points, and average response intensity of the feature maps of different levels are input into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image. Based on each second weight ratio, the local enhancement features of each target feature point in the target feature point set are determined. The coordinates of each target feature point and local enhancement features in the grayscale images of two adjacent frames are input. The coordinates are sinusoidally encoded and then concatenated with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. The cross-frame matching relationship between target feature points in two adjacent frames is constructed through self-attention and cross-attention. Based on the cross-frame matching relationships, an improved adaptive RANSAC algorithm is used to remove mismatched points. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally, video jitter compensation is completed.

[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time jitter reduction of blurry video, characterized in that, include: Obtain grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast, and texture complexity of each target grayscale image corresponding to the same grayscale image. The number of feature points in each target grayscale image is detected based on the FAST algorithm, and the average response intensity of the target grayscale image is determined based on the response intensity of each feature point in the same target grayscale image. The blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image are input into a preset first multilayer perceptron, and the first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image. Feature points are extracted from each target grayscale image according to each first weight ratio to obtain at least one target feature point, and the feature point coordinates of all target feature points are mapped to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image. Each grayscale image and the target feature point set corresponding to each grayscale image are input into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels. The blurriness, contrast, texture complexity, number of feature points, and average response intensity of the feature maps of different levels are input into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image. Based on each second weight ratio, the local enhancement features of each target feature point in the target feature point set are determined. The coordinates of each target feature point and local enhancement features in the grayscale images of two adjacent frames are input. The coordinates are sinusoidally encoded and then concatenated with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. The cross-frame matching relationship between target feature points in two adjacent frames is constructed through self-attention and cross-attention. Based on the cross-frame matching relationships, an improved adaptive RANSAC algorithm is used to remove mismatched points. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally, video jitter compensation is completed.

2. The real-time shaking removal method for blurry video according to claim 1, characterized in that, The process involves inputting each grayscale image and the corresponding target feature point set into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps at different levels. The blurriness, contrast, texture complexity, number of feature points, and average response intensity of these feature maps at different levels are then input into a preset second multilayer perceptron. The second multilayer perceptron outputs second weight ratios for each feature map corresponding to the same grayscale image. Based on these second weight ratios, the local enhancement features for each target feature point in the target feature point set are determined, including: Each grayscale image and the target feature point set corresponding to each grayscale image are input into the MobileNet network, and the MobileNet network outputs feature maps of different levels. Based on the FPN structure, multi-scale features are fused through top-down and lateral connections to generate pyramid features corresponding to feature maps at different levels, thus obtaining FPN feature maps at different levels. The coordinates of the target feature points in the grayscale image are mapped onto the FPN feature maps of different levels according to the downsampling rate of different FPN levels in the FPN structure, thus obtaining floating-point coordinates; The blurriness, contrast, texture complexity, number of feature points, and average response intensity of FPN feature maps at different levels are input into a preset second multilayer perceptron. The output of the second multilayer perceptron is the second weight ratio of each feature map corresponding to the same grayscale image. Local features are extracted using bilinear interpolation at floating-point coordinates, and then interpolated using built-in functions of deep learning frameworks to obtain the interpolated features. And based on the second weight ratio, the features Weighted summation is performed to obtain the local features of each feature point, wherein the expression of the local features is: ， In the formula, The local features are obtained by weighted summation. , , All weights are obtained through the second MLP. , , All features were extracted using bilinear interpolation. The number of images processed in a batch at the same time. The feature dimension obtained by bilinear interpolation. The number of feature points extracted from the frame image processing.

3. The real-time shaking removal method for blurry video according to claim 1, characterized in that, The process of eliminating mismatched points using an improved adaptive RANSAC algorithm based on the cross-frame matching relationships, calculating affine transformation parameters between adjacent frames based on the feature points after eliminating mismatched points, and finally completing video jitter compensation includes: Obtain cross-frame matching relationships, where each pair in the cross-frame matching relationship... Indicates the first frame The feature point and the second frame Matching of feature points; Extract the coordinates of matching points based on cross-frame matching relationships to obtain the initial set of matching pairs. ,in, The initial set of matching pairs contains the number of matching pairs. Let i be the coordinates of the i-th feature point in the initial matching point set. Let i be the coordinates of the i-th matching feature point; The improved adaptive RANSAC algorithm is used to filter in-internal points and remove mismatched point pairs. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated, and finally, video jitter compensation is completed.

4. The real-time shaking removal method for blurry video according to claim 3, characterized in that, The step of filtering in-internal points and removing mismatched point pairs according to the improved adaptive RANSAC algorithm includes: initialization Distribution parameters , Optimal Model Optimal number of interior points And the number of iterations is 35; Select 3 pairs of points without replacement from the matched point pairs, and construct an affine transformation matrix based on the 3 pairs of points. Solve the affine transformation matrix using a system of equations. The system of equations is expressed as follows: ， In the formula, The x-coordinate of the first matching feature point. The y-coordinate of the first matching feature point. The x-coordinate of the third matching feature point. The y-coordinate of the first matching feature point. For horizontal scaling, The horizontal tilt coefficient, The horizontal translation coefficient is... The vertical tilt coefficient. This is the vertical scaling factor. This is the vertical translation coefficient; Convert to matrix form And solve using the least squares method And for each matching pair, the reprojection error is calculated. The expression for calculating the reprojection error is: ， In the formula, To represent the reprojection error of the initially matched feature point pairs, Let x be the x-coordinate of the i-th matching feature point. Let x be the x-coordinate of the i-th feature point in the initial matching point set. Let be the ordinate of the i-th feature point in the initial matching point set. Let y be the ordinate of the i-th matching feature point; Point pairs whose reprojection error is less than a threshold are considered interior points, and the number of interior points in the current model is counted. ,like Then update the optimal model: , ; like If not, then do not update. and ,renew Distribution parameters: and And calculate the expected proportion of interior points. ; Recalculate the total number of iterations , The confidence level is the number of iterations in the current iteration. If the condition is met, the loop terminates; otherwise, the loop continues.

5. A real-time shaking removal system for blurry videos, characterized in that, include: The acquisition module is configured to acquire grayscale images corresponding to each original frame image in the blurred video, scale each grayscale image to a target grayscale image with a preset number of layers, and calculate the blur, contrast and texture complexity of each target grayscale image corresponding to the same grayscale image. The determination module is configured to detect the number of feature points in each target grayscale image based on the FAST algorithm, and determine the average response intensity of the target grayscale image based on the response intensity of each feature point in the same target grayscale image; The first output module is configured to input the blur, contrast, texture complexity, number of feature points and average response intensity of the same target grayscale image into a preset first multilayer perceptron, and the first multilayer perceptron outputs the first weight ratio of each target grayscale image corresponding to the same grayscale image. The mapping module is configured to extract feature points in each target grayscale image according to each first weight ratio, obtain at least one target feature point, and map the feature point coordinates of all target feature points to a two-dimensional coordinate system containing the grayscale image to obtain the target feature point set of the grayscale image. The second output module is configured to input each grayscale image and the target feature point set corresponding to each grayscale image into a preset MobileNet-FPN network. The MobileNet-FPN network outputs feature maps of different levels and inputs the blurriness, contrast, texture complexity, number of feature points and average response intensity of the feature maps of different levels into a preset second multilayer perceptron. The second multilayer perceptron outputs the second weight ratio of each feature map corresponding to the same grayscale image and determines the local enhancement features of each target feature point in the target feature point set according to each second weight ratio. The module is configured to take as input the coordinates of each target feature point and local enhancement features in the grayscale images of two adjacent frames. The coordinates are sinusoidally encoded and then concatenated with the local enhancement features to obtain the input feature matrix of the pre-trained transformer network. The cross-frame matching relationship between target feature points in two adjacent frames is constructed through self-attention and cross-attention. The anti-shake module is configured to use an improved adaptive RANSAC algorithm to remove mismatched points based on the cross-frame matching relationship. Based on the feature points after removing mismatched points, the affine transformation parameters between adjacent frames are calculated to finally complete the video shake compensation.

6. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 4.

Images