A deep online video stabilization method based on correlation guided temporal attention

Abstract

The application provides a deep online video stabilization method based on correlation guided temporal attention, which realizes video stabilization through a correlation guided temporal attention (CTA) module and a pixel based warping field generation (PbWM) module; the CTA module: resamples input historical unstable frames according to a self-defined metric; learns the correlation between each resampled historical unstable frame and an input unstable frame to be stabilized to obtain the weight of each resampled historical unstable frame; performs residual connection based on the weight to obtain a new input frame sequence; the PbWM module: inputs the new continuous sequence, learns a warping mapping, and outputs a two-channel warping field; performs warping operation based on the two-channel warping field to generate a stable video. The performance of the stabilization method is comparable to that of a traditional offline method and is superior to that of an existing deep online method, especially in the case of fewer frames and higher speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital video, in particular, to a deep online video stabilization method based on correlation guided temporal attention. BACKGROUND

[0002] In recent years, with the rapid progress of deep learning technology, deep neural networks (DNN), especially convolutional neural networks (CNN) and their variants, have been widely applied in the field of digital video stabilization. Compared with traditional video stabilization techniques, DNN-based methods are significantly different in that they mainly rely on supervised learning mechanisms, learning from pairs of unstable and stable videos rather than directly estimating camera motion trajectories. The core advantage of this approach is that it can automatically extract high-dimensional trajectory features, freeing itself from the dependence on manually designed features, thus showing higher efficiency and robustness.

[0003] For edge computing platforms, implementing deep learning-based online video stabilization technology faces many challenges, the core of which is that the algorithm needs to run efficiently under limited computing resources. So far, traditional online video stabilization techniques have relied on local features to track trajectories, but this approach often fails to meet expectations in practical applications, especially in complex scenes where camera motion needs to be separated from object motion or in low-texture environments, the reliability of motion estimation is greatly compromised, making it difficult to guarantee the stability of the video quality.

[0004] Further, although deep learning has shown great potential in the field of video stabilization, most current methods still rely on future frame information, which limits their application in real-time online scenarios. Although a few online methods such as StabNet have been proposed, how to achieve high-quality video stabilization with limited historical frame information remains a key problem to be solved.

[0005] To this end, we innovatively propose a deep learning stabilization method named CTAStabNet, which is designed specifically for online video stabilization of spherical robots, aiming to achieve efficient video stabilization by optimizing algorithm structure and reducing computational demand. SUMMARY

[0006] In view of the defects in the prior art, the purpose of the present application is to provide a deep online video stabilization method based on correlation guided temporal attention.

[0007] According to one aspect of the present application, a deep online video stabilization method based on correlation guided temporal attention is provided, which realizes video stabilization through a correlation guided temporal attention (CTA) module and a pixel-based warping field generation (PbWM) module.

[0008] The CTA module comprises:

[0009] Resample the inputted history unstable frames according to the self-defined metric;

[0010] Learn the correlation between each resampled history unstable frame and the inputted unstable frame to be stabilized, and obtain the weight of each resampled history unstable frame;

[0011] Based on the weight, perform residual connection to obtain a new input frame sequence;

[0012] Wherein, the PbWM module:

[0013] Input the new continuous sequence, learn the warping mapping, and output a two-channel warping field;

[0014] Based on the two-channel warping field, perform warping operation to generate a stable video.

[0015] Further, the inputted history unstable frame is an unstable history frame sampled at timestamp t The inputted unstable frame is an unstable frame Both of them are combined to form an inputted continuous frame sequence The width and height of each frame are denoted as W and H respectively;

[0016] I t Resample by the CTA module, select N key history frames (N<τ), denoted as Form a new continuous frame sequence

[0017] Learn the correlation between and by using the CTA module, as the corresponding weight;

[0018] After the value is passed into a residual block, the output of the CTA module is denoted as a new continuous sequence

[0019] Input the new continuous sequence into the PbWM module; the output of the PbWM module is a two-channel warping field T t = [T t x , T t y ]; the warping matrix T t x and T t y are the same size as the input frame, T t has a shape of 2×W×H;

[0020] By applying generate the corresponding stable frame, where denotes​​ Warping operation.

[0021] Further, the re-sampling of the input history unstable frames according to the customized metrics comprises:

[0022] Resampling the frames according to the key scores of each history frame, the key score is defined as S kc (n) = k1S movement (n) + k2S clarity (n) + k3S temporal (n), wherein S movement represents the motion intensity in the frame image, S clarity represents the definition of the frame image, S temporal represents the time correlation between two frame images, n = t-τ,…, t-1, k1=k2=k3=1 represents the constant weight value;

[0023] According to the value of S kc (n), the first N large history frames are selected, and the key frame sequence is arranged in time sequence

[0024]

[0025] Combining the key frame sequence with the input unstable frame to obtain the re-sampled frame sequence, which is represented as

[0026]

[0027] Further, the motion intensity S movement in the frame image is estimated by using the inter-frame difference algorithm, specifically: movement

[0028] For the history frame , the adjacent two frames and are converted into corresponding gray images

[0029] The inter-frame difference is defined as The sum of the values of all pixels of D n is obtained to obtain S n ;

[0030] The S movement (n) is calculated as wherein S a = [S t-τ ,...,S t-1 ];

[0031] The definition S clarity of the frame image is estimated by using the image gradient, specifically:

[0032] ​For the history frame The gradients in horizontal direction and vertical direction are calculated respectively using Sobel gradient operator; the combined gradient is defined as The variance of which is denoted as σ n ;

[0033] S clarity is calculated as where σ a = [σ t-τ ,...,σ t-1 ];

[0034] The time correlation S temporal between the two frames of images is calculated as temporal S temporal (n) = e -λ(t-n) , where n = t-τ,...,t-1 and λ takes 0.05.

[0035] The correlation between the key frame after resampling of each frame and the input unstable frame is learned using the attention mechanism based on correlation guidance, and the resampled frame sequence is given a weight, specifically:

[0036] The cosine similarity is used to measure the correlation between the history frame and the current input frame to be stabilized, specifically:

[0037] The frames and are converted into gray-scale images with a shape of WxH;

[0038] The two gray-scale images are rearranged into two one-dimensional vectors (A, B) with a length of WH respectively;

[0039] The cosine similarity is calculated as where is used as the channel descriptor of the history frame ,

[0040] The similarity vector is denoted as

[0041] The similarity vector is nonlinearly transformed using two fully connected layers with Sigmoid activation function to obtain the weight vector W v = [w1,...,w N+1 ].

[0042] Further, the residual connection is performed based on the weight to obtain a new continuous sequence, including:

[0043] where representing the Hadamard product.

[0044] Further, the PbWM module comprises a lightweight U-shaped network (LU-Net), which comprises a symmetrical encoder, a decoder, a skip-connection part and a warping generation part.

[0045] The encoder comprises 6 down-sampling layers; the channel number of each down-sampling layer is {32, 64, 128, 256, 256, 512}; the first three down-sampling layers comprise an inverse residual block (InReBlock) and a max-pooling layer (MP); each InReBlock comprises an expansion layer, a depth convolution, a point convolution and a residual connection; the last three down-sampling layers directly use a 3×3 two-dimensional convolution (Conv2d);

[0046] The decoder comprises 6 up-sampling layers, and the up-sampling operator uses a transpose convolution (TanCov);

[0047] The skip-connection part combines the shallow layer (low-level feature) of the encoder with the deep layer (high-level feature) of the decoder;

[0048] The warping generation part generates the final warping field by combining the outputs of the encoder, the decoder and the skip-connection part.

[0049] Further, the skip-connection part uses a GAB unit;

[0050] The execution process of the GAB unit is as follows:

[0051] The high-order feature is reduced in dimension by using a 1×1 convolution, and is up-sampled by using a bilinear interpolation to adjust the size of the high-level feature to match the size of the low-level feature, including the channel size, the width and the height;

[0052] The high-level feature and the low-level feature are evenly divided into 4 groups, and each group of high-level feature is connected with a corresponding group of low-level feature to form 4 groups of connected features;

[0053] Different groups are integrated by using expansion convolutions with different expansion rates {1, 2, 5, 7}, and the four groups are connected along the channel dimension and 1×1 convolution to realize the interaction between different scale features;

[0054] The output of each GAB unit is connected with the output of the previous up-sampling layer and input to the next up-sampling layer to realize the skip-connection.

[0055] Further, the warping generation part comprises:

[0056] After down-sampling, a homographic transformation H is performed tThe radiometric transformation comprises two down-sampling Conv2d layers and a linear layer after the sixth layer of the encoder;

[0057] The radiometric transformation H t is multiplied by a constant matrix to obtain T1;

[0058] After up-sampling, a Conv2d layer is used to obtain T2;

[0059] T2 and T1 are added to obtain the final warping field T t =T1+T2.

[0060] Further, the total loss function is defined as:

[0061] L=αL context +βL shape +γL temporal

[0062] L context is a content loss function, L shape is a shape loss function; and L temporal is a temporal loss function.

[0063] Compared with the prior art, the present application has the beneficial effects as follows:

[0064] The present application proposes a deep online video stabilization method based on correlation-guided temporal attention, which innovatively constructs a video stabilization network named CTAStabNet. The network is composed of two core modules: one is a correlation-guided temporal attention (CTA) module, and the other is a pixel-based warping map (PbWM) module. This invention marks the first time that attention mechanism has been successfully introduced into the field of deep online video stabilization.

[0065] In network design, the selection of key frames no longer follows the traditional arbitrary sampling method, but intelligently resamples according to the correlation between historical unstable frames and incoming unstable frames. In addition, an efficient lightweight U-shaped network architecture is designed, which can accurately learn and generate pixel-by-pixel warping fields from consecutive historical unstable frames.

[0066] In order to further improve the network performance, a Group Aggregation Bridge (GAB) unit is introduced, which can effectively fuse low-level and high-level features of different scales, rather than simply connecting between layers. This innovative design enables the network to fully utilize multi-scale information, thereby generating more accurate pixel-by-pixel warping maps.

[0067] During the training process, a carefully designed loss function is used to optimize the CTAStabNet. Experimental results show that in most application scenarios, the video stabilization method of the present application not only achieves a performance level comparable to traditional offline methods, but also outperforms existing deep online methods in scenarios with fewer frames and higher speed requirements. BRIEF DESCRIPTION OF DRAWINGS

[0068] Other features, objects, and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments thereof, read in conjunction with the accompanying drawings:

[0069] Figure 1 The architecture diagram of the CTAStabNet of the present application;

[0070] Figure 2 The architecture diagram of the CTA module of the present application;

[0071] Figure 3 The architecture diagram of the LU-Net of the present application;

[0072] Figure 4 The architecture diagram of the GAB unit of the present application. DETAILED DESCRIPTION

[0073] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made. These all belong to the protection scope of the present application.

[0074] When implementing online video stabilization of spherical robots, only historical frames can be used, and future frames cannot be used. In order to utilize the optimal historical framework, the present application introduces a CTA mechanism to construct a CTAStabNet. The goal of this method is to perform online video stabilization using fewer and optimal historical frames while maintaining high performance. The selection of historical frames is converted to rely on the correlation between each historical frame and the incoming frame, rather than being artificially determined or fixed at intervals.

[0075] Based on the above inventive concept, in one preferred embodiment of the present application, a deep online video stabilization method based on correlation-guided temporal attention is provided, which realizes video stabilization through a correlation-guided temporal attention CTA module and a pixel-based warping field generation PbWM module. The implementation process can include the following steps:

[0076] The CTA module performs:

[0077] S100, resampling the input historical unstable frames according to a self-defined metric;

[0078] S200, learning the correlation between each frame of the resampled historical unstable frame and the input unstable frame to be stabilized, to obtain the weight of each resampled historical unstable frame;

[0079] S300, performing residual connection based on the weight to obtain a new input frame sequence;

[0080] The PbWM module performs:

[0081] S400, inputting the new continuous sequence, learning the warping mapping, and outputting a two-channel warping field;

[0082] S500, performing warping operation based on the two-channel warping field to generate a stable video.

[0083] It is to be noted that the architecture of the CTAStabNet is as shown in Figure 1 It is based on a twin network with two branches sharing network weights. The biggest advantage of the twin network is to maintain the temporal consistency of the continuous stable frame. In the training stage, two time-continuous unstable frame sequences are input into two branches respectively, and two sets of warping mappings are estimated respectively. However, in the test stage, only one branch is used to generate a stable frame.

[0084] Based on the above embodiment, in another embodiment of the present application, the depth online video stabilization method based on correlation-guided temporal attention is further explained and described.

[0085] The input of the CTAStabNet is a continuous frame sequence, which is composed of the incoming unstable frame and the unstable historical frame sampled at any time stamp t, which are combined as The width and height of each frame are denoted as W and H respectively. First, I t will be resampled by the CTA module, and N key historical frames (N < τ) are selected and denoted as By this method, a new continuous frame sequence is formed and the correlation between is learned as the corresponding weight. After inputting a residual block, the output of the CTA module can be represented as a series of frames which are then input into the PbWM module. The output of the PbWM module is a two-channel warping field T t =[T t x ,T t y ]. The warping matrices T t x and Tt y The size of the input frame is the same, so T t The shape is 2xWxH. Then by applying The corresponding stable frame is generated, where represents Warping operation.

[0086] In order to realize video stabilization based on deep learning, a continuous historical frame sequence must be provided to train the deep neural network. In this case, the quality of each historical frame inevitably has a significant impact on the stabilization performance. However, this problem has not been considered in existing methods. For example, in PixStabNet, 15 frames are sampled before the input frame as input historical frames. In this case, more redundant information may be included, thereby increasing the computational cost. StabNet only uses 6 historical frames, but the way of sampling may cause the key historical frames to be ignored, thereby resulting in low stabilization performance. To solve the above problems, in an embodiment of the present application, an optimal scheme for resampling effective historical frames is provided, and for this purpose, a CTA module is proposed, the detailed architecture of which is shown in Figure 2 as shown, mainly including frame resampling, similarity calculation, fully connected layer (FC) and residual connection.

[0087] Further, in a preferred embodiment, S100 is implemented to perform frame resampling. The specific process is as follows:

[0088] S101, frame resampling is performed according to the key score of each historical frame, which is defined as follows

[0089] S kc (n) = k1S movement (n) + k2S clarity (n) + k3S temporal (n) (1)

[0090] where S movement represents the motion intensity in the frame image, S clarity represents the sharpness of the frame image, S temporal represents the time correlation between two frame images, n = t - τ, …, t - 1, k1 = k2 = k3 = 1 represents constant weight.

[0091] S movement is defined to measure the motion intensity of various objects in the historical frame, which can be used to reflect whether the frame contains more motion information. For video stabilization, if the historical frame has more motion information, it will provide more useful indications of camera motion and scene changes. In this way, it is easier to capture the motion characteristics of the camera, thereby improving the stabilization performance. Here, the inter-frame difference algorithm is used to estimate S movement ​Two adjacent frames (i.e. and ) are converted to corresponding gray scale images Then the inter-frame difference is defined as The sum of the values of all pixels of D n is denoted as S n . Finally, S movement (n) is calculated as

[0092]

[0093] where S a = [S t-τ ,...,S t-1 ].

[0094] S clarity is defined to measure the sharpness of the frame images, which can be used to reflect the degree of sharpness. The sampled frames are prone to blur due to camera motion or scene changes, resulting in interruptions or noise. Blurred pictures are not conducive to stabilization. In practice, the more sharp the historical frames are, the better the stabilization performance is. Generally speaking, the edges and textures in sharp frames usually have higher gradient values than those in blurred frames. Therefore, this paper estimates S clarity using image gradients. For historical frame , the Sobel gradient operator is used to calculate the gradients in the horizontal direction and the vertical direction , respectively.

[0095] The combined gradient is defined as and its variance is denoted as σ n . Finally, S clarity is calculated as

[0096]

[0097] where σ a = [σ t-τ ,...,σ t-1 ].

[0098] S temporal is defined to measure the temporal correlation of the historical frames with the current input frame. Generally speaking, the closer the historical frames are to the current frame, the more useful the information contained in them is to stabilize the video. Here, S temporal is estimated using an exponential function, which is calculated as

[0099] S temporal (n) = e -λ(t-n) (4)

[0100] where n = t - τ,..., t - 1 and λ = 0.05.

[0101] S102, based on the above formula (1)-(4), according to S kc (n) value size selection of the first N large historical frame, arranged in chronological order as the key (history) frame sequence The key frame sequence is combined with the input to be stabilized frame to obtain the re-sampled frame sequence, which is represented as

[0102] After selecting the key frame, the sequence The importance of the information contained in each key frame is still different. In order to make the model pay more attention to the information of more important frames, each frame in is regarded as a channel, and then a multi-channel attention mechanism is used to model the correlation between the historical frame and the current input to be stabilized frame. In a preferred embodiment of the present application, S200 is implemented to perform similarity calculation. Specifically, cosine similarity is used to measure the correlation between the historical frame and the current input to be stabilized frame, and the process is as follows:

[0103] S201, convert the frame and to a gray image with a shape of WxH;

[0104] S202, rearrange the two gray images into two one-dimensional vectors (A, B) with a length of WH respectively;

[0105] S203, calculate the cosine similarity according to the following formula

[0106]

[0107] wherein is used as the channel descriptor of the historical frame , in particular,

[0108] S206, these similarity vectors are represented as

[0109] S207, use two fully connected layers with Sigmoid activation function to perform nonlinear transformation on the similarity vector to obtain the weight vector W v =[w1,...,w N+1 ].

[0110] In a preferred embodiment of the present application, S300 is implemented to perform residual connection to calculate the output of the CTA module wherein represents Hadamard product.

[0111] The output of the CTA module in the above embodiment is represented as By retaining a small number of but crucial historical frames, effective stabilization of the video is achieved. This significantly shortens the calculation time while ensuring high performance, thereby improving overall efficiency. In addition, the key historical frames are obtained based on correlation, which is more objective and less affected by fluctuations in the frame rate of the video, thereby ensuring the accuracy and reliability of the selection results.

[0112] In PWStableNet, a multi-stage cascaded encoder-decoder architecture is often used to learn the distortion mapping, thereby increasing the complexity and computational cost of the network. Therefore, in one preferred embodiment of the present application, a lightweight U-shaped network (LU-Net) is used to construct the PbWM module, as shown in Figure 3 The input of the LU-Net is The LU-Net is composed of symmetric encoder-decoder parts, and also includes a skip-connection part and a warping generation part.

[0113] The encoder is composed of 6 down-sampling layers, and the number of channels of each down-sampling layer is {32, 64, 128, 256, 256, 512}. The first three down-sampling layers are composed of an Inverted Residual Block (InReBlock) and a Max Pooling layer (MP). In addition, the InReBlock also includes an expansion layer, a depth convolution, a point convolution, and a residual connection. The residual connection is achieved by adding the input of the expansion layer and the output of the point convolution. The expansion layer uses a 1x1 convolution to expand the number of channels, thereby improving the representation and learning ability. The depth convolution applies a single convolution filter to each input channel to perform lightweight filtering. The point convolution uses a 1x1 convolution to calculate a linear combination of the depth convolution output, thereby generating new features. Both the expansion layer and the depth convolution use Leaky Rectified Linear Unit (LeakyReLU) as the nonlinear activation function. Compared with the classic residual block, the InReBlock improves the representation and learning ability while maintaining low parameters and computational complexity. The last three down-sampling layers directly use a 3x3 two-dimensional convolution (Conv2d), which can preserve more features.

[0114] On the other hand, the decoder is composed of 6 up-sampling layers, and the up-sampling operator uses a transposed convolution (TanCov). The advantage of this method is that while preserving more spatial information, the network can automatically learn how to up-sample the features. In order to improve the accuracy of the warping map, a skip-connection is introduced to combine the shallow features (low-level) of the encoder with the deep features (high-level) of the decoder. In this way, multi-scale information can be generated.

[0115] In a preferred embodiment of the present application, the preferred structure of the skip connection part is introduced. The GroupAggregation Bridge (GAB) unit in EGE-UNet is modified to effectively integrate low-level features and high-level features instead of using simple skip connections in U-Net. The detailed structure of the GAB unit is shown in Figure 4 .

[0116] Firstly, the high-order features are reduced in dimension by using 1x1 convolution and up-sampled by using bilinear interpolation. In this way, the size of the high-level features is adjusted to match the size of the low-level features, including the channel size, width and height.

[0117] Then the high-level features and low-level features are evenly divided into 4 groups. Each group of high-level features is connected with the corresponding group of low-level features to form 4 groups of connected features. Then different groups are integrated using unfolding convolution with different unfolding rates {1, 2, 5, 7}. The four groups are connected along the channel dimension, followed by a 1x1 convolution to realize the interaction between different scale features.

[0118] Finally, the output of each GAB unit is connected with the output of the previous up-sampling layer, and then input to the next up-sampling layer to realize the skip connection.

[0119] In a preferred embodiment of the present application, the preferred scheme of the warping generation part is introduced. Specifically, the LU-Net generates two warping fields (T1 and T2) with the same size. T1 is used to roughly estimate the pixel-by-pixel relationship between unstable frames and real stable frames. T1 is generated by a 2x3 affine transformation H t , which is generated by two down-sampling Conv2d layers and a linear layer after the sixth layer of the encoder. Then H t is multiplied by a constant matrix T2 is used to restore the residual motion that T1 has not captured. T2 is generated using a 3x3 Conv2d layer after the up-sampling layer. Finally, T1 and T2 are added with the same size (2xWxH) to generate the final warping field T t =T1+T2.

[0120] In an embodiment of the present application, the training process of CTAStabNet depends on three types of losses of the reference PWStableNet. The total loss function is defined as follows:

[0121] L=αL context +βL shape +γL temporal (7)

[0122] Where α=1, β=1, γ=10 are the weights.

[0123] Content loss (L context ) measures the degree of alignment between the stabilized frame and the ground truth stabilized frame (F t r ) which is defined as:

[0124]

[0125] where the mean squared error (MSE) loss (L MSE ) aims to make each pixel in the input frame to be stabilized close to the pixel in the stabilized frame, while the perceptual loss (VGG) aims to mitigate the blurring and over-smoothing caused by L MSE

[0126] Shape loss (L shape ) aims to preserve the shape of the warped frame and avoid deformation, which is defined as

[0127]

[0128] For the pixel-wise warping field, it is required to achieve the pixel-wise alignment between the unstable frame and the stabilized frame. To guide the warping field to make the correct transformation and align the frames, it is required to map some sparse keypoints to the desired positions under a given warping field. In this way, the training process can be accelerated. For this purpose, feature alignment loss (L feature ) is introduced to constrain some sparse keypoints to the desired positions. First, the scale-invariant feature transform (SIFT) features of the stabilized frame and the unstable frame are extracted. The matching between them is calculated by dividing the frames into 2x2 sub-images and then fitting a homography matrix in each corresponding sub-image using the RANSAC algorithm. Then the matching feature points are matched in the 2x2 sub-images. At time stamp t, m = 32 pairs of matching feature points are matched, which are denoted as

[0129] Then the corresponding positions of the feature points in the unstable frame can be calculated based on the model-generated warping field T t x and T t y

[0130]

[0131] where the superscripts ‘x’ and ‘y’ represent the horizontal and vertical components, respectively. Finally, L feature is calculated as the average alignment error of the matching feature points, as follows:

[0132]

[0133] ​​To further avoid the shape distortion in the warped frame, we use L grid to constrain the loss T2. First, we sample a region [x0, y0, w0, h0] in and , where (x0, y0) is the coordinate of the left top vertex, and w0, h0 represent the width and height respectively. Then we uniformly sample M points in the region [x0, y0, w0, h0], and then compute L grid according to the homography matrix.

[0134] Temporal loss (L temporal ) measures the smoothness of adjacent stabilized frames. In Figure 1 , the twin network takes adjacent unstable frames as input and shares weights between two branches. Two consecutive stabilized frames are then defined as L temporal as the mean square error between two adjacent stabilized frames where According to the pre-computed optical flow, we warp the stabilized frame to the stabilized frame

[0135] In an embodiment of the present application, the CTAStabNet is trained using the DeepStab dataset. The dataset consists of 61 pairs of synchronized unstable and stabilized videos. For training, the spatial dimension of each video is adjusted to W = 256 and H = 256. The weights of the entire network are initialized according to the normal distribution, and the Adam optimizer is selected with parameters (β1 = 0.9, β2 = 0.999). The initial learning rate is set to 0.0001, and multiplied by 0.1 every 10 iterations. The batchsize and the number of epochs are set to 16 and 40 respectively. The number of frames related settings are τ = 15 and N = 5. The method of the above embodiment can be applied to the practical application of spherical robots, and a high-performance online video stabilization can be achieved using fewer historical frames, and the video stabilization is not limited by the computing power.

[0136] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The above preferred features can be used in combination as long as they are not in conflict with each other.

Claims

1. A deep online video stabilization method based on correlation guided temporal attention, characterized in that, The video stabilization is implemented by a correlation guidance based time attention (CTA) module and a PbWM module based on a pixel based warping field generation; The CTA module comprises: Resampling the input historical unstable frames according to a self-defined metric; Learning the correlation between each resampled historical unstable frame and the input unstable frame to be stabilized to obtain the weight of each resampled historical unstable frame; Residual connection based on the weight to obtain a new continuous sequence; The PbWM module comprises: Inputting the new continuous sequence, learning a warping mapping, and outputting a two-channel warping field; Performing warping operation based on the two-channel warping field to generate a stabilized video.

2. The deep online video stabilization method based on correlation guided temporal attention according to claim 1, wherein, The input history unstable frame is an unstable history frame sampled at time stamp t arbitrarily The input unstable frame is an unstable frame Both are combined to form an input continuous frame sequence The width and height of each frame are denoted as W and H respectively I t Resampled by CTA module, select N key history frames (N < τ), denoted as Form a new continuous frame sequence Learning with CTA modules and corresponding weights; After taking in a residual block, the output of the CTA module represents a new continuous sequence The new continuous sequence is input to the PbWM module; the output of the PbWM module is a two-channel warping field Warping matrix and The size of the input frame, T t is 2 x W x H; By applying A corresponding stable frame is generated, in which the representation Warping operation.

3. The deep online video stabilization method based on correlation guided temporal attention according to claim 2, wherein, The resampling of the input historical unstable frames according to the self-defined metric comprises: Frame resampling according to keyness score of each history frame, define keyness score as S kc (n) = k1S movement (n) + k2S clarity (n) + k3S temporal (n), where S movement represents the motion intensity in frame image, S clarity represents the definition of frame image, S temporal represents the time correlation between two frame images, n = t - τ, …, t - 1, k1 = k2 = k3 = 1 represents constant weight S kc (n) Select the top N largest history frames by value size, arrange them in time order as key frame sequence combining the sequence of key frames with the input set of frames to be stabilized to obtain a resampled frame sequence is represented as 4. The deep online video stabilization method based on correlation guided temporal attention according to claim 3, wherein, The motion intensity S in the frame image movement The frame difference algorithm is adopted to S movement Estimation, specifically: for the history frame converts two adjacent frames and to corresponding gray scale images Define the inter-frame difference as Sum the values of all pixels of D n to get S n ; Computations where S a = [S t-τ ,...,S t-1 ]; the sharpness S of the frame image clarity The estimation is performed using image gradients, in particular: For historical frames The gradients in horizontal direction and vertical direction are calculated using Sobel gradient operators respectively; the combined gradient is defined as The variance of which is denoted as σ n ; Compute S clarity For where σ a = [σ t-τ ,..., σ t-1 ] ; the temporal correlation S between the two images temporal S is approximated using an exponential function temporal S is approximated using an exponential function temporal (n) = e -λ(t-n) where n = t - τ,..., t - 1 and λ takes the value 0.

05.

5. The deep online video stabilization method based on correlation guided temporal attention according to claim 2, wherein, The correlation between each resampled key frame and the input unstable frame is learned by using the correlation guidance based attention mechanism, and the resampled frame sequence is given a weight, specifically: The correlation between the historical frame and the current input unstable frame to be stabilized is measured by using the cosine similarity, specifically: frame and converted into a grayscale image of shape W x H; The two gray scale images are rearranged into two one-dimensional vectors (A, B) with a length of WH respectively; Computing cosine similarity wherein Used as a history frame of the channel descriptor, The similarity vector is represented as The similarity vector is non-linearly transformed using two fully connected layers with a sigmoid activation function to obtain a weight vector W v = [w1,..., wn]. N+1 ].

6. The deep online video stabilization method based on correlation guided temporal attention according to claim 5, wherein, The residual connection based on the weight to obtain a new continuous sequence comprises: wherein represents a Hadamard product.

7. The deep online video stabilization method based on correlation guided temporal attention according to claim 2, wherein, The PbWM module comprises a lightweight U-shaped network (LU-Net), and the LU-Net comprises a symmetric encoder, a decoder, a skip connection part and a warping generation part. The encoder comprises six down-sampling layers, and the channel number of each down-sampling layer is {32, 64, 128, 256, 256, 512}; the first three down-sampling layers comprise an inverse residual block (InReBlock) and a maximum pooling layer (MP); each InReBlock comprises an expansion layer, a deep convolution, a point convolution and a residual connection; and the last three down-sampling layers directly use a 3×3 two-dimensional convolution (Conv2d). The decoder comprises six up-sampling layers, and the up-sampling operator uses a transposed convolution (TanCov). The skip connection part combines the shallow layer (low-level feature) of the encoder with the deep layer (high-level feature) of the decoder. The warping generation part generates the final warping field by combining the outputs of the encoder, the decoder and the skip connection part.

8. The deep online video stabilization method based on correlation guided temporal attention according to claim 7, wherein, The skip connection part uses a GAB unit. The execution process of the GAB unit comprises: The high-order feature is reduced in dimension by using a 1×1 convolution, up-sampled by using a bilinear interpolation, and the size of the high-level feature is adjusted to match the size of the low-level feature, including the channel size, the width and the height; The high-level feature and the low-level feature are evenly divided into four groups, each group of high-level feature is connected with a corresponding group of low-level feature to form four groups of connected features; Different groups are integrated by using different expansion rates of expansion convolution {1, 2, 5, 7}, and the four groups are connected along the channel dimension and 1×1 convolution to realize the interaction between different scale features; The output of each GAB unit is connected with the output of the previous up-sampling layer and input to the next up-sampling layer to realize the skip connection.

9. The deep online video stabilization method based on correlation guided temporal attention according to claim 7, wherein, The warping generation part comprises: After downsampling, a radial transform H t The radial transform includes two down-sampling Conv2d layers and a linear layer after the sixth layer of the encoder. Transform H t with a constant matrix T1; After up-sampling, a Conv2d layer is used to obtain T2. T2 and T1 are added to produce the final warping field T t = T1 + T2.

10. The deep online video stabilization method based on correlation guided temporal attention according to claim 1, wherein, The total loss function is defined as: L = aL + βL + γL context L = aL + βL + γL shape L = aL + βL + γL temporal L context is a content loss function, L shape is a shape loss function; L temporal is a temporal loss function.

Images