Depth online video stabilization method of time attention based on correlation guidance

Abstract

The invention provides a time attention depth online video stabilization method based on correlation guidance. Video stabilization is realized through a time attention CTA module based on correlation guidance and a distortion field generation PbWM module based on pixels. Wherein the CTA module is used for resampling an input historical unstable frame according to a self-defined measurement; learning the correlation between each resampled historical unstable frame and an input unstable frame to be stabilized, and obtaining the weight of each resampled historical unstable frame; performing residual connection based on the weight to obtain a new input frame sequence; wherein the PbWM module is used for inputting the new continuous sequence, learning distortion mapping and outputting a dual-channel warping field; and performing warping operation based on the dual-channel warping field to generate a stable video. The performance of the stabilizing method provided by the invention is equivalent to that of a traditional offline method, and is superior to that of an existing deep online method, and the performance is higher particularly under the conditions of less frames and higher speed.

Description

Technical Field

[0001] The present invention relates to the field of digital video technology, and in particular to a deep online video stabilization method based on correlation-guided temporal attention. Background Art

[0002] In recent years, with the rapid progress of deep learning technology, deep neural networks (DNN), especially convolutional neural networks (CNN) and their variants, have been widely used in the field of digital video stabilization. Compared with traditional video stabilization technology, DNN-based methods are significantly different. They mainly rely on supervised learning mechanisms and learn by comparing unstable and stable video pairs, rather than directly estimating the motion trajectory of the camera. The core advantage of this method is that it can automatically extract high-dimensional trajectory features, get rid of the dependence on manually designed features, and thus show higher efficiency and robustness.

[0003] For edge computing platforms, implementing online video stabilization technology based on deep learning faces many challenges, the core of which is that the algorithm needs to run efficiently under limited computing resources. So far, traditional online video stabilization technology has mostly relied on local features to track trajectories, but this method is often inadequate in practical applications, especially in complex scenes or low-texture environments where camera motion needs to be separated from object motion. The reliability of its motion estimation is greatly reduced, making it difficult to ensure the stable quality of the video.

[0004] Furthermore, although deep learning has shown great potential in the field of video stabilization, most current methods still rely on future frame information, which limits their application in real-time online scenarios. Although a few online methods such as StabNet have been proposed, how to achieve high-quality video stabilization with limited historical frame information is still a key issue that needs to be solved.

[0005] To this end, we innovatively proposed a deep learning stabilization method called CTAStabNet, which is specially designed for online video stabilization of spherical robots. It aims to achieve efficient video stabilization effects by optimizing the algorithm structure and reducing computational requirements. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a deep online video stabilization method based on correlation-guided temporal attention.

[0007] According to one aspect of the present invention, a deep online video stabilization method based on correlation-guided temporal attention is provided, and video stabilization is achieved through a correlation-guided temporal attention CTA module and a pixel-based warp field generation PbWM module;

[0008] Among them, CTA module:

[0009] Resample the input historical unstable frames according to a custom metric;

[0010] Learn the correlation between the historical unstable frames resampled for each frame and the input unstable frames to be stabilized, and obtain the weight of each resampled historical unstable frame;

[0011] Perform residual connection based on the weights to obtain a new input frame sequence;

[0012] Among them, PbWM module:

[0013] Input the new continuous sequence, learn the warp mapping, and output a dual-channel warp field;

[0014] A warping operation is performed based on the dual-channel warping field to generate a stabilized video.

[0015] Furthermore, the inputted historical unstable frame is an unstable historical frame sampled arbitrarily at timestamp t. The input unstable frame is an unstable frame The two are combined to form a continuous frame sequence of the input The width and height of each frame are denoted as W and H respectively;

[0016] I t Resampled by the CTA module, N key history frames (N < τ) are selected and expressed as Form a new sequence of continuous frames

[0017] Learn with CTA modules and The correlation between them is used as the corresponding weight;

[0018] After the values ​​are passed into a residual block, the output of the CTA module is represented as a new continuous sequence

[0019] The new continuous sequence is input to the PbWM module; the output of the PbWM module is a two-channel warp field T t =[T t x ,T t y ]; warping matrix T t x and T t y The same size as the input frame, T t The shape is 2×W×H;

[0020] By Application Generate the corresponding stable frame, which represents Warping operation.

[0021] Furthermore, resampling the input historical unstable frames according to the custom metric includes:

[0022] Frame resampling is performed based on the key score of each historical frame, and the key score is defined as S kc (n) = k1S movement (n)+k2S clarity (n)+k3S temporal (n), where S movement Indicates the motion intensity in the frame image, S clarity Indicates the clarity of the frame image, S temporal represents the temporal correlation between two frames of images, n = t-τ,…,t-1, k1 = k2 = k3 = 1 represents a constant weight;

[0023] Press S kc (n) Select the first N largest historical frames and arrange them in chronological order as the key frame sequence

[0024]

[0025] The key frame sequence is combined with the input frame to be stabilized, and the resampled frame sequence is expressed as

[0026]

[0027] Furthermore, the motion intensity S in the frame image movement The inter-frame difference algorithm is used to movement Make an estimate, specifically:

[0028] For historical frames The two adjacent frames and Convert to the corresponding grayscale image

[0029] Define the frame difference as Right D n The sum of all pixel values ​​is S n ;

[0030] Calculate S movement (n) Where S a =[S t-τ ,...,S t-1 ];

[0031] The definition S of the frame image clarity Image gradient is used for estimation, specifically:

[0032] For historical frames Use the Sobel gradient operator to calculate the horizontal direction and vertical direction The gradient of the combined gradient is defined as Calculate its variance and express it as σ n ;

[0033] Calculate S clarity for where σ a =[σ t-τ ,...,σ t-1 ];

[0034] The temporal correlation S between the two frames of images temporal Use exponential function to calculate S temporal Make an estimate, specifically: S temporal (n) = e -λ(t-n) , where n=t-τ,...,t-1 and λ is 0.05.

[0035] The correlation-guided attention mechanism is used to learn the correlation between the key frame after each frame resampled and the input unstable frame, and to assign weights to the resampled frame sequence, specifically:

[0036] The cosine similarity is used to measure the correlation between the historical frame and the currently input frame to be stabilized, specifically:

[0037] Frame and Convert to a grayscale image with a shape of W×H;

[0038] Rearrange the two grayscale images into two one-dimensional vectors (A, B) with length WH;

[0039] Calculate cosine similarity in Use as a historical frame The channel descriptor,

[0040] The similarity vector is represented as

[0041] Use two fully connected layers with Sigmoid activation function to perform nonlinear transformation on the similarity vector to obtain the weight vector W v =[w1,…,w N+1 ].

[0042] Furthermore, the performing residual connection based on the weights to obtain a new continuous sequence includes:

[0043] in stands for Hadamard product.

[0044] Furthermore, the PbWM module includes a lightweight U-type network LU-Net, wherein the LU-Net includes a symmetric encoder, a decoder, a skip connection part, and a warp generation part;

[0045] The encoder includes 6 downsampling layers; the channel number of each downsampling layer is {32, 64, 128, 256, 256, 512}; the first three downsampling layers include an inverted residual block InReBlock and a maximum pooling layer MP; each InReBlock includes an expansion layer, a depth convolution, a point convolution and a residual connection; the last three downsampling layers directly use 3×3 two-dimensional convolution Conv2d;

[0046] The decoder consists of 6 upsampling layers, and the upsampling operator adopts transposed convolution TanCov;

[0047] The skip connection part combines the shallow, i.e., low-level features of the encoder with the deep, i.e., high-level features of the decoder;

[0048] The warp generation part will combine the outputs of the encoder, decoder and skip connection part to generate the final warp field.

[0049] Further, the jump connection part adopts a GAB unit;

[0050] The execution process of the GAB unit is specifically as follows:

[0051] Use 1×1 convolution to reduce the dimension of high-level features, use bilinear interpolation for upsampling, and adjust the size of high-level features to match the size of low-level features, including channel size, width, and height;

[0052] The high-level features and low-level features are evenly divided into 4 groups, and each group of high-level features is connected with a corresponding group of low-level features to form 4 groups of connected features;

[0053] Different groups are integrated using expanded convolutions {1, 2, 5, 7} with different expansion rates. These four groups are connected along the channel dimension and 1×1 convolution is performed to achieve interaction between features of different scales.

[0054] The output of each GAB unit is connected to the output of the previous upsampling layer and input to the next upsampling layer to achieve skip connection.

[0055] Furthermore, the warpage generating part comprises:

[0056] After downsampling, the radiometric transformation H is performed t, the radiometric transformation consists of two downsampling Conv2d layers and a linear layer after the sixth encoder layer;

[0057] Transform the radiation H t Multiply by a constant matrix Get T1;

[0058] After upsampling, a Conv2d layer is used to obtain T2;

[0059] Add T2 and T1 to produce the final warp field T t =T1+T2.

[0060] Furthermore, the total loss function is defined as:

[0061] L=αL context +βL shape +γL temporal

[0062] L context is the content loss function, L shape is the shape loss function; L temporal is the time loss function.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] This paper proposes a deep online video stabilization method based on correlation-guided temporal attention, which innovatively constructs a video stabilization network called CTAStabNet. This network consists of two core modules: one is the correlation-guided temporal attention (CTA) module, and the other is the pixel-based distortion map (PbWM) module. This invention marks the first successful introduction of the attention mechanism into the field of deep online video stabilization.

[0065] In the network design, the selection of key frames no longer follows the traditional arbitrary sampling method, but is intelligently resampled based on the correlation between historical unstable frames and incoming unstable frames. In addition, an efficient and lightweight U-shaped network architecture is designed, which can accurately learn and generate pixel-by-pixel distortion fields from consecutive historical unstable frames.

[0066] In order to further improve the network performance, the Group Aggregation Bridge (GAB) unit is introduced, which can effectively deeply fuse low-level and high-level features of different scales instead of simply connecting them between layers. This innovative design enables the network to make full use of multi-scale information to generate more accurate pixel-by-pixel distortion maps.

[0067] During the training process, a carefully designed loss function is used to optimize CTAStabNet. Experimental results show that in most application scenarios, the video stabilization method of the present invention not only achieves a performance level comparable to that of traditional offline methods, but also outperforms existing deep online methods in scenarios with fewer frames and higher speed requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0069] Figure 1 is a diagram of the architecture of CTAStabNet of the present invention;

[0070] Figure 2 This is a CTA module architecture diagram of the present invention;

[0071] Figure 3 It is the architecture diagram of LU-Net of the present invention;

[0072] Figure 4 FIG. 4 is a schematic diagram of the GAB unit of the present invention. DETAILED DESCRIPTION

[0073] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0074] When implementing online video stabilization of a spherical robot, only historical frames can be used, not future frames. In order to utilize the optimal historical frames, the present invention introduces the CTA mechanism to construct CTAStabNet. The goal of this method is to use fewer and optimal historical frames for online video stabilization while maintaining high performance. The selection of historical frames is converted to rely on the correlation between each historical frame and the incoming frame, rather than manual decisions or fixed intervals.

[0075] Based on the above inventive concept, in a preferred embodiment of the present invention, a deep online video stabilization method based on correlation-guided temporal attention is provided, and video stabilization is achieved through a correlation-guided temporal attention CTA module and a pixel-based warp field generation PbWM module. The implementation process may include the following steps:

[0076] CTA module execution:

[0077] S100, resample the input historical unstable frames according to the custom metric;

[0078] S200, learning the correlation between the historical unstable frames resampled for each frame and the input unstable frames to be stabilized, and obtaining the weight of each resampled historical unstable frame;

[0079] S300, performing residual connection based on the weights to obtain a new input frame sequence;

[0080] PbWM module performs:

[0081] S400, inputting the new continuous sequence, learning the warp mapping, and outputting a dual-channel warp field;

[0082] S500, performing a warping operation based on the dual-channel warping field to generate a stabilized video.

[0083] It should be noted that the architecture of CTAStabNet is as follows Figure 1 As shown in Figure 1, it is based on a Siamese network with two branches sharing network weights. The biggest advantage of the Siamese network is to maintain the temporal consistency of consecutive stable frames. During the training phase, two temporally consecutive unstable frame sequences are input into the two branches respectively to estimate two sets of warp maps. However, during the testing phase, only one branch is used to generate a stable frame.

[0084] Based on the above embodiment, in another embodiment of the present invention, a deep online video stabilization method based on correlation-guided temporal attention is further explained.

[0085] The input of CTAStabNet is a continuous frame sequence consisting of the incoming unstable frames and an unstable history frame sampled arbitrarily at timestamp t Composition, merged into The width and height of each frame are denoted as W and H respectively. First, I t The CTA module will resample and select N key history frames (N<τ), which can be expressed as In this way, a new sequence of continuous frames is formed. Then, use the CTA mechanism to learn and After passing a residual block, the output of the CTA module can be represented as a series of frames It is then input to the PbWM module. The output of the PbWM module is a two-channel warp field T t =[T t x ,T t y ]. Warping matrix T t x and Tt y The same size as the input frame, so T t The shape is 2×W×H. Then by applying Generate the corresponding stable frame, which represents Warping operation.

[0086] In order to achieve deep learning-based video stabilization, a continuous sequence of historical frames must be provided to train the deep neural network. In this case, the quality of each historical frame inevitably has a significant impact on the stabilization performance. However, this issue has not been taken into account in existing methods. For example, in PixStabNet, 15 frames are sampled as input historical frames before the input frame. In this case, more redundant information may be included, thereby increasing the computational cost. StabNet only uses 6 historical frames, but the sampling method may lead to the neglect of key historical frames, resulting in low stabilization performance. In order to solve the above problems, in one embodiment of the present invention, an optimal solution for resampling valid historical frames is provided. For this purpose, a CTA module is proposed, and its detailed architecture is as follows: Figure 2 As shown, it mainly includes frame resampling, similarity calculation, fully connected layer (FC) and residual connection.

[0087] Furthermore, in a preferred embodiment, S100 is implemented to perform frame resampling. The specific process is as follows:

[0088] S101, frame resampling is performed according to the key score of each historical frame, which is defined as follows

[0089] S kc (n) = k1S movement (n)+k2S clarity (n)+k3S temporal (n) (1)

[0090] Where S movement Indicates the motion intensity in the frame image, S clarity Indicates the clarity of the frame image, S temporal represents the temporal correlation between two frames of images, n = t-τ,…,t-1, k1 = k2 = k3 = 1 means the weight is constant.

[0091] S movement The definition is used to measure the motion intensity of various objects in the historical frame, which can be used to reflect whether the frame contains more motion information. For video stabilization, if the historical frame has more motion information, it will provide a more useful indication of camera motion and scene changes. In this way, it is easier to capture the motion characteristics of the camera, thereby improving the stabilization performance. Here, the inter-frame difference algorithm is used to stabilize S movement For the historical frame The two adjacent frames (i.e. and ) is converted to the corresponding grayscale image Then define the frame difference as Right D n The sum of all pixel values ​​is S n Finally, calculate S movement (n)

[0092]

[0093] Where S a =[S t-τ ,…,S t-1 ].

[0094] S clarity is defined to measure the clarity of the frame image, which can be used to reflect the degree of clarity. Due to camera motion or scene changes, the sampled frames are easily blurred, resulting in interruptions or noise. Blurred images are not conducive to stability. In practice, the clearer the historical frame, the better the stability performance. Generally speaking, the edges and textures in the clear frame usually have higher gradient values ​​than those in the blurred frame. Therefore, this paper uses the image gradient to measure the sharpness of S clarity For the historical frame Use the Sobel gradient operator to calculate the horizontal direction and vertical direction gradient.

[0095] Define the combined gradient as Calculate its variance and express it as σ n Finally, calculate S clarity for

[0096]

[0097] where σ a =[σ t-τ ,...,σ t-1 ].

[0098] S temporal is defined to measure the temporal correlation between the historical frame and the current input frame. Generally speaking, the closer the historical frame is to the current frame, the more useful the information contained in it is for stabilizing the video. Here, an exponential function is used to calculate S temporal The estimated calculation method is

[0099] S temporal (n) = e -λ(t-n) (4)

[0100] Where n = t-τ,…,t-1 and λ = 0.05.

[0101] S102, based on the above formulas (1) to (4), according to S kc (n) Select the first N largest historical frames and arrange them in chronological order as the key (historical) frame sequence Then combine the key frame sequence with the input frame to be stabilized, and the resampled frame sequence is expressed as

[0102] After selecting a keyframe, the sequence The importance of the information contained in each key frame in is still different. In order to make the model pay more attention to the information of more important frames, Each frame in is regarded as a channel, and then the multi-channel attention mechanism is used to model the correlation between the historical frame and the currently input frame to be stabilized. In a preferred embodiment of the present invention, S200 is implemented to perform similarity calculation. Specifically, the cosine similarity is used to measure the correlation between the historical frame and the currently input frame to be stabilized, and the process is:

[0103] S201, frame and Convert to a grayscale image with a shape of W×H;

[0104] S202, rearrange the two grayscale images into two one-dimensional vectors (A, B) with a length of WH;

[0105] S203, calculate the cosine similarity according to the following formula

[0106]

[0107] in Use as a historical frame The channel descriptor, in particular,

[0108] S206, these similarity vectors are represented as

[0109] S207, use two fully connected layers with Sigmoid activation function to perform nonlinear transformation on the similarity vector to obtain the weight vector W v =[w1,...,w N+1 ].

[0110] In a preferred embodiment of the present invention, S300 is implemented to perform residual connection and calculate the output of the CTA module in stands for Hadamard product.

[0111] The output of the CTA module in the above embodiment is expressed as By retaining a small number of critical historical frames, the video is effectively stabilized. This significantly shortens the calculation time and improves the overall efficiency while ensuring high performance. In addition, obtaining key historical frames based on correlation is more objective and fair, and is less affected by video frame rate fluctuations, thus ensuring the accuracy and reliability of the selection results.

[0112] In PWStableNet, a multi-stage cascaded encoder-decoder architecture is often used to learn the warp mapping, which increases the complexity and computational cost of the network. To this end, a preferred embodiment of the present invention uses a lightweight U-type network (LU-Net) to construct the PbWM module, such as Figure 3 As shown. The input of LU-Net is LU-Net consists of a symmetric encoder-decoder part, a skip connection part, and a warp generation part.

[0113] The encoder consists of 6 downsampling layers, each with a channel number of {32, 64, 128, 256, 256, 512}. The first three downsampling layers consist of an inverted residual block (InReBlock) and a maximum pooling layer (MP). InReBlock also includes an expansion layer, a depthwise convolution, a pointwise convolution, and a residual connection. The residual connection is implemented by adding the input of the expansion layer and the output of the pointwise convolution. The expansion layer uses 1×1 convolution to expand the number of channels, thereby improving representation and learning capabilities. The depthwise convolution applies a single convolution filter to each input channel to perform lightweight filtering. The pointwise convolution uses 1×1 convolution to calculate the linear combination of the depthwise convolution output to produce new features. Both the expansion layer and the depthwise convolution use Leaky Rectified Linear Unit (LeakyReLU) as the nonlinear activation function. Compared with the classic residual block, InReBlock improves representation and learning capabilities while maintaining low parameters and computation. The last three downsampling layers directly use 3×3 two-dimensional convolution (Conv2d), which can retain more features.

[0114] On the other hand, the decoder consists of 6 upsampling layers, and the upsampling operator is performed using transposed convolution (TanCov). The advantage of this method is that the network can automatically learn how to upsample features while retaining more spatial information. In order to improve the accuracy of the warped graph, skip connections are introduced to combine the shallow features (low-level) of the encoder with the deep features (high-level) of the decoder. In this way, multi-scale information can be generated.

[0115] In a preferred embodiment of the present invention, a preferred structure of the skip connection part is introduced, which modifies the Group Aggregation Bridge (GAB) unit in EGE-UNet so that it can effectively integrate low-level features and high-level features instead of using the simple skip connection in U-Net. The detailed structure of the GAB unit is as follows Figure 4 shown.

[0116] First, the high-level features are reduced in dimension using 1×1 convolution and upsampled using bilinear interpolation; in this way, the size of the high-level features is adjusted to match the size of the low-level features, including channel size, width, and height.

[0117] The high-level features and low-level features are then divided equally into 4 groups. Each group of high-level features is concatenated with a corresponding group of low-level features to form 4 groups of concatenated features. Different groups are then integrated using expanded convolutions {1, 2, 5, 7} with different expansion rates. The four groups are concatenated along the channel dimension, followed by a 1×1 convolution to enable interaction between features of different scales.

[0118] Finally, the output of each GAB unit is connected to the output of the previous upsampling layer and then input to the next upsampling layer to achieve skip connection.

[0119] In a preferred embodiment of the present invention, a preferred scheme for warp generation is introduced. Specifically, LU-Net generates two warp fields (T1 and T2) of the same size. T1 is used to roughly estimate the pixel-by-pixel relationship between the unstable frame and the true stable frame. T1 is a 2×3 affine transformation H t The affine transformation is generated by two downsampling Conv2d layers and a linear layer after the sixth encoder layer. Then H t Multiplying a constant matrix T2 is used to recover the residual motion not captured by T1. T2 is generated using a 3×3Conv2d layer after the upsampling layer. Finally, T1 and T2 are added with the same size (2×W×H) to produce the final warp field T t =T1+T2.

[0120] In one embodiment of the present invention, the training process of CTAStabNet relies on three types of losses of reference PWStableNet. The total loss function is defined as follows:

[0121] L=αL context +βL shape +γL temporal (7)

[0122] Among them, α=1, β=1, γ=10 are weights.

[0123] Content loss (L context ) measures the stability of the frame and the real stable frame (F t r ) is aligned, which is defined as:

[0124]

[0125] Among them, the mean square error (MSE) loss (L MSE ) aims to make each pixel in the input frame to be stabilized close to the pixel in the stabilized frame, while the perceptual loss (VGG) aims to alleviate the MSE Caused blur and over-smoothing.

[0126] Shape loss (L shape ) aims to maintain the shape of the warped frame and avoid deformation, which is defined as

[0127]

[0128] For the pixel-by-pixel warping field, it is necessary to achieve pixel-coordinated mapping between unstable frames and stable frames. In order to guide the warping field to perform the correct transformation and align the frames, some sparse key points need to be mapped to the required positions under a given warping field. In this way, the training process can be accelerated. To this end, the feature alignment loss (L feature ) to constrain some sparse key points to the desired location. First, the scale-invariant feature transform (SIFT) features of the stable frame and the unstable frame are extracted. The matching between them is calculated by dividing the frame into 2×2 sub-images and then fitting a homography matrix in each corresponding sub-image using the RANSAC algorithm. The feature points are then matched in the 2×2 sub-images. At timestamp t, m=32 pairs of matching feature points are matched, which are denoted as

[0129] Then based on the distortion field T generated by the model t x and T t y , the corresponding position of the feature point in the unstable frame can be calculated as

[0130]

[0131] The superscripts 'x' and 'y' denote the horizontal and vertical components respectively. feature The calculation method is the average alignment error of the matching feature points, as follows:

[0132]

[0133] To further avoid shape distortion in frames stabilized by the warp field, L grid To constrain the loss T2. First, and Sample an area [x0, y0, w0, h0] in the image, where (x0, y0) is the coordinate of the left vertex, w0, h0 represents the width and height respectively. Then uniformly sample M points in the area [x0, y0, w0, h0], and then calculate L according to the homography matrix grid .

[0134] Temporal loss (L temporal ) measures the smoothness of adjacent stable frames. Figure 1 In the , the twin network separates adjacent unstable frames As input, and share weights between the two branches. Generate two consecutive stable frames Then define L temporal is the mean square error between two adjacent stable frames in According to the pre-calculated optical flow, the TV-L1 algorithm is used to stabilize the frame Warp to stable frame

[0135] In a specific embodiment of the present invention, the DeepStab dataset is used to train CTAStabNet. The dataset consists of 61 pairs of synchronized unstable videos and stable videos. For training, the spatial dimensions of each video are adjusted to W=256 and H=256. The weights of the entire network are initialized according to the normal distribution, and the Adam optimizer is selected with parameters (β1=0.9, β2=0.999). The initial learning rate is set to 0.0001 and multiplied by 0.1 every 10 iterations. The batchsize and the number of epochs are set to 16 and 40, respectively. The number of frames is related to τ=15 and N=5. The method of the above embodiment can be applied to the actual application of spherical robots, and high-performance online video stabilization can be maintained using fewer historical frames, and its video stability is not limited by computing power.

[0136] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A deep online video stabilization method based on correlation-guided temporal attention, characterized in that Video stabilization is achieved through the correlation-guided temporal attention (CTA) module and the pixel-based warp field generation (PbWM) module; Among them, CTA module: Resample the input historical unstable frames according to a custom metric; Learn the correlation between the historical unstable frames resampled for each frame and the input unstable frames to be stabilized, and obtain the weight of each resampled historical unstable frame; Perform residual connection based on the weights to obtain a new continuous sequence; Among them, PbWM module: Input the new continuous sequence, learn the warp mapping, and output a dual-channel warp field; A warping operation is performed based on the dual-channel warping field to generate a stabilized video.

2. A deep online video stabilization method based on correlation-guided temporal attention according to claim 1, characterized in that: The input historical unstable frame is an unstable historical frame sampled arbitrarily at timestamp t The input unstable frame is an unstable frame The two are combined to form a continuous frame sequence of the input The width and height of each frame are denoted as W and H respectively; I t Resampled by the CTA module, N key history frames (N < τ) are selected and expressed as Form a new sequence of continuous frames Learn with CTA modules and The correlation between them is used as the corresponding weight; After the values ​​are passed into a residual block, the output of the CTA module is represented as a new continuous sequence The new continuous sequence is input to the PbWM module; the output of the PbWM module is a two-channel warp field Warp Matrix and The same size as the input frame, T t The shape is 2×W×H; By Application Generate the corresponding stable framework, which represents Warping operation.

3. A deep online video stabilization method based on correlation-guided temporal attention according to claim 2, characterized in that: The resampling of the input historical unstable frames according to the custom metric includes: Frame resampling is performed based on the key score of each historical frame, and the key score is defined as S kc (n) = k1S movement (n)+k2S clarity (n)+k3S temporal (n), where S movement Indicates the motion intensity in the frame image, S clarity Indicates the clarity of the frame image, S temporal represents the temporal correlation between two frames of images, n = t-τ,…,t-1, k1 = k2 = k3 = 1 represents a constant weight; Press S kc (n) Select the first N largest historical frames and arrange them in chronological order as the key frame sequence The key frame sequence is combined with the input frame to be stabilized, and the resampled frame sequence is expressed as 4. A deep online video stabilization method based on correlation-guided temporal attention according to claim 3, characterized in that: The motion intensity S in the frame image movement The inter-frame difference algorithm is used to movement Make an estimate, specifically: For historical frames The two adjacent frames and Convert to the corresponding grayscale image Define the frame difference as Right D n The sum of all pixel values ​​is S n ; calculate Where S a =[S t-τ ,...,S t-1 ]; The definition S of the frame image clarity Image gradient is used for estimation, specifically: For historical frames Use the Sobel gradient operator to calculate the horizontal direction and vertical direction The gradient of the combined gradient is defined as Calculate its variance and express it as σ n ; Calculate S clarity for where σ a =[σ t-τ ,...,σ t-1 ]; The temporal correlation S between the two frames of images temporal Use exponential function to calculate S temporal Make an estimate, specifically: S temporal (n) = e -λ(t-n) , where n=t-τ,...,t-1 and λ is 0.

05.

5. The deep online video stabilization method based on correlation-guided temporal attention according to claim 2, characterized in that: The correlation-guided attention mechanism is used to learn the correlation between the key frame after each frame resampled and the input unstable frame, and to assign weights to the resampled frame sequence, specifically: The cosine similarity is used to measure the correlation between the historical frame and the currently input frame to be stabilized, specifically: Frame and Convert to a grayscale image with a shape of W×H; Rearrange the two grayscale images into two one-dimensional vectors (A, B) with length WH; Calculate cosine similarity in Use as a historical frame The channel descriptor, The similarity vector is represented as Use two fully connected layers with Sigmoid activation function to perform nonlinear transformation on the similarity vector to obtain the weight vector W v =[w1,…,w N+1 ].

6. A deep online video stabilization method based on correlation-guided temporal attention according to claim 5, characterized in that: The performing residual connection based on the weight to obtain a new continuous sequence includes: in stands for Hadamard product.

7. A deep online video stabilization method based on correlation-guided temporal attention according to claim 2, characterized in that: The PbWM module includes a lightweight U-shaped network LU-Net, which includes a symmetrical encoder, decoder, skip connection part and warp generation part; The encoder includes 6 downsampling layers; the channel number of each downsampling layer is {32, 64, 128, 256, 256, 512}; the first three downsampling layers include an inverted residual block InReBlock and a maximum pooling layer MP; each InReBlock includes an expansion layer, a depth convolution, a point convolution and a residual connection; the last three downsampling layers directly use 3×3 two-dimensional convolution Conv2d; The decoder consists of 6 upsampling layers, and the upsampling operator adopts transposed convolution TanCov; The skip connection part combines the shallow, i.e., low-level features of the encoder with the deep, i.e., high-level features of the decoder; The warp generation part will combine the outputs of the encoder, decoder and skip connection part to generate the final warp field.

8. A deep online video stabilization method based on correlation-guided temporal attention according to claim 7, characterized in that: The skip connection part adopts a GAB unit; The execution process of the GAB unit is specifically as follows: Use 1×1 convolution to reduce the dimension of high-level features, use bilinear interpolation for upsampling, and adjust the size of high-level features to match the size of low-level features, including channel size, width, and height; The high-level features and low-level features are evenly divided into 4 groups, and each group of high-level features is connected with a corresponding group of low-level features to form 4 groups of connected features; Different groups are integrated using expanded convolutions {1, 2, 5, 7} with different expansion rates. These four groups are connected along the channel dimension and 1×1 convolutions are used to achieve interaction between features of different scales. The output of each GAB unit is connected to the output of the previous upsampling layer and input to the next upsampling layer to achieve skip connection.

9. A deep online video stabilization method based on correlation-guided temporal attention according to claim 7, characterized in that: The warpage generating part comprises: After downsampling, the radiometric transformation H is performed t , the radiometric transformation consists of two downsampling Conv2d layers and a linear layer after the sixth encoder layer; Transform the radiation H t Multiply by a constant matrix Get T1; After upsampling, a Conv2d layer is used to obtain T2; Add T2 and T1 to produce the final warp field T t =T1+T2.

10. The deep online video stabilization method based on correlation-guided temporal attention according to claim 1, characterized in that: The total loss function is defined as: L=αL context +βL shape +γL temporal L context is the content loss function, L shape is the shape loss function; L temporal is the time loss function.

Images