Pixel-level video image stabilization method and device based on Gaussian attention optical flow

Abstract

The embodiment of the invention discloses a pixel-level video image stabilization method and device based on Gaussian attention optical flow. A specific embodiment of the method comprises the following steps: adjusting each video data frame in a scene video into a stable video frame sequence and an unstable video frame sequence; extracting a Gaussian attention optical flow field of a current frame in the unstable video frame sequence through a Gaussian attention optical flow network to obtain a Gaussian attention optical flow field image of the current frame; performing optical flow feature extraction on the current frame of Gaussian attention optical flow field image through an optical flow lower sampling coupling layer, and adding the extracted optical flow features to a stable network to output a stable distortion graph; and performing distortion processing on each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and sending the processed stable video frame sequence to a display terminal. According to the embodiment, the problem of excessive cutting or local blurring caused by too rough processing can be avoided, and a stable video image is obtained.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of video stabilization, and in particular to a method and device for pixel-level video stabilization based on Gaussian attention optical flow. Background Art

[0002] In today's digital age, video has become one of the main ways for people to obtain information, exchange ideas and record their lives. With the continuous development of digital photography technology, more and more people have begun to use mobile phones, cameras and other devices to shoot videos. The convenience and popularity of digital photography have made video the main medium for the public to record life and share experiences. Not only that, video has a wide range of applications in different fields, including social media, online education, medical imaging, etc. In these application scenarios, users have higher and higher requirements for video quality and stability. For example, in medical imaging, a stable video can provide more accurate diagnostic information; in social media, users are more inclined to share high-quality and smooth videos; in drone scenes, jitter causes low video quality. However, due to the instability of handheld shooting, many users' videos have obvious jitter and shaking, which seriously affects the viewing effect. Video stabilization technology has therefore come into being. Its research aims to solve the jitter problem in the video through algorithmic means and improve the quality and stability of the video. Due to this practical demand, more and more experts and scholars have also devoted themselves to research in this field.

[0003] Video stabilization is one of the cutting-edge research directions in the field of computer vision and image processing, and it also promotes the cross-integration of related fields. The quality of the video directly affects the downstream tasks to a certain extent. For example, target tracking, action recognition, video deraining and defogging, video question answering, etc. Traditional algorithms have been widely studied due to their fast speed and good robustness, but they face the challenge of occlusion caused by depth of field changes and interference of foreground objects on camera posture judgment. Their results usually contain a large number of missing areas at the frame boundaries, especially when there is intense motion in the video. Traditional algorithms still have significant defects when dealing with depth changes, complex camera movements and scene changes. For example, although existing methods can achieve this transformation, they are still too rough, and there are still deficiencies in cropping and grasping local information, resulting in problems such as excessive cropping or local blur.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a pixel-level video stabilization method and device based on Gaussian attention optical flow to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a pixel-level video stabilization method based on Gaussian attention optical flow, the method comprising: obtaining a scene video captured by a head-mounted shooting device; adjusting each video data frame in the above scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the above stable video frame sequence and the above unstable video frame sequence is the same as the number of frames in the above scene video; extracting the Gaussian attention optical flow field of the current frame in the above unstable video frame sequence through a Gaussian attention optical flow network to obtain a Gaussian attention optical flow field image of the current frame; extracting optical flow features from the Gaussian attention optical flow field image of the current frame through an optical flow downsampling coupling layer, and adding the extracted optical flow features to a stable network to output a stable distortion map; using the above stable distortion map, distorting each unstable video frame in the above unstable video frame sequence to generate a processed stable video frame sequence, and sending the processed stable video frame sequence to a display terminal to complete video stabilization.

[0008] In a second aspect, some embodiments of the present disclosure provide a pixel-level video stabilization device based on Gaussian attention optical flow, the device comprising: an acquisition unit, configured to acquire a scene video captured by a head-mounted camera device; an adjustment unit, configured to adjust each video data frame in the above scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the above stable video frame sequence and the above unstable video frame sequence is the same as the number of frames in the above scene video; an image extraction unit, configured to extract the Gaussian attention optical flow field of the current frame in the above unstable video frame sequence through a Gaussian attention optical flow network, and obtain a Gaussian attention optical flow field image of the current frame; a feature extraction unit, configured to perform optical flow feature extraction on the Gaussian attention optical flow field image of the current frame through an optical flow downsampling coupling layer, and add the extracted optical flow features to a stable network to output a stable distortion map; an image distortion processing unit, configured to use the above stable distortion map to perform distortion processing on each unstable video frame in the above unstable video frame sequence to generate a processed stable video frame sequence, and send the processed stable video frame sequence to a display terminal to complete video stabilization.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure, the problem of excessive cropping or local blur can be avoided. Specifically, the reason for the problem of excessive cropping or local blur is: the occlusion caused by the change in depth of field and the interference of the foreground object on the camera posture judgment. Their results usually contain a large number of missing areas at the frame boundaries, especially when there is violent motion in the video. Traditional algorithms still have significant defects when dealing with depth changes, complex camera movements and scene changes. For example, although the existing methods can achieve this conversion, they are still too rough and still have deficiencies in cropping and grasping local information. Based on this, the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure, first, by dividing the scene video into a stable video frame sequence and an unstable video frame sequence, it is easy to extract unstable image features in the video frame. Then, by introducing the Gaussian attention optical flow network, the Gaussian attention optical flow field image can be extracted. In this way, it is easy to use optical flow features instead of feature trajectories to deal with spatially changing motions, and use optical flow to directly learn stability. Next, by introducing the optical flow downsampling coupling layer, the optical flow features of the Gaussian attention optical flow field image can be extracted, and the optical flow features can be refined at the pixel level. At the same time, a more accurate stable distortion map is output based on the stable network. Finally, each unstable video frame can be corrected by the stable distortion map. Thanks to the accuracy of motion estimation by optical flow, the excessive cropping caused by the error of motion estimation is reduced, and the cropping rate is improved. The generalization and robustness of the model are improved in various scenarios. Thus, the video is stabilized at the pixel level based on Gaussian attention optical flow. This avoids the problem of excessive cropping or local blurring caused by overly rough processing, and obtains a stable video image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of a pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure;

[0014] Figure 2 is a schematic diagram of Gaussian constrained attention according to some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow disclosed in the present invention;

[0015] Figure 3 is a schematic diagram of the overall structure of a motion stabilization framework of Gaussian attention optical flow according to some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow disclosed in the present invention;

[0016] Figure 4 is a schematic diagram of a Gaussian guided attention module according to some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow disclosed in the present invention;

[0017] Figure 5 is a schematic diagram of the overall structure of an optical flow downsampling coupling layer according to some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow disclosed in the present invention;

[0018] Figure 6 is a schematic diagram of the overall structure of a pixel-level video stabilization model according to some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow disclosed in the present invention;

[0019] Figure 7 is a schematic structural diagram of some embodiments of a pixel-level video stabilization device based on Gaussian attention optical flow according to the present disclosure;

[0020] Figure 8 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0023] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0024] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0026] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0027] Figure 1 The flowchart 100 of some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure is shown. The pixel-level video stabilization method based on Gaussian attention optical flow comprises the following steps:

[0028] Step 101, obtaining a scene video captured by a head-mounted camera device.

[0029] In some embodiments, the execution subject of the pixel-level video stabilization method based on Gaussian attention optical flow can obtain the scene video shot by the head-mounted shooting device in a wired or wireless manner. The head-mounted shooting device can be a head-mounted network camera. The scene video can be a video shot in any scene.

[0030] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0031] Step 102: Adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence.

[0032] In some embodiments, the execution subject may adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence. The number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video. The stable video frame in the stable video frame sequence may be a frame with a smaller overall change between the previous and next frames of the video. Secondly, the video data frames in the scene video may be adjusted into unstable video frames by image processing. An unstable video frame may be a frame with a larger overall change between the previous and next frames of the video. Here, the image processing method may include but is not limited to image rotation, image affine transformation, image scaling and the like. In addition, each unstable video frame corresponds one-to-one to each stable video frame.

[0033] Optionally, before the execution subject extracts the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through the Gaussian attention optical flow network and obtains the Gaussian attention optical flow field image of the current frame, it may further include the following steps:

[0034] In the first step, each unstable video frame in the unstable video frame sequence is resized to generate a target size video image sequence, wherein the image resizing may include adjusting the height value, width value and number of channels of the image.

[0035] Specifically, due to the addition of optical flow information, in order to carry out subsequent training and calculation, the unstable video frame sequence is adjusted to the same size as the work in Chapter 3, that is, H=256, W=256, C=3.

[0036] The second step is to extract feature points from the above-mentioned target size video image sequence to generate a feature point description information set. Among them, feature points can be extracted from the above-mentioned target size video image sequence through a preset extraction algorithm to generate a feature point description information set. Here, the feature points between the two frames can be matched through the RANSAC (Random Sample Consensus) random sampling consistency algorithm to obtain a matching current frame feature point set and a next frame feature point set.

[0037] As an example, the extraction algorithm may include but is not limited to at least one of the following: SIFT (Scale-invariant Feature Transform) algorithm, Surf (Speeded Up Robust Features) algorithm, Harris corner detection, etc.

[0038] The third step is to normalize the feature point coordinates included in each feature point description information in the feature point description information set to generate a processed feature point description information set. In order to facilitate calculation and training, all coordinates are normalized to the interval [-1,1].

[0039] The fourth step is to determine the affine matrix between each two adjacent frames of the target size video image sequence in accordance with the feature point description information set after the above processing, as another input of the Gaussian attention optical flow network to generate the Gaussian attention optical flow field image of the current frame.

[0040] As an example, we can first construct the following coordinate matrix A based on the coordinates of the feature points:

[0041]

[0042] Secondly, construct the feature point coordinate vector B:

[0043] B=[x1,x2,...,x n , y1, y2, ..., y n ] T .

[0044] Then, the product of the coordinate matrix A as the dependent variable and the radiation vector X is equal to B. Here, X can be expanded as:

[0045] X=[a1, a2, a3, a4, a5, a6] T .

[0046] Next, the affine matrix is ​​calculated frame by frame from the unstable frame to the corresponding stable frame. Then, it is converted into the corresponding distortion map. Where A is a 2×3 real number matrix, which is the deformed X.

[0047] In practice, in order to solve the technical problems mentioned in the background technology, CNN and the synergy of CNN and Transformer dual-branch networks are often used. First, by verifying the effectiveness of using UNet, the dual branch fuses two types of feature information by using the HAFM module, and the model has achieved performance improvements in cropping rate, distortion score, and stability score. This proves the effectiveness of CNN and Transformer dual branches. The benefits brought by adding the Transformer branch are attributed to the Transformer's ability to continuously utilize the attention features in the frame. This makes the extraction of motion information more refined and the temporal feature information in unstable videos more reasonably learned.

[0048] However, the above implementation still has some limitations. Due to the limited motion patterns in the data set, the model cannot adapt to all motion patterns well. In some complex motion patterns (such as fast scene changes, large-area moving objects in the frame sequence, etc.), the distortion map will have inaccurate estimation problems, resulting in poor viewing of the stable frame sequence. Due to the lack of depth estimation, the model may have blurred contours near those objects in scenes with large depth of field and strong parallax. Since the stabilization method based on the optical flow method estimates the global affine transformation matrix and optimizes a single camera trajectory based on this, there are often multiple parts with inconsistent motion directions or local feature changes in the scene, and each block in the image has its own motion trajectory. If the global estimate is converted into a local block estimate, additional resources will be consumed. Many methods have exposed their limitations, including but not limited to inaccurate optical flow estimation, local distortion and distortion, jitter of objects in the image, etc., despite excellent indicator data, it still causes discomfort to the human eye. In response to the above technical problem 2, the inventor decided to adopt the following solution to solve it.

[0049] Step 103, extracting the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through the Gaussian attention optical flow network, and obtaining the Gaussian attention optical flow field image of the current frame.

[0050] In some embodiments, the execution subject may extract the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through a Gaussian attention optical flow network to obtain a Gaussian attention optical flow field image of the current frame.

[0051] In some optional implementations of some embodiments, the Gaussian attention optical flow network may include: a motion backbone network, a Gaussian constraint layer, a feature processing module, a context backbone network, a convolutional network, a Gaussian attention guidance module and a decoder. And the above-mentioned execution subject extracts the Gaussian attention optical flow field of the current frame in the above-mentioned unstable video frame sequence through the Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame, which may include the following steps:

[0052] In the first step, the motion backbone network is used to extract motion features of the unstable video frames corresponding to time t and time t+1 in the unstable video frame sequence to obtain the moving object mask of the current frame. The motion backbone network can be used to extract motion features between two adjacent frames.

[0053] In the second step, the context features corresponding to the unstable video frame at time t are extracted through the context backbone network and the convolutional network layer. The context backbone network can be used to extract the context features of the unstable video frame at time t (i.e., the current frame).

[0054] In the third step, the current frame moving object mask is input into a feature processing module combined with the Gaussian Constrained Layer to generate a feature map. The Gaussian Constrained Layer (GCL) can output a feature map in combination with a feature encoder.

[0055] In practice, the Gaussian Constrained Layer (GCL) is essentially used to obtain fine-grained structural information and local discriminative representation for feature matching. The structure is designed based on the standard Transformer block. Specifically, given the basic feature X, its formula is:

[0056] X′=GCA(LN(T))+X

[0057] Y=FFN(LN(T′))+X′.

[0058] Where LN(·) and FFN(·) represent the layer normalization and feedforward network in the Transformer block, respectively. GCA(·) represents Gaussian constrained attention. T represents feature mapping. Y represents motion features, which may include the value vector V(Value), the deformed Gaussian kernel, the Gaussian kernel bias, and the Gaussian kernel amplitude. Figure 2 As shown, first, the k×k deformed Gaussian kernel Glr is obtained by taking the dot product of the Gaussian distribution N(0, σ) and the amplitude. The Gaussian kernel here is obtained by representing the feature processing module. It is initialized to a standard Gaussian distribution and is learnable. Then the query vector Q(Query), key vector K(Key) and value vector V(Value) of the motion feature are obtained. The similarity matrix is ​​obtained by multiplying Q and K. After passing it through the linear layer Linear, the product with the Gaussian kernel Glr is calculated. The result vector is calculated with the softmax value and then multiplied with the value vector V to obtain the Gaussian constrained attention score. The formula is:

[0059]

[0060] Among them, during the inference process, Glr is used as a constraint mask to reorganize the weights of attention feature aggregation.

[0061] The fourth step is to input the above context features and feature maps into the above Gaussian Guided Attention Module to generate Gaussian self-attention features. Among them, the Gaussian Guided Attention Module (GGAM) uses a spatially deformed Gaussian kernel to smooth motion features. Because the kernel is rigid or fixed and clutter will inevitably provide wrong guidance. For this reason, bias and amplitude are added to guide the attention calculation. Figure 3As shown in the figure, firstly, the value vector V(Value), bias matrix and amplitude matrix of motion feature Y are obtained through three convolutional layers. Then the dot product of the amplitude matrix and the Gaussian distribution is calculated, and the result and the bias matrix are input into the warping function to obtain the Gaussian kernel. Then the similarity matrix is ​​calculated according to the query vector Q(Query) and key vector K(Key) of the context feature X. After the similarity matrix is ​​input into the linear layer, it is dot-producted with the Gaussian kernel. Then the result vector is dot-producted with the value vector V(Value) of the motion feature Y to obtain the Gaussian guided attention score, which is as follows:

[0062]

[0063] Among them, G is the Gaussian kernel after adding the amplitude. is the bias. Warp(·) is the warping function.

[0064] As an example, the overall structure of the motion stabilization framework of Gaussian attention optical flow can be as follows Figure 4 shown.

[0065] The fifth step is to decode the Gaussian self-attention features through the decoder to generate the Gaussian attention optical flow field image of the current frame.

[0066] Step 104, extracting optical flow features from the Gaussian attention optical flow field image of the current frame through the optical flow downsampling coupling layer, and adding the extracted optical flow features to the stabilization network to output a stable distortion map.

[0067] In some embodiments, the execution entity may extract optical flow features from the Gaussian attention optical flow field image of the current frame through an optical flow downsampling coupling layer, and add the extracted optical flow features to a stable network to output a stable distortion map.

[0068] In some optional implementations of some embodiments, the execution subject adds the extracted optical flow features to the stabilization network to output a stable distortion map, which may include the following steps:

[0069] In the first step, each unstable video frame in the unstable video frame sequence is grayed to generate a grayscale image sequence.

[0070] In the second step, the grayscale images in the grayscale image sequence are encoded through the encoding module in the above-mentioned stable network, and the above-mentioned optical flow features are added to the three network layers in the encoding module in the above-mentioned stable network for joint encoding to generate multi-head attention encoding features and normalized encoding features.

[0071] As an example, the overall structure of the optical flow downsampling coupling layer (unit) is as follows Figure 5As shown in Figure 1, the stable network can play a role in motion smoothing in the process. TUNet (Transformer U-shaped Network) is used as the stable network. Figure 6 The overall structure diagram of the pixel-level video stabilization model is shown in Figure 1. The TUNet branch consists of an encoder and a decoder. The resolution of the feature map decreases with the depth of the network, while the number of channels increases. The left side is the feature extraction network (encoder), and the right side is the feature fusion network (decoder). The encoder is used to capture the contextual information of the image, while the decoder is used to restore details and generate prediction results with the same resolution as the input image, so as to achieve modeling of local details and global context at the same time.

[0072] The TUNet model couples the convolutional features in the branches with the embedded features (FDCU) output by the optical flow downsampling coupling layer (unit) in the following way: convolutional modules are used along the branches to gradually integrate feature maps and patch embeddings in an interactive manner. In order to maintain a consistent embedding size, an asymmetric sampling step strategy is adopted in the interactive process. Specifically, since the dimension of the convolutional feature is C×H×W (C, H, W represent the number of channels, height and width of the tensor respectively), and the shape of the embedding is (T+1)×E (T and E represent the number of image embeddings and the embedding dimension respectively), it involves the mutual conversion of two-dimensional and three-dimensional features. Since the dimension size of the embedded feature is always constant, while the convolutional feature is always changing with the up and down sampling process, this makes the direct connection method invalid. To achieve this, upsampling and downsampling are used to connect the two, and in the feature conversion process, different step sizes are used for up and down sampling, and the convolution and deconvolution of each step are controlled to maintain a consistent feature size through different step sizes. In terms of spatial dimensions, different sizes of embedded features are used to ensure that the embedding dimension is the same after each sampling.

[0073] In the third step, the multi-head attention encoding features and the normalized encoding features are decoded through the decoding module in the above-mentioned stable network to generate a stable distortion map.

[0074] Among them, you can Figure 5 As shown, the decoding module of the TUNet model can output a stable distortion map based on the above multi-head attention encoding features and the above normalized encoding features.

[0075] In practice, GAFSF obtains the mask of the moving object in the stable frame sequence through the semantic network, and extracts the optical flow of the moving object in the stable frame sequence as the true value in combination with the mask. The difference is that this process only acts on the proposed loss function, and it is no longer necessary to judge whether the moving object is in motion.

[0076] Then the grayscale image sequence of the unstable frames of two adjacent frames is used as input to TUNet for dual-branch calculation:

[0077] T i-1，i =BN(Act(Conv(T i-1,i )))

[0078] T i-1，i =MLP(MHSA(T i-1，i )).

[0079] Secondly, add optical flow information to the second to fourth modules:

[0080] T i-1，i =BN(Act(Conv(T i-1,i )))+BN(Act(Conv(G i )))

[0081] T i-1，i =MLP((MHSA(T i-1,i ))+MLP(MHSA(G i )).

[0082] Then, after four modules, the distortion graph Wi is obtained:

[0083]

[0084] Finally, the final stable frame can be generated through the final distortion map Wi. Since the optical flow calculation process needs to combine the previous and next frames, the final stable sequence has 1 less frame than the original one.

[0085] Step 105 , using the stable warp map, warping each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and sending the processed stable video frame sequence to a display terminal to complete video stabilization.

[0086] In some embodiments, the execution entity may utilize the stable distortion map to perform distortion processing on each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and send the processed stable video frame sequence to a display terminal to complete video stabilization.

[0087] Optionally, the Gaussian attention optical flow network, the optical flow downsampling coupling layer and the stabilization network constitute a pixel-level video stabilization model, and the training samples of the pixel-level video stabilization model are generated by the following steps:

[0088] The first step is to obtain a sample video frame sequence.

[0089] The second step is to adjust the sample video frame sequence into a stable sample frame sequence and an unstable sample frame sequence, and adjust the size of each stable sample frame and each unstable sample frame.

[0090] The third step is to perform feature point detection on the stable sample frame sequence and the unstable sample frame sequence respectively to generate a stable frame sample feature point information set and an unstable frame sample feature point information set, wherein the sample feature point description information includes the sample feature point coordinates.

[0091] The fourth step is to determine the first sample affine matrix between every two adjacent stable sample frames in the stable sample frame sequence according to the stable frame sample feature point information set, wherein the first sample affine matrix is ​​the true value label in the model training process.

[0092] The fifth step is to calculate the second sample affine matrix between each unstable sample frame in the unstable sample frame sequence and the corresponding stable sample frame according to the unstable frame sample feature point information set.

[0093] In the sixth step, each first sample affine matrix and the second sample affine matrix corresponding to the same frame are taken as sample pairs to obtain training samples.

[0094] Here, during the training of the pixel-level video stabilization model, the experimental platform is based on the Linux operating system platform with Python 3.6 and PyTorch 1.7.0 framework. The GAflow model uses GAflow-S, a lightweight GAflow model, as the Gaussian attention optical flow extraction model, where the Gaussian constraint layer (GCL) σ = 9. In the Gaussian guided attention module (GGAM), σ = 20.

[0095] GAFSF uses Adam as the optimizer. The parameters are set to β1=0.5 and β2=0.999. The initial learning rate lr=0.002. It is then reduced to half every 10 epochs. The batch size used during training is batchsize=8. The GAFSF network is trained for 60 epochs. GAFSF uniformly adjusts the frame size used to 640*360.

[0096] Optionally, the image stabilization training process can be: according to the needs of the network, after obtaining the height and width H, W of the frame and the number of frames n, scale the frame sequence to 256*256 size. First, input the unstable frame sequence into the Gaussian attention optical flow network to obtain the corresponding optical flow features. Then use the optical flow features and the unstable frame sequence as the input of the stable framework TUNet to obtain the distortion map of the corresponding frame. During this process, GAFSF still uses the input target frame and the frame sequence of the 15 frames before and after as a single input. GAFSF starts using optical flow in the second frame. The reasoning process of the first frame remains the same as before. The example is shown below:

[0097]

[0098] Optionally, define points in the frame according to their coordinates in the frame as Where i is the timestamp of the frame where the point is located. Then the movement of the point between frames is defined to be completed by the distortion mapping function Fi(·). Fi(·) means that the point is mapped using the i-th distortion map, then:

[0099]

[0100] Due to the addition of optical flow, GAFSF must ensure the effectiveness of this process as much as possible. Therefore, it is necessary to constrain this process, and for this purpose, motion loss is added to the original loss function:

[0101]

[0102] Among them, G(·) represents the use of real optical flow to change the point. Since the identification process of moving objects is not added to the training process, the training cost of the model is greatly reduced. The effective part of the real optical flow only contains all the information of the moving object. The calculation of the motion loss of GAFSF is actually only performed on the moving objects in the frame, that is:

[0103]

[0104] Among them, δ is the mask area.

[0105] Secondly, the content loss is used to guide the model towards the true value and minimize the difference between them. It is defined as follows:

[0106] L content =∑ i∈[t，t+1] MSE(S i -P i )+L VGG (S i , P i ).

[0107] Among them, S represents the basic facts of the stable frame. P is the predicted stable frame. MSE is the mean square error function. This paper uses the MSE loss function to penalize the deviation between the predicted value and the true value. The maximum pooling layer before the fully connected layer of the pre-trained VGG16 network is used to calculate the mean square loss in the LVGG(Si,Pi) feature space as follows:

[0108] L VGG (S i , P i )=MSE(VGG 16 (S i )-VGG16 (P i )).

[0109] Next, simple content loss alone is not enough to solve the above technical problems. Although the predicted frame can be close to the true value in a large frame, the pixel-based distorted image will cause serious distortion of some local details in the stable frame. For this reason, shape loss is defined to minimize artifacts, expressed as:

[0110] L shape =L point (γ,S i , P i )+L grid (S i , P i ).

[0111] Among them, Lpoint(γ,Si,Pi) calculates the loss between the feature points in the two frames, which is used to penalize some abnormal feature point changes after distortion. Lgrid(Si,Pi) represents the grid loss as follows:

[0112] L grid (S i , P i )=L relative (S i , P i )+L adjacent (S i , P i ).

[0113] In addition, the features between adjacent frames often do not change much. What needs to be paid attention to is whether the low-frequency motion between consecutive frames as the subject is smooth on the time scale. This is also the main goal of the image stabilization task in this paper. The definition of time loss is as follows:

[0114]

[0115] in is a function that warps the predicted frame Pi-1 to Pi. We use FlowNet2.0 to implement the distortion of the previous frame and extract the optical flow from the ground truth.

[0116] Finally, the total loss function of the generated network is:

[0117] L stable =L content +α·L shape +β·L temporal +γ·L motion .

[0118] Among them, α, β and γ are weight parameters of shape loss, time loss and motion loss, and their values ​​are 1, 8 and 4 respectively. Thus, the total loss value in the model training process is calculated.

[0119] The above steps 102-105 and various formulas and their related contents serve as an inventive point of an embodiment of the present disclosure, which solves the second technical problem mentioned in the background technology: "There are often multiple parts in the scene with inconsistent motion directions or local feature changes, and each block in the image has its own motion trajectory. If you consider converting the global estimate into a local block estimate, it will consume additional resources. Many methods have exposed their limitations, including but not limited to inaccurate optical flow estimation, local distortion and distortion, jitter of objects in the image, etc. Despite excellent indicator data, it still causes discomfort to the human eye." The factors that lead to the above problems are often as follows: Due to the limited motion patterns in the data set, the model cannot adapt to all motion patterns well. In some complex motion patterns (such as rapid scene changes, large-area moving objects in the frame sequence, etc.), the distortion map will have inaccurate estimation problems, resulting in poor perception of the stable frame sequence. Due to the lack of depth estimation, the model may have blurred contours near those objects in scenes with large depth of field and strong parallax. Since the stabilization method based on optical flow estimates the global affine transformation matrix and optimizes a single camera trajectory based on it, there are often multiple parts with inconsistent motion directions or local feature changes in the scene, and each block in the image has its own motion trajectory. If the above factors are solved, the accuracy of optical flow estimation can be improved. It can greatly avoid local image distortion and distortion. At the same time, it can reduce the jitter of objects in the image. In order to achieve this effect, firstly, based on the above analysis, this paper starts from optimizing the interaction between optical flow features and convolution and attention features and using high-performance optical flow models to complete the video stabilization task, and proposes a video stabilization model based on Gaussian attention optical flow. The model mainly includes three contributions: (1) In the actual scene with a variety of motion patterns, considering the role of optical flow in extracting motion information, the motion stabilization framework GAFSF (Gaussian Attention Optical Flow Stabilization Framework) based on Gaussian attention optical flow is introduced to extract and correct motion information. (2) In order to address the image distortion caused by slight estimation errors, a simple optical flow downsampling coupling unit FDCU (Flow Downsampling Coupling Unit) is introduced to process the optical flow information more finely, and then merged with the convolution module. Better use of optical flow features to improve model performance. (3) In order to address the local jitter that may be caused by moving objects, a unique motion loss function is designed to constrain moving objects in the frame. Finally, in order to make it more sensitive to slight estimation errors, an optical flow downsampling coupling unit FDCU (Flow Downsampling Coupling Unit) is designed in GAFSF to refine the processing of optical flow features.Since there is a Transformer branch in the stable network of GAFSF, in order to ensure the integrity of the optical flow information. GAFSF uses a Vit block and a convolution block to downsample the optical flow information and connect it. Specifically, since TUNet is a dual-branch structure, the three FDCUs in GAFSF also introduce a dual-branch structure to introduce optical flow information to TUNet. FDCU uses 12-head multi-head attention in the attention branch and uses the same 16×16 size patch for Patch Embendding. A separate 3×3 convolution block is used in the convolution branch for downsampling, and the number of channels is adjusted to the number of channels of the corresponding convolution module in TUNet to introduce optical flow features. Thereby, the accuracy of optical flow estimation is improved. In addition, local distortion and distortion of the image are greatly avoided. At the same time, the jitter of objects in the image is reduced.

[0120] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure, the problem of excessive cropping or local blur can be avoided. Specifically, the reason for the problem of excessive cropping or local blur is: the occlusion caused by the change in depth of field and the interference of the foreground object on the camera posture judgment. Their results usually contain a large number of missing areas at the frame boundaries, especially when there is violent motion in the video. Traditional algorithms still have significant defects when dealing with depth changes, complex camera movements and scene changes. For example, although the existing methods can achieve this conversion, they are still too rough and still have deficiencies in cropping and grasping local information. Based on this, the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure, first, by dividing the scene video into a stable video frame sequence and an unstable video frame sequence, it is easy to extract unstable image features in the video frame. Then, by introducing the Gaussian attention optical flow network, the Gaussian attention optical flow field image can be extracted. In this way, it is easy to use optical flow features instead of feature trajectories to deal with spatially changing motions, and use optical flow to directly learn stability. Next, by introducing the optical flow downsampling coupling layer, the optical flow features of the Gaussian attention optical flow field image can be extracted, and the optical flow features can be refined at the pixel level. At the same time, a more accurate stable distortion map is output based on the stable network. Finally, each unstable video frame can be corrected by the stable distortion map. Thanks to the accuracy of motion estimation by optical flow, the excessive cropping caused by the error of motion estimation is reduced, and the cropping rate is improved. The generalization and robustness of the model are improved in various scenarios. Thus, the video is stabilized at the pixel level based on Gaussian attention optical flow. This avoids the problem of excessive cropping or local blurring caused by overly rough processing, and obtains a stable video image.

[0121] Further references Figure 7As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a pixel-level video stabilization device based on Gaussian attention optical flow. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 7 As shown, a pixel-level video stabilization device 700 based on Gaussian attention optical flow in some embodiments includes: an acquisition unit 701, an adjustment unit 702, an image extraction unit 703, a feature extraction unit 704 and an image distortion processing unit 705. Among them, the acquisition unit 701 is configured to acquire the scene video captured by the head-mounted shooting device; the adjustment unit 702 is configured to adjust each video data frame in the above scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the above stable video frame sequence and the above unstable video frame sequence is the same as the number of frames in the above scene video; the image extraction unit 703 is configured to extract the Gaussian attention optical flow field of the current frame in the above unstable video frame sequence through a Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame; the feature extraction unit 704 is configured to extract the optical flow features of the Gaussian attention optical flow field image of the current frame through an optical flow downsampling coupling layer, and add the extracted optical flow features to the stable network to output a stable distortion map; the image distortion processing unit 705 is configured to use the above stable distortion map to perform distortion processing on each unstable video frame in the above unstable video frame sequence to generate a processed stable video frame sequence, and send the processed stable video frame sequence to the display terminal to complete video stabilization.

[0123] It is understood that the units described in the device 700 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 700 and the units included therein, and will not be described in detail here.

[0124] Reference below Figure 8 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 800 suitable for implementing some embodiments of the present disclosure. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0125] like Figure 8As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory 802 or a program loaded from a storage device 808 into a random access memory 803. Various programs and data required for the operation of the electronic device 800 are also stored in the random access memory 803. The processing device 801, the read-only memory 802, and the random access memory 803 are connected to each other via a bus 804. An input / output interface 805 is also connected to the bus 804.

[0126] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0127] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a read-only memory 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0128] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0129] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0130] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the scene video captured by the head-mounted camera; adjusts each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; extracts the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through the Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame; extracts the optical flow features of the Gaussian attention optical flow field image of the current frame through the optical flow downsampling coupling layer, and adds the extracted optical flow features to the stable network to output a stable distortion map; uses the stable distortion map to distort each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and sends the processed stable video frame sequence to the display terminal to complete video stabilization.

[0131] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor includes an acquisition unit, an adjustment unit, an image extraction unit, a feature extraction unit, and an image distortion processing unit. The names of these units do not constitute limitations on the units themselves in certain cases, for example, the image distortion processing unit may also be described as a "unit for performing distortion processing on each unstable video frame in an unstable video frame sequence".

[0134] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0135] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A pixel-level video stabilization method based on Gaussian attention optical flow, comprising: Obtain the scene video captured by the head-mounted camera device; Adjusting each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; Extracting the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through a Gaussian attention optical flow network to obtain a Gaussian attention optical flow field image of the current frame; Extracting optical flow features from the Gaussian attention optical flow field image of the current frame through an optical flow downsampling coupling layer, and adding the extracted optical flow features to a stabilization network to output a stable distortion map; Using the stable warp map, warping is performed on each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and the processed stable video frame sequence is sent to a display terminal to complete video stabilization.

2. The method according to claim 1, wherein: The Gaussian attention optical flow network includes: a motion backbone network, a Gaussian constraint layer, a feature processing module, a context backbone network, a Gaussian attention guidance module and a decoder; and The Gaussian attention optical flow field of the current frame in the unstable video frame sequence is extracted through the Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame, including: Extract motion features of unstable video frames corresponding to time t and time t+1 in the unstable video frame sequence through the motion backbone network to obtain a moving object mask of the current frame; Extracting context features corresponding to the unstable video frame at time t through the context backbone network and the convolutional network layer; Inputting the current frame moving object mask into a feature processing module combined with the Gaussian constraint layer to generate a feature map; Inputting the context feature and the feature map into the Gaussian attention guidance module to generate a Gaussian self-attention feature; The Gaussian self-attention feature is decoded by the decoder to generate a Gaussian attention optical flow field image of the current frame.

3. The method according to claim 2, wherein: The optical flow downsampling coupling layer is a double-branch structure, and the optical flow downsampling coupling layer includes a first network branch and a second network branch; as well as The optical flow feature extraction is performed on the Gaussian attention optical flow field image of the current frame through the optical flow downsampling coupling layer, including: Performing image segmentation on the current frame Gaussian attention optical flow field image to generate an optical flow field image block set; By using the first network branch, extracting features from each optical flow field image block in the optical flow field image block set to generate a multi-head attention feature; Through the second network branch, feature extraction is performed on the Gaussian attention optical flow field image of the current frame to generate normalized features; The multi-head attention features and the normalized features are determined as the extracted optical flow features.

4. The method according to claim 3, wherein: The step of adding the extracted optical flow features to a stable network to output a stable distortion map includes: Graying each unstable video frame in the unstable video frame sequence to generate a grayscale image sequence; The grayscale images in the grayscale image sequence are encoded by the encoding module in the stable network, and the optical flow features are respectively added to the three network layers in the encoding module in the stable network for joint encoding to generate multi-head attention encoding features and normalized encoding features; The multi-head attention encoding features and the normalized encoding features are decoded through a decoding module in the stable network to generate a stable distortion map.

5. The method according to claim 1, wherein: Before extracting the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through the Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame, the method further includes: Performing image resizing on each unstable video frame in the unstable video frame sequence to generate a target size video image sequence, wherein the image resizing includes adjusting a height value, a width value, and a number of channels of the image; Extracting feature points from the target size video image sequence to generate a feature point description information set; Normalizing the feature point coordinates included in each feature point description information in the feature point description information set to generate a processed feature point description information set; According to the processed feature point description information set, an affine matrix between each two adjacent frames of the target size video image sequence is determined as another input of the Gaussian attention optical flow network to generate a Gaussian attention optical flow field image of the current frame.

6. The method according to claim 4, wherein: The Gaussian attention optical flow network, the optical flow downsampling coupling layer and the stabilization network constitute a pixel-level video stabilization model, and the training samples of the pixel-level video stabilization model are generated by the following steps: Get a sample video frame sequence; Adjusting the sample video frame sequence into a stable sample frame sequence and an unstable sample frame sequence, and adjusting the size of each stable sample frame and each unstable sample frame; Performing feature point detection on the stable sample frame sequence and the unstable sample frame sequence respectively to generate a stable frame sample feature point information set and an unstable frame sample feature point information set, wherein the sample feature point description information includes sample feature point coordinates; Determine, according to the stable frame sample feature point information set, a first sample affine matrix between every two adjacent stable sample frames in the stable sample frame sequence, wherein the first sample affine matrix is ​​a true value label in the model training process; Calculating a second sample affine matrix from each unstable sample frame in the unstable sample frame sequence to a corresponding stable sample frame according to the unstable frame sample feature point information set; Each first sample affine matrix and a second sample affine matrix corresponding to the same frame are taken as a sample pair to obtain a training sample.

7. A pixel-level video stabilization device based on Gaussian attention optical flow, comprising: An acquisition unit, configured to acquire a scene video captured by a head-mounted camera device; an adjustment unit, configured to adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; An image extraction unit is configured to extract the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through a Gaussian attention optical flow network to obtain a Gaussian attention optical flow field image of the current frame; A feature extraction unit is configured to extract optical flow features from the Gaussian attention optical flow field image of the current frame through an optical flow downsampling coupling layer, and add the extracted optical flow features to a stabilization network to output a stabilized distortion map; The image distortion processing unit is configured to use the stable distortion map to perform distortion processing on each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and send the processed stable video frame sequence to a display terminal to complete video stabilization.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images