Pixel-level video stabilization method and device based on gaussian attention optical flow

Abstract

Embodiments of the present disclosure disclose a pixel-level video stabilization method and device based on Gaussian attention optical flow. A specific implementation of the method comprises: adjusting each video data frame in a scene video into a stable video frame sequence and an unstable video frame sequence; extracting a Gaussian attention optical flow field of a current frame in the unstable video frame sequence through a Gaussian attention optical flow network to obtain a current frame Gaussian attention optical flow field image; performing optical flow feature extraction on the current frame Gaussian attention optical flow field image through an optical flow down-sampling coupling layer, and adding the extracted optical flow feature to a stable network to output a stable distortion map; performing distortion processing on each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and sending the processed stable video frame sequence to a display terminal. The implementation can avoid the problem of excessive cropping or local blur caused by excessively rough processing, and obtain a stable video image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of video stabilization, and in particular to a pixel-level video stabilization method and device based on Gaussian attention optical flow. BACKGROUND

[0002] In today's digital age, video has become one of the main ways for people to obtain information, exchange ideas and record life. With the continuous development of digital photography technology, more and more people begin to use mobile phones, cameras and other devices for video shooting. The convenience and popularity of digital photography make video the main medium for people to record life and share experiences. Not only that, video has a wide range of applications in different fields, including social media, online education, medical imaging, etc. In these application scenarios, users have increasingly high requirements for video quality and stability. For example, in medical imaging, a stable video can provide more accurate diagnostic information; in social media, users prefer to share high-quality, smooth videos; in the context of unmanned aerial vehicles, the low video quality caused by shaking. However, due to the instability of handheld shooting, many videos shot by users have obvious shaking and shaking, which seriously affects the viewing effect. Video stabilization technology has thus emerged, and its research aims to solve the shaking problem in video through algorithmic means to improve the quality and stability of video. Due to this real demand, more and more experts and scholars have also devoted themselves to the research in this field.

[0003] Video stabilization processing is one of the frontiers of research in the field of computer vision and image processing, and also promotes the cross-fusion of related fields. The quality of the video directly affects the downstream tasks to some extent. For example, target tracking, action recognition, video rain and fog removal, video question and answer, etc. Traditional algorithms have been widely studied due to their speed and robustness, but they face the challenge of occlusion caused by depth changes and interference of foreground objects on camera pose determination. Their results usually contain a large number of missing areas at the frame boundary, especially when the video has violent motion. Traditional algorithms still have significant defects in dealing with depth changes, complex camera movements and scene changes. For example, although existing methods can achieve this transformation, they are still too rough and lack of understanding of cropping and local information, which may result in excessive cropping or local blurring.

[0004] The above information disclosed in this BACKGROUND section is only for the purpose of enhancing the understanding of the background of the present inventive concepts, and therefore, it can contain information that does not form the prior art that is already known in the country to those of ordinary skill in the art. SUMMARY

[0005] This summary of the disclosure is presented in a simplified form to introduce some concepts that will be described in more detail below in the detailed description. This summary of the disclosure is not intended to identify key or essential features of the claimed technology nor is it intended to be used to limit the scope of the claimed technology.

[0006] Some embodiments of the present disclosure propose a pixel-level video stabilization method and device based on Gaussian attention optical flow to solve one or more of the technical problems mentioned in the background section.

[0007] In a first aspect, some embodiments of the present disclosure provide a pixel-level video stabilization method based on Gaussian attention optical flow, which comprises: acquiring a scene video captured by a head-mounted shooting device; adjusting each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; extracting a Gaussian attention optical flow field of a current frame in the unstable video frame sequence through a Gaussian attention optical flow network to obtain a current frame Gaussian attention optical flow field image; performing optical flow feature extraction on the current frame Gaussian attention optical flow field image through an optical flow down-sampling coupling layer, and adding the extracted optical flow features to a stabilization network to output a stable warp map; using the stable warp map to perform warp processing on each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and sending the processed stable video frame sequence to a display terminal to complete video stabilization.

[0008] In a second aspect, some embodiments of the present disclosure provide a pixel-level video stabilization device based on Gaussian attention optical flow, which comprises: an acquisition unit configured to acquire a scene video captured by a head-mounted shooting device; an adjustment unit configured to adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; an image extraction unit configured to extract a Gaussian attention optical flow field of a current frame in the unstable video frame sequence through a Gaussian attention optical flow network to obtain a current frame Gaussian attention optical flow field image; a feature extraction unit configured to perform optical flow feature extraction on the current frame Gaussian attention optical flow field image through an optical flow down-sampling coupling layer, and add the extracted optical flow features to a stabilization network to output a stable warp map; and an image warp processing unit configured to use the stable warp map to perform warp processing on each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and send the processed stable video frame sequence to a display terminal to complete video stabilization.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0011] The above various embodiments of the present disclosure have the following beneficial effects: through the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure, the problems of excessive cropping or local blur can be avoided. Specifically, the reason for the problems of excessive cropping or local blur is that occlusion caused by depth changes and interference of foreground objects to camera pose determination. Their results usually contain a large number of missing areas at the frame boundary, especially when there is a violent motion in the video. Traditional algorithms still have significant defects when dealing with depth changes, complex camera movements and scene changes. For example, although the existing method can achieve such a transformation, it is still too rough and still has deficiencies in cropping and local information. Based on this, the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure, first, by dividing the scene video into stable video frame sequences and unstable video frame sequences, the unstable image features in the video frames can be extracted. Then, by introducing a Gaussian attention optical flow network, a Gaussian attention optical flow field image can be extracted. In this way, the optical flow features can be used to replace the feature trajectories to process the spatially varying motion, and the optical flow can be directly learned for stabilization. Next, by introducing a flow down-sampling coupling layer, the optical flow features of the Gaussian attention optical flow field image can be extracted, and the optical flow features can be refined at the pixel level. At the same time, a more accurate stable warping map can be output by the stabilization network. Finally, each unstable video frame can be corrected by the stable warping map. Thanks to the accuracy of motion estimation by optical flow, the excessive cropping caused by the error of motion estimation is reduced, and the cropping rate is improved. The generalization and robustness of the model in various scenes are improved. Thus, the pixel-level video stabilization is realized based on Gaussian attention optical flow. The problems of excessive cropping or local blur caused by rough processing are avoided, and stable video images are obtained. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals can represent the same or similar elements. It should be understood that the drawings are schematic and elements and features are not necessarily to scale.

[0013] Figure 1 is a flow chart of some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure;

[0014] Figure 2 is a Gaussian constraint attention schematic diagram of some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure;

[0015] Figure 3 is a motion stabilization framework overall structure schematic diagram of Gaussian attention optical flow of some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure;

[0016] Figure 4 is a Gaussian guided attention module schematic diagram of some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure;

[0017] Figure 5 is an overall structure schematic diagram of the optical flow down-sampling coupling layer of some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure;

[0018] Figure 6 is an overall structure schematic diagram of the pixel-level video stabilization model of some embodiments of the pixel-level video stabilization method based on Gaussian attention optical flow according to the present disclosure;

[0019] Figure 7 is a structure schematic diagram of some embodiments of the pixel-level video stabilization device based on Gaussian attention optical flow according to the present disclosure;

[0020] Figure 8 is a structure schematic diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so as to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0022] It should also be noted that, for ease of description, only parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0023] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0026] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Figure 1 A flowchart 100 is shown illustrating some embodiments of a pixel-level video stabilization method based on Gaussian attention optical flow according to this disclosure. This pixel-level video stabilization method based on Gaussian attention optical flow includes the following steps:

[0028] Step 101: Obtain the scene video captured by the head-mounted camera.

[0029] In some embodiments, the entity executing the pixel-level video stabilization method based on Gaussian attention optical flow can acquire scene video captured by a head-mounted camera via wired or wireless means. The head-mounted camera can be a head-mounted network camera. The scene video can be video captured from any scene.

[0030] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future wireless connection methods.

[0031] Step 102: Adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence.

[0032] In some embodiments, the execution subject can adjust each video data frame in the above-mentioned scene video into a stable video frame sequence and an unstable video frame sequence. Wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video. Wherein the stable video frame in the stable video frame sequence can be a frame with less overall change between the front and back frames of the video. Secondly, each video data frame in the scene video can be adjusted into an unstable video frame through image processing. The unstable video frame can be a frame with greater overall change between the two frames before and after the video. Here, the image processing method can include but is not limited to image rotation, image affine transformation, image scaling, etc. In addition, each unstable video frame corresponds to each stable video frame one by one.

[0033] Optionally, before the execution subject extracts the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through the Gaussian attention optical flow network to obtain the current frame Gaussian attention optical flow field image, the execution subject can further include the following steps:

[0034] Firstly, the image size of each unstable video frame in the unstable video frame sequence is adjusted to generate a target size video image sequence. Wherein the image size adjustment can include adjusting the height value, width value and channel number of the image.

[0035] Specifically, due to the addition of optical flow information, in order to perform subsequent training and calculation, the unstable video frame sequence is adjusted to the same size as the work in the third chapter, i.e. H=256, W=256, C=3.

[0036] Secondly, feature point extraction is performed on the target size video image sequence to generate a feature point description information set. Wherein the feature point extraction can be performed on the target size video image sequence through a preset extraction algorithm to generate a feature point description information set. Here, the RANSAC (Random Sample Consensus) random sampling consistency algorithm can be used to match the feature points between two frames to obtain the matched current frame feature point set and next frame feature point set.

[0037] As an example, the extraction algorithm can include but is not limited to at least one of the following: SIFT (Scale-invariant Feature Transform) algorithm, Surf (Speeded Up Robust Features) algorithm, harris corner detection, etc.

[0038] Thirdly, the coordinates of the feature points included in each feature point description information in the above feature point description information set are normalized to generate a set of processed feature point description information. In order to facilitate calculation and training, all coordinates are normalized to the interval [-1, 1].

[0039] Fourthly, according to the above set of processed feature point description information, the affine matrix between each adjacent two frames of the target size video image sequence is determined as another input of the Gaussian attention optical flow network for generating the current frame Gaussian attention optical flow field image.

[0040] As an example, first, the coordinate matrix A can be constructed according to the coordinates of the feature points as follows:

[0041]

[0042] Secondly, the feature point coordinate vector B is constructed:

[0043] B = [x1, x2,..., x n , y1, y2,..., y n ] T .

[0044] Then, the coordinate matrix A is taken as the dependent variable and the product of the radiation vector X is equal to B. Here, X can be expanded as:

[0045] X = [a1, a2, a3, a4, a5, a6] T .

[0046] Next, the affine matrix is calculated frame by frame from the unstable frame to the corresponding stable frame. Then, it is converted into the corresponding distortion map. Where A belongs to the 2x3 real matrix, that is, the deformed X.

[0047] In practice, in order to solve the technical problems mentioned in the background art, CNN and the synergistic effect of CNN and Transformer double branch network are often used. First of all, by verifying the effectiveness of UNet, the double branch fuses two kinds of feature information by using HAFM module, and the model respectively obtains performance improvement on the cropping rate, distortion score and stability score. This proves the effectiveness of the CNN and Transformer double branch. The benefits brought by the addition of the Transformer branch are due to the fact that the Transformer can constantly use the attention features in the frame. Thus, the extraction of motion information is more refined, and the temporal feature information in the unstable video is learned more reasonably.

[0048] But the above implementation still has some limitations, limited by the limitedness of the motion pattern in the data set, the model cannot well adapt to all motion patterns, in some complex motion patterns (such as fast scene change, large area moving object appears in the frame sequence, etc.), the distorted graph will appear inaccurate estimation problem, resulting in poor visual effect of the stable frame sequence. Due to the lack of depth estimation, the model may have blurred contours near objects in scenes with large depth of field and strong parallax. Since the image stabilization method based on optical flow estimates the global affine transformation matrix and optimizes the single camera trajectory based on it, there are often parts in the scene with inconsistent motion directions or local feature changes, and each block in the image has its own motion trajectory. If global estimation is converted to local block estimation, additional resources will be consumed. Many methods have exposed their limitations, including but not limited to inaccurate optical flow estimation, local distortion and distortion, and image object jitter, etc. Although there are excellent index data, it will still cause the human eye to feel uncomfortable. In view of the above technical problem two, the inventors decided to solve it by using the following scheme.

[0049] Step 103, extracting the Gaussian attention optical flow field of the current frame in the unstable video frame sequence by the Gaussian attention optical flow network to obtain a current frame Gaussian attention optical flow field image.

[0050] In some embodiments, the above execution subject can extract the Gaussian attention optical flow field of the current frame in the unstable video frame sequence by the Gaussian attention optical flow network to obtain a current frame Gaussian attention optical flow field image.

[0051] In some optional implementations of some embodiments, the Gaussian attention optical flow network can include a motion backbone network, a Gaussian constraint layer, a feature processing module, a context backbone network, a convolution network, a Gaussian attention guide module and a decoder. And the above execution subject extracts the Gaussian attention optical flow field of the current frame in the unstable video frame sequence by the Gaussian attention optical flow network to obtain a current frame Gaussian attention optical flow field image, which can include the following steps:

[0052] First, the motion feature of the unstable video frame corresponding to time t and time t+1 in the unstable video frame sequence is extracted by the motion backbone network to obtain the current frame motion object mask. The motion backbone network can be used to extract the motion feature between the adjacent two frames.

[0053] Second, the context feature corresponding to the unstable video frame at time t is extracted by the context backbone network and the convolution network layer. The context backbone network can be used to extract the context feature of the unstable video frame at time t (i.e. the current frame).

[0054] In the third step, the current frame motion object mask is input into a feature processing module combined with a Gaussian constrained layer to generate a feature map. The Gaussian constrained layer (GCL) can output the feature map in combination with a feature encoder.

[0055] In practice, the Gaussian constrained layer (GCL) is essentially used to obtain fine-grained structural information and local discriminative representation for feature matching. The structure is designed based on a standard Transformer block. Specifically, given a basic feature X, the formula is as follows:

[0056] X' = GCA(LN(T))+X

[0057] Y = FFN(LN(T'))+X'.

[0058] where LN(·) and FFN(·) represent layer normalization and feed-forward network in the Transformer block, respectively. GCA(·) represents Gaussian constrained attention. T represents the feature map. Y represents the motion feature, which can include a value vector V, a deformed Gaussian kernel, a Gaussian kernel bias, and a Gaussian kernel amplitude. As shown in Figure 2 the shape of the k x k deformed Gaussian kernel Glr is obtained by taking the dot product of the Gaussian distribution N(0, σ) and the amplitude. The Gaussian kernel here is obtained from the feature processing module. It is initialized as a standard Gaussian distribution and is learnable. Then, the query vector Q, the key vector K, and the value vector V of the motion feature are obtained. The product of Q and K is calculated to obtain the similarity matrix. After passing through the linear layer Linear, the product of the similarity matrix and the Gaussian kernel Glr is calculated. The resulting result vector is calculated after calculating the softmax value to obtain the Gaussian constrained attention score. The formula is as follows:

[0059]

[0060] where, in the inference process, Glr is used as a constraint mask to reorganize the weight of attention feature aggregation.

[0061] In the fourth step, the context feature and the feature map are input into the Gaussian attention guided module to generate a Gaussian self-attention feature. The Gaussian guided attention module (GGAM) uses a spatially deformed Gaussian kernel to smooth the motion feature. Because the kernel is rigid or fixed and clutter inevitably provides false guidance. Therefore, a bias and an amplitude are added to guide the attention calculation. As shown in Figure 3As shown, first, the value vector V (Value) of the motion feature Y, the bias matrix and the amplitude matrix are obtained through three convolution layers respectively. Then, the dot product of the amplitude matrix and the Gaussian distribution is calculated, and the result is input into the warp function together with the bias matrix to obtain the Gaussian kernel. Then, the similarity matrix of the context feature X is calculated according to the query vector Q (Query) and the key vector K (Key). The similarity matrix is input into the linear layer and then dot product operation is performed with the Gaussian kernel. Then, the result vector is dot product operation with the value vector V (Value) of the motion feature Y to obtain the Gaussian guided attention score, and the formula is:

[0062]

[0063] Where G is the Gaussian kernel with amplitude. is the bias. Warp(·) is the warp function.

[0064] As an example, the overall structure of the motion stable framework of the Gaussian attention optical flow can be as shown in Figure 4 .

[0065] Fifth, the Gaussian self-attention feature is decoded by the above-mentioned decoder to generate the current frame Gaussian attention optical flow field image.

[0066] Step 104, through the optical flow down-sampling coupling layer, the optical flow feature of the current frame Gaussian attention optical flow field image is extracted, and the extracted optical flow feature is added to the stable network to output the stable warp map.

[0067] In some embodiments, the above-mentioned execution body can extract the optical flow feature of the current frame Gaussian attention optical flow field image through the optical flow down-sampling coupling layer, and add the extracted optical flow feature to the stable network to output the stable warp map.

[0068] In some optional implementations of some embodiments, the above-mentioned execution body adds the extracted optical flow feature to the stable network to output the stable warp map, which can include the following steps:

[0069] First, each unstable video frame in the above-mentioned unstable video frame sequence is grayed to generate a gray image sequence.

[0070] Second, the gray image in the above-mentioned gray image sequence is encoded by the encoding module in the above-mentioned stable network, and the above-mentioned optical flow feature is added to the three network layers in the encoding module in the above-mentioned stable network for joint encoding to generate multi-head attention encoding features and normalized encoding features.

[0071] As an example, the overall structure of the optical flow down-sampling coupling layer (unit) is as shown in Figure 5As shown. A stabilizing network can play a role in smoothing motion during the process. TUNet (Transformer U-shaped Network) is used as the stabilizing network. Figure 6 The diagram shows the overall structure of a pixel-level video stabilization model. The TUNet branch consists of an encoder and a decoder. The resolution of the feature maps decreases with network depth, while the number of channels increases. The left side shows the feature extraction network (encoder), and the right side shows the feature fusion network (decoder). The encoder captures the contextual information of the image, while the decoder recovers details and generates predictions at the same resolution as the input image, enabling simultaneous modeling of local details and global context.

[0072] The TUNet model couples convolutional features in branches with the embedding features (FDCU, Flow Downsampling Coupling Unit) output from optical flow downsampling coupling layers (units) in the following way: convolutional modules are used along the branches to progressively integrate feature maps and patch embeddings interactively. To maintain a consistent embedding size, an asymmetric sampling stride strategy is employed during the interaction. Specifically, since the convolutional feature dimension is C×H×W (C, H, and W represent the number of channels, height, and width of the tensor, respectively), while the embedding shape is (T+1)×E (T and E represent the number of image embeddings and the embedding dimension, respectively), mutual conversion between two-dimensional and three-dimensional features is involved. Because the dimensionality of the embedding features remains constant while the convolutional features change during upsampling and downsampling, direct connection methods fail. To address this, upsampling and downsampling are used to connect the two, and different strides are used for upsampling and downsampling during feature transformation to control the convolution and deconvolution at each step, maintaining a consistent feature size through different strides. In terms of spatial dimension, different sizes of embedding features are used to ensure that the embedding dimension is the same after each sampling.

[0073] The third step involves using the decoding module in the stable network to decode the multi-head attention coding features and the normalized coding features to generate a stable distortion graph.

[0074] Among them, it can be like Figure 5 As shown, the decoding module of the TUNet model can output a stable twisted graph based on the multi-head attention encoding features and the normalized encoding features described above.

[0075] In practice, GAFSF obtains the moving object mask in the stable frame sequence through a semantic network, and then uses the mask to extract the optical flow of the moving objects in the stable frame sequence as the ground truth. The difference is that this process only applies to the proposed loss function and no longer requires determining whether the moving object is in motion.

[0076] The grayscale image sequence of the unstable frames of the two adjacent frames is then input into the TUNet for double-branch calculation:

[0077] T i-1，i = BN(Act(Conv(T i-1,i )))

[0078] T i-1，i = MLP(MHSA(T i-1，i )).

[0079] Secondly, optical flow information is added to the second to fourth modules:

[0080] T i-1，i = BN(Act(Conv(T i-1,i )))+BN(Act(Conv(G i )))

[0081] T i-1，i = MLP((MHSA(T i-1,i ))+MLP(MHSA(G i ))).

[0082] Then, the warped image Wi is obtained after the four modules:

[0083]

[0084] Finally, the final warped image Wi can be used to generate the final stable frame. Since the optical flow calculation process combines the two frames before and after, the final stable sequence is less than the original frame number by 1.

[0085] In step 105, the stable warped image is used to warp each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and the processed stable video frame sequence is sent to a display terminal to complete video stabilization.

[0086] In some embodiments, the above execution subject can use the above stable warped image to warp each unstable video frame in the above unstable video frame sequence to generate a processed stable video frame sequence, and send the above processed stable video frame sequence to a display terminal to complete video stabilization.

[0087] Optionally, the Gaussian attention optical flow network, the optical flow down-sampling coupling layer, and the stable network constitute a pixel-level video stabilization model, and the training sample of the pixel-level video stabilization model is generated by the following steps:

[0088] First, a sample video frame sequence is obtained.

[0089] Secondly, the sample video frame sequence is adjusted into a stable sample frame sequence and an unstable sample frame sequence, and the size of each stable sample frame and each unstable sample frame is adjusted.

[0090] Thirdly, feature point detection is performed on the stable sample frame sequence and the unstable sample frame sequence respectively to generate a stable frame sample feature point information set and an unstable frame sample feature point information set. The sample feature point description information includes sample feature point coordinates.

[0091] Fourthly, according to the stable frame sample feature point information set, a first sample affine matrix between each two adjacent stable sample frames in the stable sample frame sequence is determined. The first sample affine matrix is a true value label in the model training process.

[0092] Fifthly, according to the unstable frame sample feature point information set, a second sample affine matrix between each unstable sample frame and the corresponding stable sample frame in the unstable sample frame sequence is calculated.

[0093] Sixthly, each first sample affine matrix and the second sample affine matrix corresponding to the same frame are taken as a sample pair to obtain a training sample.

[0094] Here, in the pixel-level video stabilization model training process, the experimental platform is based on a Linux operating system platform of Python 3.6 and PyTorch 1.7.0 framework. The GAflow model uses GAflow-S, i.e., a lightweight GAflow model, as the extraction model of the Gaussian attention optical flow, wherein the Gaussian constraint layer (GCL) σ = 9. In the Gaussian guided attention module (GGAM), σ = 20.

[0095] The optimizer of the GAFSF uses Adam. The parameter settings are β1 = 0.5 and β2 = 0.999. The initial learning rate lr = 0.002. Then it is reduced to half of the original value every 10 epochs. The batch size used in training is batchsize = 8. The GAFSF network is trained for 60 epochs. The frame size used by the GAFSF is adjusted to 640*360.

[0096] Optionally, the stabilization training process can be: according to the needs of the network, after obtaining the height and width H, W of the frame and the number of frames n, the frame sequence is scaled to 256*256 size. First, the unstable frame sequence is input into the Gaussian attention optical flow network to obtain the corresponding optical flow features. Then, the optical flow features and the unstable frame sequence are taken as the input of the stable framework TUNet to obtain the corresponding frame distortion map. In this process, the GAFSF still uses the input target frame and the frame sequence of the previous and next 15 frames as the input for a single time. The GAFSF uses optical flow from the second frame. The inference process of the first frame remains the same as before. The following examples are shown as follows:

[0097]

[0098] Optionally, the points in the frame are defined as where i is the timestamp of the frame that the point is in. Then the movement of the inter-frame points is done by warping mapping function Fi(·). Fi(·) means using the i-th warping map to map the points, then we have:

[0099]

[0100] Due to the addition of optical flow, GAFSF wants to ensure the effectiveness of this process as much as possible. Therefore, it is necessary to constrain this process, and for this purpose, a motion loss is added to the original loss function:

[0101]

[0102] where G(·) means using the real optical flow to change the points. Since the process of distinguishing moving objects is not added to the training process, it greatly reduces the training overhead of the model. The effective part of the real optical flow only contains all the information of the moving objects. The motion loss of GAFSF is actually only for moving objects in the frame, that is:

[0103]

[0104] where δ is the mask area.

[0105] Secondly, the content loss is used to guide the model to tend to the true value and minimize the difference between them. Its definition is as follows:

[0106] L content =∑ i∈[t，t+1] MSE(S i -P i )+L VGG (S i ,P i )。

[0107] where, where S represents the basic fact of the stable frame. P is the predicted stable frame. MSE is the mean square error function. This paper uses the MSE loss function to punish the deviation between the predicted value and the true value. The maximum pooling layer before the fully connected layer of the pre-trained VGG16 network is used to calculate the mean square loss in the LVGG(Si, Pi) feature space as follows:

[0108] L VGG (S i ,P i )=MSE(VGG 16 (S i )-VGG16 (P i ))。

[0109] Next, simply by the content loss is not enough to solve the above technical problems. Although the predicted frame can be close to the true value in a large framework. But the pixel-based distortion image will appear serious distortion in some local details in stable frame. For this define shape loss to reduce artifacts as much as possible, expressed as:

[0110] L shape =L point (γ,S i ,P i )+L grid (S i ,P i )。

[0111] Where Lpoint(γ,Si,Pi) calculates the loss between feature points in two frames, used to punish some abnormal feature point changes after distortion. Lgrid(Si,Pi) represents the grid loss as follows:

[0112] L grid (S i ,P i )=L relative (S i ,P i )+L adjacent (S i ,P i )。

[0113] In addition, the features between adjacent frames tend to change little, and attention should be paid to whether the low-frequency motion between consecutive frames as the main body is smooth in the time scale. This is also the main goal of the image stabilization task in this paper. The definition of time loss is as follows:

[0114]

[0115] Where is the function of distorting the predicted frame Pi-1 to Pi. In this paper, FlowNet2.0 is used to realize the distortion of the previous frame, and the optical flow is extracted from the ground truth.

[0116] Finally, the total loss function of the generation network part is:

[0117] L stable =L content +α·L shape +β·L temporal +γ·L motion 。

[0118] wherein a, β and γ are weight parameters of shape loss, time loss and motion loss respectively, and the values are 1, 8, 4 respectively. Thus, the total loss value in the calculation model training process is calculated.

[0119] The steps 102-105 and the respective formulas and their related content are one of the invention points of the embodiments of the present disclosure, and solve the second technical problem mentioned in the background that there are often multiple motion directions inconsistent or local feature changes in the scene, and each block in the image has its own motion trajectory. If global estimation is converted to local sub-block estimation, additional resources will be consumed. Many methods have exposed their limitations, including but not limited to inaccurate optical flow estimation, local distortion and distortion, shaking of objects within the image, etc. Although there are excellent index data, it still causes the human eye to feel uncomfortable. The factors leading to the above problems are often as follows: limited by the limited motion patterns in the data set, the model cannot well adapt to all motion patterns. In some complex motion patterns (such as rapid scene changes, large-area moving objects in the frame sequence, etc.), the distorted graph will have inaccurate estimation problems, resulting in poor viewing experience of the stable frame sequence. Due to the lack of depth estimation, the model may have blurred contours near objects in scenes with large depth of field and strong parallax. Since the image stabilization method based on optical flow estimates the global affine transformation matrix and optimizes the single camera trajectory based on it, there are often multiple motion directions inconsistent or local feature changes in the scene, and each block in the image has its own motion trajectory. If the above factors are solved, the accuracy of optical flow estimation can be improved. Local image distortion and distortion can be greatly avoided. At the same time, the shaking phenomenon of objects within the image is reduced. In order to achieve this effect, first of all, based on the above analysis, this paper completes the video stabilization task from two aspects of optimizing the interaction between the optical flow features and the convolution and attention features and using high-performance optical flow models, and proposes a video stabilization model based on Gaussian attention optical flow. The model mainly includes three contributions: (1) In view of the actual scene with various motion patterns, considering the extraction of motion information by optical flow, a Gaussian attention optical flow-based motion stabilization framework GAFSF (Gaussian Attention optical Flow Stabilization Framework) is introduced to extract and correct the motion information. (2) In view of the image internal distortion caused by the subtle estimation error, a simple flow downsampling coupling unit FDCU (Flow Downsampling Coupling Unit) is introduced to process the flow information more finely, and then fused with the convolution module. The flow features are better utilized to improve the model performance. (3) In view of the local shaking that may be caused by moving objects, a unique motion loss function is designed to constrain the moving objects in the frame. Finally, in order to be more sensitive to subtle estimation errors, the flow downsampling coupling unit FDCU (Flow Downsampling Coupling Unit) is designed in GAFSF to refine the flow features.To ensure the integrity of the optical flow information, GAFSF uses a Vit block and a convolution block to downsample the optical flow information. Specifically, since TUNet is a double-branch structure, the three FDCUs in GAFSF also introduce a double-branch structure to introduce optical flow information for TUNet. The FDCU uses 12-headed attention in the attention branch, and uses the same 16x16 size patch for Patch Embendding. In the convolution branch, a separate 3x3 convolution block is used for downsampling, and the number of channels is adjusted to the number of channels of the corresponding convolution module in TUNet to introduce optical flow features for it. Thus, the accuracy of optical flow estimation is improved. Further, the image local distortion and distortion are greatly avoided. At the same time, the shaking phenomenon of the objects in the image is reduced.

[0120] The above various embodiments of the present disclosure have the following beneficial effects: the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure can avoid the problems of excessive cropping or local blur. Specifically, the reason for the problem of excessive cropping or local blur is that the occlusion caused by the change of depth and the interference of the foreground object to the camera pose determination. Their results usually contain a large number of missing areas at the frame boundary, especially when there is a violent motion in the video. Traditional algorithms still have significant defects when dealing with depth changes, complex camera movements and scene changes. For example, although the existing method can achieve such a transformation, it is still too rough and still has deficiencies in cropping and local information. Based on this, the pixel-level video stabilization method based on Gaussian attention optical flow of some embodiments of the present disclosure first divides the scene video into stable video frame sequences and unstable video frame sequences, which can facilitate the extraction of unstable image features in the video frame. Then, by introducing a Gaussian attention optical flow network, a Gaussian attention optical flow field image can be extracted. In this way, the optical flow features can be used instead of feature trajectories to process spatially varying motion, and the optical flow is directly learned for stabilization. Next, by introducing a flow downsampling coupling layer, the Gaussian attention optical flow field image can be used to extract optical flow features, and the optical flow features can be refined at the pixel level. At the same time, a more accurate stable distortion map is output by the stable network. Finally, each unstable video frame can be corrected by the stable distortion map. Due to the accuracy of the motion estimation of the optical flow, the excessive cropping caused by the error of the motion estimation is reduced, and the cropping rate is improved. The generalization and robustness of the model in various scenes are improved. Thus, the Gaussian attention optical flow is realized to stabilize the video at the pixel level. The problem of excessive cropping or local blur caused by too rough processing is avoided, and a stable video image is obtained.

[0121] Further reference is made to Figure 7As an implementation of the methods shown in the above figures, the disclosure provides some embodiments of a pixel-level video stabilization device based on Gaussian attention optical flow, which device embodiments correspond to those method embodiments shown in Figure 1 The device can be specifically applied in various electronic devices.

[0122] As shown in Figure 7 The pixel-level video stabilization device 700 based on Gaussian attention optical flow of some embodiments includes an acquisition unit 701, an adjustment unit 702, an image extraction unit 703, a feature extraction unit 704, and an image warping processing unit 705. The acquisition unit 701 is configured to acquire a scene video captured by a head-mounted shooting device. The adjustment unit 702 is configured to adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video. The image extraction unit 703 is configured to extract a Gaussian attention optical flow field of a current frame in the unstable video frame sequence through a Gaussian attention optical flow network, to obtain a current frame Gaussian attention optical flow field image. The feature extraction unit 704 is configured to perform optical flow feature extraction on the current frame Gaussian attention optical flow field image through an optical flow down-sampling coupling layer, and add the extracted optical flow features to a stabilization network to output a stable warping map. The image warping processing unit 705 is configured to warp each unstable video frame in the unstable video frame sequence using the stable warping map to generate a processed stable video frame sequence, and send the processed stable video frame sequence to a display terminal to complete video stabilization.

[0123] It can be understood that the units described in the device 700 correspond to the steps in the method described with reference to Figure 1 The operations, features, and beneficial effects described above for the method also apply to the device 700 and the units included therein, and will not be repeated here.

[0124] Reference is made below to Figure 8 which shows a structural schematic diagram of an electronic device (e.g., a computing device) 800 suitable for implementing some embodiments of the disclosure. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the disclosure.

[0125] As shown in Figure 8As shown, the electronic device 800 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801 that can perform various appropriate actions and processes according to programs stored in a read-only memory 802 or loaded from a storage device 808 into a random access memory 803. Various programs and data required for the operation of the electronic device 800 are also stored in the random access memory 803. The processing device 801, the read-only memory 802, and the random access memory 803 are connected to each other through a bus 804. An input / output interface 805 is also connected to the bus 804.

[0126] Generally, the following devices can be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication devices 809 can allow the electronic device 800 to communicate with other devices wirelessly or through wires to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that all of the illustrated devices are not required, and more or fewer devices can alternatively be implemented. Figure 8 Each block shown in the flowcharts can represent a device, or multiple devices, as necessary.

[0127] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 809, or installed from the storage devices 808, or installed from the read-only memory 802. When the computer program is executed by the processing device 801, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

[0128] Note that the computer readable medium in some embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus or device, or that can be used by or in connection with an instruction execution system, apparatus or device. In some embodiments of the present disclosure, the computer readable signal medium can include a computer readable program code propagated in or on a carrier medium, in which the computer readable program code is embodied. Such propagated computer readable program code can take many forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer readable medium can be transmitted using any suitable medium, including but not limited to, wire, cable, wireless, RF, infrared or any suitable combination of the foregoing.

[0129] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0130] The computer readable medium can be included in the electronic device, or exist separately from the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a scene video captured by a head-mounted camera; adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; extract a Gaussian attention optical flow field of a current frame in the unstable video frame sequence by a Gaussian attention optical flow network, to obtain a current frame Gaussian attention optical flow field image; perform optical flow feature extraction on the current frame Gaussian attention optical flow field image by an optical flow down-sampling coupling layer, and add the extracted optical flow feature to a stable network to output a stable distortion map; warp each unstable video frame in the unstable video frame sequence by using the stable distortion map to generate a processed stable video frame sequence, and send the processed stable video frame sequence to a display terminal to complete video stabilization.

[0131] Computer program code for carrying out operations of some embodiments of the disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0132] The computer program product of the first aspect can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to implement a method as described above; and instructions for causing a computer to operate based on a system as described above. The computer readable storage medium can include one or more of: a magnetic disk; a magnetic tape; a magneto-optical disk; a semiconductor memory (e.g., a RAM, a ROM, a flash memory, etc.); and an optical disk.

[0133] The units described in some embodiments of the present disclosure can be implemented by means of software, or by means of hardware. The described units can also be provided in a processor, for example, a processor can be described as comprising an acquisition unit, an adjustment unit, an image extraction unit, a feature extraction unit, and an image warping processing unit. In some cases, the names of these units do not constitute a limitation on the units themselves, for example, the image warping processing unit can also be described as "a unit for performing warping processing on each unstable video frame in the sequence of unstable video frames".

[0134] The functions described above in the detailed description can be performed by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0135] The above description is merely illustrative of the embodiments of the present disclosure and the principles of the technology employed. It is understood that modifications and variations of the disclosed embodiments are possible, and also that it is intended to cover such modifications and variations that are within the scope of the disclosure. It is also to be understood that specific details are not to be interpreted in flexing the scope of the application to the concrete embodiments described, but rather have been included for merely illustrative purposes.

Claims

1. A pixel-level video stabilization method based on Gaussian attention optical flow, comprising: Acquire scene videos captured by a head-mounted camera; Each video data frame in the scene video is adjusted into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; The Gaussian attention optical flow field of the current frame in the unstable video frame sequence is extracted using a Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame. Optical flow features are extracted from the current frame Gaussian attention optical flow field image through an optical flow downsampling coupling layer, and the extracted optical flow features are added to a stabilizing network to output a stable distortion map. Using the stable distortion map, each unstable video frame in the unstable video frame sequence is distorted to generate a processed stable video frame sequence, and the processed stable video frame sequence is sent to the display terminal to complete video stabilization. The optical flow downsampling coupling layer has a dual-branch structure, comprising a first network branch and a second network branch; and The step of extracting optical flow features from the current frame Gaussian attention optical flow field image through an optical flow downsampling coupling layer includes: The current frame Gaussian attention optical flow field image is divided into image blocks to generate an optical flow field image block set; Through the first network branch, feature extraction is performed on each optical flow field image patch in the optical flow field image patch set to generate multi-head attention features; The second network branch is used to extract features from the current frame Gaussian attention optical flow field image to generate normalized features; The multi-head attention features and the normalized features are determined as the extracted optical flow features.

2. The method according to claim 1, wherein, The Gaussian attention optical flow network includes: a motion backbone network, a Gaussian constraint layer, a feature processing module, a context backbone network, a Gaussian attention guidance module, and a decoder; and The Gaussian attention optical flow field of the current frame in the unstable video frame sequence is extracted using a Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame, including: Motion features are extracted from the unstable video frames at times t and t+1 in the unstable video frame sequence using the motion backbone network to obtain the moving object mask of the current frame. The context features corresponding to the unstable video frame at time t are extracted through the context backbone network and convolutional network layers. The mask of the moving object in the current frame is input into the feature processing module combined with the Gaussian constraint layer to generate a feature map; The context features and the feature mapping are input into the Gaussian attention guidance module to generate Gaussian self-attention features; The Gaussian self-attention features are decoded using the decoder to generate the Gaussian attention optical flow field image of the current frame.

3. The method according to claim 2, wherein, The step of adding the extracted optical flow features to the stabilizing network to output a stable distortion map includes: Each unstable video frame in the unstable video frame sequence is converted to grayscale to generate a grayscale image sequence; The grayscale images in the grayscale image sequence are encoded through the encoding module in the stable network, and the optical flow features are added to the three network layers in the encoding module of the stable network for joint encoding to generate multi-head attention encoding features and normalized encoding features. The multi-head attention coding features and the normalized coding features are decoded by the decoding module in the stable network to generate a stable distortion map.

4. The method according to claim 1, wherein, Before extracting the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through the Gaussian attention optical flow network to obtain the Gaussian attention optical flow field image of the current frame, the method further includes: The image size of each unstable video frame in the unstable video frame sequence is adjusted to generate a video image sequence of the target size. The image size adjustment includes adjusting the height value, width value and number of channels of the image. Feature points are extracted from the target-size video image sequence to generate a feature point description information set; The feature point coordinates included in each feature point description information in the feature point description information set are normalized to generate a processed feature point description information set. Based on the processed feature point description information set, the affine matrix between each two adjacent frames of the target size video image sequence is determined, and used as another input to the Gaussian attention optical flow network to generate the Gaussian attention optical flow field image of the current frame.

5. The method according to claim 3, wherein, The Gaussian attention optical flow network, the optical flow downsampling coupling layer, and the stabilization network constitute a pixel-level video stabilization model. The training samples for the pixel-level video stabilization model are generated through the following steps: Obtain the sample video frame sequence; The sample video frame sequence is adjusted into a stable sample frame sequence and an unstable sample frame sequence, and the size of each stable sample frame and each unstable sample frame is adjusted. Feature point detection is performed on the stable sample frame sequence and the unstable sample frame sequence respectively to generate a stable frame sample feature point information set and an unstable frame sample feature point information set, wherein the sample feature point description information includes the sample feature point coordinates; Based on the stable frame sample feature point information set, the first sample affine matrix between every two adjacent stable sample frames in the stable sample frame sequence is determined, wherein the first sample affine matrix is ​​the ground truth label during the model training process. Based on the unstable frame sample feature point information set, calculate the second sample affine matrix between each unstable sample frame and its corresponding stable sample frame in the unstable sample frame sequence; The training samples are obtained by taking each first sample affine matrix and the corresponding second sample affine matrix of the same frame as a sample pair.

6. A pixel-level video stabilization device based on Gaussian attention optical flow, comprising: The acquisition unit is configured to acquire scene video captured by the head-mounted camera. The adjustment unit is configured to adjust each video data frame in the scene video into a stable video frame sequence and an unstable video frame sequence, wherein the number of frames in the stable video frame sequence and the unstable video frame sequence is the same as the number of frames in the scene video; The image extraction unit is configured to extract the Gaussian attention optical flow field of the current frame in the unstable video frame sequence through a Gaussian attention optical flow network, and obtain the Gaussian attention optical flow field image of the current frame. The feature extraction unit is configured to extract optical flow features from the current frame Gaussian attention optical flow field image through an optical flow downsampling coupling layer, and to add the extracted optical flow features to a stabilizing network to output a stable distortion map. The image distortion processing unit is configured to use the stable distortion map to distort each unstable video frame in the unstable video frame sequence to generate a processed stable video frame sequence, and to send the processed stable video frame sequence to a display terminal to complete video stabilization. The optical flow downsampling coupling layer has a dual-branch structure, comprising a first network branch and a second network branch; and The step of extracting optical flow features from the current frame Gaussian attention optical flow field image through an optical flow downsampling coupling layer includes: The current frame Gaussian attention optical flow field image is divided into image blocks to generate an optical flow field image block set; Through the first network branch, feature extraction is performed on each optical flow field image patch in the optical flow field image patch set to generate multi-head attention features; The second network branch is used to extract features from the current frame Gaussian attention optical flow field image to generate normalized features; The multi-head attention features and the normalized features are determined as the extracted optical flow features.

7. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images