Online video anti-shake device, online video anti-shake method and learning method thereof

Abstract

The application discloses an online video anti-shake device, an online video anti-shake method and a learning method thereof, and belongs to the technical field of video processing. The learning method for video anti-shake comprises the following steps: acquiring training data; training a neural network model based on the training data; acquiring the training data comprises: acquiring a jitter video and a stable video; extracting first inter-frame motion of the jitter video; transforming each frame of the stable video based on the first inter-frame motion of the jitter video to obtain a processed video; and taking the stable video and the processed video as training data. The learning method migrates the motion of a jitter video to a stable video to synthesize an unstable video corresponding to the original stable video, and then takes the original stable video and the corresponding unstable video as the training data required by the video anti-shake method. The application does not need to specially synchronize shooting of the stable video and the jitter video, and the picture content can be irrelevant.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video processing, and particularly relates to an online video anti-shake device, an online video anti-shake method and a learning method thereof. BACKGROUND

[0002] Video stabilization aims to convert a shaky video into a satisfactory stabilized video by smoothing the camera trajectory, and has been widely applied in the fields of smart phones, drones and security. Video stabilization can be divided into three categories: mechanical stabilization, optical stabilization and digital stabilization. Mechanical stabilization usually uses sensors and mechanical structures to complete the task. Optical stabilization detects the angle and speed of movement through a set of lenses and sensors to achieve video stabilization. Digital stabilization technology does not use specific equipment and is only implemented in software, so digital video stabilization can be regarded as a problem in the field of video processing and computer vision. Because digital stabilization only relies on software algorithms, it not only saves costs and reduces the requirements for specific equipment, but is also the only method that can stabilize already recorded videos.

[0003] Digital video stabilization can consider two different environments: offline stabilization and online stabilization. In the offline case, information from all frames of the video can be used, so better results can be achieved, which is particularly important in the post-processing of recorded videos. In the online case, the stabilization of the video does not use future frames, and can stabilize in real time during video recording, so it is important for real-time streaming scenarios.

[0004] Traditional digital stabilization methods first detect feature points in video frames, then estimate a 2D transformation, such as homography, optical flow and mesh flow, or estimate a 3D camera pose as a representation of movement, and finally smooth the camera path formed by the movement to achieve video stabilization. In the traditional method, the anti-shake method based on deep learning uses a neural network model, such as a convolutional neural network model, to directly learn the mapping relationship from an unstable video to a stabilized video. However, the traditional method has the following disadvantages: 1. The traditional method is subject to feature algorithms, and may fail to detect and track features in low-quality videos, resulting in failed stabilization. 2. Although the deep learning method performs well in low-quality videos, it is very dependent on the quality and quantity of training data, and usually directly uses video frames as input, so it is also affected by the texture of the picture. 3. The deep learning training data for video stabilization are obtained by using two identical video recording devices to synchronously record stabilized and unstable video pairs with and without external mechanical auxiliary stabilization devices, which has the problems of high cost, low efficiency and path divergence. SUMMARY

[0005] The present application provides an online video anti-shake device, an online video anti-shake method and a learning method thereof, which can synthesize training data for a video anti-shake task without the need for double-machine shooting.

[0006] The present application is achieved by the following technical solutions:

[0007] In one aspect, the present application provides a learning method for video anti-shake, comprising the following steps: obtaining training data; training a neural network model based on the training data; obtaining the training data comprises: obtaining a jitter video and a stable video; extracting a first inter-frame motion of the jitter video; transforming each frame of the stable video based on the first inter-frame motion of the jitter video to obtain a processed video; and taking the stable video and the processed video as training data.

[0008] In some embodiments, the loss function of the neural network model to be trained is:

[0009] L = L MC + αL SC + βL SP

[0010] wherein, L MC is a motion consistency loss function, L SC is a shape consistency loss function, L SP is a scale preservation loss function, and α and β are balance parameters for balancing the contributions of the three loss functions.

[0011] In some embodiments, the motion consistency loss function is:

[0012]

[0013] wherein, B′ t and B′ t-1 represent the network-estimated transformation field maps of adjacent two frames, and represent the true values of the adjacent two frames transformation field maps;

[0014] The shape consistency loss function is:

[0015]

[0016] wherein, v i represents the i-th grid vertex, and N represents the total number of grid vertices;

[0017] The scale preservation loss function is:

[0018]

[0019] wherein s represents a scale factor.

[0020] In another aspect, the application provides a minimum-delay online video anti-shake method, comprising the following steps: acquiring an unstable frame in a video; extracting, by a preset neural network model, second inter-frame motion of the video formed by the unstable frame and continuous frames before the unstable frame; performing path smoothing on the unstable frame based on the second inter-frame motion and the neural network model after training to obtain a transformation field map; and resetting the unstable frame by the transformation field map.

[0021] In some embodiments, resetting the unstable frame by the transformation field map comprises the following steps: adjusting positions of all pixels on the unstable frame according to displacement vectors of all pixel points provided by the transformation field map to obtain a stable frame.

[0022] In some embodiments, the neural network model after training is a convolutional neural network model.

[0023] In some embodiments, the second inter-frame motion is represented in the form of a sparse grid; after extracting the second inter-frame motion of the video formed by the unstable frame and the continuous frames before the unstable frame, before performing path smoothing on the unstable frame based on the second inter-frame motion and the neural network model after training to obtain the transformation field map, the method comprises the following steps: processing input data of the convolutional neural network model: interpolating the sparse grid formed by the second inter-frame motion to obtain a flow field map; the flow field map comprises a channel dimension, a high dimension and a wide dimension; using a sliding window to sequentially splice the flow field map in the channel dimension to form the input data of the convolutional neural network model.

[0024] The application also provides a minimum-delay online video anti-shake device, comprising: a motion extraction device for extracting second inter-frame motion of a video; a path smoothing device for performing path smoothing on the video; a memory having a computer program stored thereon; and a processor for executing the computer program to implement the minimum-delay online video anti-shake method of any one of the above embodiments.

[0025] Compared with the prior art, the application has the following advantages and beneficial effects:

[0026] The learning method for video anti-shake provided by the application moves the motion of a shaking video to a stable video to synthesize an unstable video corresponding to the original stable video, and then uses the original stable video and the corresponding unstable video as training data required by the video anti-shake method. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings in the embodiments will be briefly introduced as follows. Obviously, the drawings below only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained from the drawings without creative labor.

[0028] Figure 1 The relationship between the motion of adjacent two frames and the transform field map in the video anti-shake method based on the deep learning method provided by the embodiments of the present application is shown in the schematic diagram.

[0029] Figure 2 The synthesis relationship diagram of processing video in the video anti-shake method based on the deep learning method provided by the embodiments of the present application is shown in the schematic diagram.

[0030] Figure 3 The flowchart of the video anti-shake method based on the deep learning method provided by the embodiments of the present application is shown in the schematic diagram.

[0031] Figure 4 The relationship between the motion of adjacent two frames and the transform field map in the loss function provided by the embodiments of the present application is shown in the schematic diagram.

[0032] Figure 5 The effect comparison diagram of the video anti-shake method based on the deep learning method provided by the embodiments of the present application is shown in the schematic diagram.

[0033] Figure 6 The path diagram of the existing double-camera shooting video is shown in the schematic diagram.

[0034] Figure 7 The path diagram of the video anti-shake method based on the deep learning method provided by the embodiments of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments.

[0036] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0037] In addition, the terms "horizontal", "vertical", and the like in the description of the present application do not mean that the components must be absolutely horizontal or vertical, but can be slightly inclined. For example, "horizontal" only means that it is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0038] In the description of the present application, it should also be noted that, unless otherwise specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood broadly, for example, they can be fixedly connected, or can be detachably connected, or integrally connected; can be mechanically connected, or can be electrically connected; can be directly connected, or can be indirectly connected through an intermediate medium, or can be connected inside two elements. For those skilled in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0039] The terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but can optionally include steps or modules not listed, or can optionally include other steps or modules inherent to the process, method, product or device.

[0040] In one aspect, the embodiments of the present application provide a learning method for video stabilization, comprising the following steps:

[0041] S10, obtaining training data. In S10, first, the first inter-frame motion of a jitter video is extracted using video motion estimation, and the motion is expressed in the form of a grid flow. Then, based on the first inter-frame motion, each frame of a stable video is transformed to obtain a new jitter video. The first inter-frame motion is distinguished from the second inter-frame motion in the following text. The first inter-frame motion refers to the inter-frame motion of a known jitter video obtained in the process of obtaining training data; the second inter-frame motion refers to the inter-frame motion of specified consecutive frames in a video to be processed in the process of video stabilization. The present method does not require special synchronization shooting of stable videos and jitter videos, and the picture content can be irrelevant.

[0042] S10 can specifically include the following steps:

[0043] S101, obtaining a jitter video V ust and a stable video V stb . Wherein, the jitter video V ust and the stable video V stb may be irrelevant, that is, the content of the jitter video V ust may be different from the stable video V stb .

[0044] S102, extracting the first inter-frame motion of the shaking video. In S102, a deep neural network model, such as the Deep MeshFlow method, can be used to estimate the shaking video V ust and stabilized video V stb The first inter-frame motion and

[0045] S103, based on the first inter-frame motion of the shaking video, stabilize the video V stb Each frame is transformed to obtain a new processed video V syn In S103, a first inter-frame motion of the shaking video is transferred to a stable video to synthesize a shaking video V ust The shaking effect of the picture and the main path are the same as the stable video V stb Keep consistent with the new processing video V syn For the convenience of explanation, Respectively represent the frames of these three videos, obtained through S103 Come to Transform to synthesize

[0046]

[0047] With the above settings, each stabilized video can be synthesized into a new processed video. A set of stabilized videos and the corresponding synthesized new processed videos can form a stabilized / shaky video pair that can be used for network training. Figure 2 , each video has the following relationship:

[0048]

[0049] because and has been pre-calculated, so It can be expressed as:

[0050]

[0051] In subsequent training, the path smoothing network will be As input, the output is Supervised training for ground truth.

[0052] S104: Use the stabilized video and its corresponding processed video as training data.

[0053] S20. Train the training data to obtain a neural network model.

[0054] In the deep learning method, the loss function used by the neural network model to be trained is mainly as follows:

[0055] Motion-consistency Loss:

[0056]

[0057] where B′ t and B′ t-1 represent the network-estimated transformation field maps of the adjacent two frames, and represent the true values of the adjacent two frame transformation field maps. The motion-consistency loss function is responsible for constraining the network to learn a reasonable de-shake result while maintaining the continuity between frames.

[0058] Shape-consistency Loss:

[0059]

[0060] where v i represents the i-th mesh vertex, represents different mesh vertices, please refer to Figure 4 , and N represents the total number of mesh vertices. The shape-consistency loss function constrains the output result of the convolutional neural network model to not deviate greatly from the general mesh shape, otherwise it will cause the result picture to appear distorted and distorted.

[0061] Scale-preserving Loss:

[0062]

[0063] where s represents the scale factor. Because we are converting the sparse motion in the form of a mesh into a dense flow field map and predicting a meshed transformation field map, a scale-preserving loss function needs to be introduced to ensure that the network can guarantee the consistency of the output result in this scale transformation.

[0064] Thus, the final total loss function is as follows:

[0065] L = L MC + αL SC + βL SP

[0066] where α and β are balance parameters used to balance the contributions of the three loss functions, and the values taken here can be 0.01.

[0067] On the other hand, the present application provides a video stabilization method based on the deep learning method in any of the above embodiments, first using a neural network model to estimate the second inter-frame motion of the input video, specifically, a deep neural network model can be used. Preferably, the neural network model to be trained can adopt a convolutional neural network model, and then use a sliding window to input the second inter-frame motion sequence of the input video into the convolutional neural network model with an attention mechanism for path smoothing, and output the transformation field map of the last frame of the sliding window, and finally use the transformation field map to transform the shape and position of the last frame in the window to achieve stabilization. Different motion estimation methods may express motion in different ways, so we design to convert the motion estimated by different methods into a unified dense flow field map according to the offset generated by their action on each pixel position, so as to solve the problem of inconsistent motion expression, and it is also naturally suitable as the input of the convolutional neural network model.

[0068] Specifically, the video stabilization method includes the following steps:

[0069] T10, obtaining unstable frames in the video. In T10, the unstable frames of the video can be directly captured by existing software. For example, the video recording device can capture the unstable frame I at time t. t For example.

[0070] T20, extract the unstable frame I including capturing the moment t through the preset neural network model t The continuous frames before the unstable frame form the second inter-frame motion of the video. The preset neural network model can be set to be the same as the deep neural network model in step S102, and then a fixed window is used to record I t The past r video frames {I t} r = t ,I t-1 ,…,I t-r > and use them to t For stabilization. Because the whole process does not require the use of I t Any future frame of I t After being captured, it can be stabilized and the result can be output, so it is a method with minimum delay. t Another deep neural network model can be responsible for this. The path smoothing network of this application only transforms the field map based on the estimated motion prediction:

[0071] {B′ t}=φ({F t ;θ})

[0072] where φ(·) represents the camera path smoothing network and θ represents the network parameters to be optimized. ​

[0073] T30, the second inter-frame motion is represented in the form of a sparse grid. The input data of the convolutional neural network model is processed: the input data of the convolutional neural network model is processed: the flow field map is obtained by interpolating the sparse grid formed by the second inter-frame motion; the flow field map includes a channel dimension, a high dimension and a wide dimension; the flow field map is spliced in the channel dimension in time sequence using a sliding window, forming the input data of the convolutional neural network model;

[0074] T40, based on the second inter-frame motion and the trained neural network model obtained in the learning method for video anti-shake, the unstable frame is path smoothed to obtain a transformation field map. In T40, the continuous flow field maps in the sliding window can be input into the convolutional neural network model with channel attention mechanism to estimate the transformation field map of the last frame in the sliding window. The convolutional neural network model used in the present application adds a channel attention mechanism to the jump connection part based on the structure of UNet, so that the network can set weights for flow field maps at different time sequences according to the motion pattern of the input sequence, improving the anti-shake effect.

[0075] T50, the unstable frame is reset through the transformation field map. In T50, the elements in the transformation field map estimated in T40 are one-to-one corresponding to the pixel points at the same position in the original frame, representing the displacement vector of the pixel from the position on the original frame to the position on the stable frame. According to the displacement vectors of all pixel points provided by the transformation field map, the positions of all pixels on the original frame can be adjusted to synthesize a stable frame I t ′.

[0076] The embodiment of the present application also provides a kind of minimum delay online video anti-shake device, comprising:

[0077] Motion extraction device, the second inter-frame motion of video is extracted;

[0078] Path smoothing device, path smoothing is carried out to video;

[0079] Memory, computer program is stored on it;

[0080] Processor, processor executes computer program, to realize the minimum delay online video anti-shake method of any one of the above embodiments.

[0081] In the above embodiment, by setting a special device responsible for extracting motion, and the neural network model of the additional device focuses on smoothing the path, the processing efficiency can be improved.

[0082] In a specific example, the training is supervised, and real transform field maps are needed. In the training stage, the flow field map sequence of two consecutive windows is needed to be input together, because the motion consistency loss function is a time sequence loss function, and the calculation of which needs the transform field map estimation results of two consecutive frames. The shape consistency loss function and the scale preservation loss function are to constrain the quality of single estimation results, and do not need special processing. In the inference stage, the loss function does not need to be calculated, and the flow field map sequence in the window is sequentially sent into the convolutional network in the order of window sliding.

[0083] The training process uses Adam as the optimizer, sets the initial learning rate to 1e-4, and does not use the weight decay strategy. We set the three parameters β1, β2 and ∈ of the optimizer to 0.9, 0.999 and 1e-8 respectively, and train a total of 100,000 iterations, which takes about 20 hours on 2 NVIDIA 1080Ti graphics cards.

[0084] Effect display:

[0085] Please refer to Figure 5 , Figure 5 The comparison of the method proposed in the application and the two existing online anti-shake methods is shown (columns 1 and 2: the other two methods; column 3: the method of the application; column 4: the original frame). It can be seen that the method of the application can obtain good anti-shake effect in different scenes (rotation, scaling, etc.), and can also avoid problems such as excessive clipping and distortion of the results.

[0086] Please refer to Figure 6 and Figure 7 , Figure 6 and Figure 7 The effect of the method proposed in the patent on the synthesis of a jitter video is shown, Figure 6 the path comparison of a video pair shot by two cameras, Figure 7 the path comparison of a video pair synthesized by the method of the application, the dotted line is the jitter video path, and the solid line is the stable video path. It can be seen that the method proposed in the application can synthesize high-quality training data samples, and will not produce divergence with the original stable video in the path.

[0087] The embodiment of the application also provides a computer storage medium, which stores a computer program, and the computer program is loaded by a processor to execute the video anti-shake method based on the deep learning method in any of the above embodiments.

[0088] The above is only a preferred embodiment of the application, and does not limit the application in any form. Any simple modification or equivalent change made according to the technical essence of the application to the above embodiment falls within the protection scope of the application.

Claims

1. A learning method for video stabilization, characterized in that: The following steps are involved: Get training data; Training a neural network model based on the training data; The acquiring of training data comprises: Get shaky videos and stable videos; Extracting a first inter-frame motion of the shaking video; transforming each frame of the stabilized video based on the first inter-frame motion of the shaky video to obtain a processed video; Using the stabilized video and the processed video as training data; When training a neural network model, a loss function is used to constrain the training process. The loss function of the neural network model to be trained is: L=L MC +αL SC +βL SP Among them, L MC is the motion consistency loss function, L SC is the shape consistency loss function, L SP is the scale-preserving loss function, α and β are balance parameters used to balance the contributions of the three loss functions; The motion consistency loss function is: in, and represents the transformation field map of two adjacent frames estimated by the network, and Represents the true value of the transformed field images of two adjacent frames; The shape consistency loss function is: Among them, v i represents the i-th mesh vertex, and N represents the total number of mesh vertices; The scale-preserving loss function is: Where S represents the scale factor.

2. A method for stabilizing online video with the lowest delay, characterized in that: The following steps are involved: Get unstable frames in the video; Extracting a second inter-frame motion of a video including an unstable frame and its preceding continuous frames through a preset neural network model; Based on the second inter-frame motion and the trained neural network model, performing path smoothing on the unstable frame to obtain a transformation field map; Resetting the unstable frame by using the transformed field image; The neural network model being trained is a convolutional neural network model; The second inter-frame motion is represented in the form of a sparse grid; After extracting the second inter-frame motion of the video formed by the unstable frame and the continuous frames preceding it, and before performing path smoothing on the unstable frame based on the second inter-frame motion and the trained neural network model to obtain a transformed field map, the method includes the following steps: Process the input data of the convolutional neural network model: Interpolating the sparse grid formed by the second inter-frame motion to obtain a flow field map; the flow field map includes a channel dimension, a high dimension, and a width dimension; The flow field graph is spliced ​​in the channel dimension in time sequence using a sliding window to form input data of the convolutional neural network model.

3. The method for minimizing online video stabilization according to claim 2, wherein: Resetting the unstable frame by using the transformed field image comprises the following steps: According to the displacement vectors of all pixels provided by the transformation field map, the positions of all pixels on the unstable frame are adjusted to obtain a stable frame.

4. A minimum-latency online video stabilization device, characterized in that: include: A motion extraction device extracts the motion between the second frames of the video; A path smoothing device for smoothing the video path; a memory having a computer program stored thereon; A processor, wherein the processor executes the computer program to implement the minimum delay online video stabilization method according to any one of claims 2 or 3.

Images