Improved video stabilization based on machine learning models

Abstract

Apparatus and methods related to stabilizing video content are provided. An exemplary method includes a mobile computing device receiving one or more image parameters associated with a video frame of a plurality of video frames. The method further includes receiving motion data associated with the video frame from a motion sensor of the mobile computing device. The method also includes predicting a stabilized version of the video frame by applying a neural network to the one or more image parameters and the motion data.

Description

[Background technology]

[0001] background Many modern computing devices, including mobile phones, personal computers, and tablets, include image capture devices such as video cameras. Some image capture devices and / or computing devices can correct or otherwise modify the captured image. For example, the camera or the subject may move during capture, causing the footage to appear blurry or distorted. Thus, some image capture devices can correct this blurring and / or distortion. After the captured image has been corrected, the corrected image can be stored, displayed, transmitted, and / or otherwise used. Summary of the Invention

[0002] overview

[0001] The present disclosure generally relates to stabilization of video content. In one aspect, an image capture device may be configured to stabilize input video. The image capture device may be configured to operate with a system of machine learning components to stabilize the video to remove distortions and other imperfections caused by unintentional vibration of the image capture device, motion blur caused by movement of objects in the video, and / or artifacts that may be introduced into the video image while the video is being captured.

[0003] In some aspects, a mobile device may be configured with these features to enable real-time enhancement of input video. In some cases, video may be automatically enhanced by the mobile device. In other aspects, a mobile phone user may non-destructively enhance video to suit their preferences. For example, video previously present in a user's video library may also be enhanced based on the techniques described herein.

[0004] In a first aspect, a computer-implemented method is provided, the method including receiving, by a mobile computing device, one or more image parameters associated with a video frame of a plurality of video frames, the method also including receiving, from a motion sensor of the mobile computing device, motion data associated with the video frame, and the method further including predicting a stabilized version of the video frame by applying a neural network to the one or more image parameters and the motion data.

[0005] In a second aspect, a device is provided, the device including one or more processors operable to perform operations including receiving, by a mobile computing device, one or more image parameters associated with a video frame of a plurality of video frames, the operations further including receiving, from a motion sensor of the mobile computing device, motion data associated with the video frame, and predicting a stabilized version of the video frame by applying a neural network to the one or more image parameters and the motion data.

[0006] In a third aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory computer-readable medium having stored thereon program instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations including receiving, by a mobile computing device, one or more image parameters associated with a video frame of a plurality of video frames. The operations further include receiving, from a motion sensor of the mobile computing device, motion data associated with the video frame. The operations also include predicting a stabilized version of the video frame by applying a neural network to the one or more image parameters and the motion data.

[0007] In a fourth aspect, a system is provided that includes means, by a mobile computing device, for receiving one or more image parameters associated with a video frame of a plurality of video frames, means for receiving motion data associated with the video frame from a motion sensor of the mobile computing device, and means for predicting a stabilized version of the video frame by applying a neural network to the one or more image parameters and the motion data.

[0008] Other aspects, embodiments and implementations will become apparent to those skilled in the art upon reading the following detailed description, where appropriate with reference to the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a neural network for video stabilization according to an exemplary embodiment. [Figure 2] FIG. 10 illustrates another neural network for video stabilization according to an exemplary embodiment. [Figure 3]FIG. 1 illustrates a Long Short Term Memory (LSTM) network for video stabilization according to an exemplary embodiment. [Figure 4] FIG. 1 illustrates a deep neural network for video stabilization according to an exemplary embodiment. [Figure 5] FIG. 2 illustrates an exemplary optical flow according to an exemplary embodiment. [Figure 6] FIG. 1 illustrates the training and inference phases of a machine learning model, according to an exemplary embodiment. [Figure 7] FIG. 1 illustrates a distributed computing architecture according to an exemplary embodiment. [Figure 8] FIG. 1 is a block diagram of a computing device according to an exemplary embodiment. [Figure 9] FIG. 1 illustrates a network of computing clusters arranged as a cloud-based server system, according to an exemplary embodiment. [Figure 10] 1 is a flowchart of a method according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Detailed Description Exemplary methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or features. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0011] Accordingly, the exemplary embodiments described herein are not intended to be limiting. The aspects of the present disclosure, as generally described herein and illustrated in the Figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0012] Furthermore, unless the context suggests otherwise, features shown in each of the drawings may be used in combination with one another. Thus, the drawings should generally be viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are required for each embodiment.

[0013] I. Overview This application relates to video stabilization using machine learning techniques, such as, but not limited to, neural network techniques. When a user of a mobile computing device captures video, the resulting image is not always smooth and / or stable. Sometimes, this can be caused by unintentional shaking of the user's hands. For example, when video is captured from a moving vehicle or while walking and / or running, the camera may vibrate, and the resulting video image will not appear stable. Therefore, image processing-related technical problems related to video stabilization arise.

[0014] To remove unwanted motion during image capture, some techniques apply models based on convolutional neural networks to stabilize the captured video. In some examples, motion data can be combined with optical image stabilization (OIS) data to output stabilized video. Such techniques are generally fast and can be efficiently executed on mobile devices. Also, in some examples, because image data is not used, such techniques can be robust to possible scene and lighting changes.

[0015] Image-based techniques are available as desktop applications for post-editing video. These techniques generally require more computational power because they involve feature extraction from images, optical flow extraction, and global optimization. Existing neural network-based techniques involve taking image frames as input and inferring a warping grid as output to generate stabilized video. However, image distortions may exist due to the lack of rigidity control of the warping grid.

[0016] The techniques described herein may include aspects of image-based techniques combined with techniques based on motion data and optical image stabilization (OIS) data. A neural network, such as a convolutional neural network, can be trained and applied to perform one or more aspects described herein. In some examples, the neural network can be configured as an encoder / decoder neural network.

[0017] In one example, a deep neural network (DNN) has a U-net structure. The DNN takes one or more video frames as input to the encoder and converts the data into a low-dimensional latent space representation. In some implementations, the latent space representation is based on the real camera pose. For example, the DNN determines the real camera pose from motion data, which is added to the latent space representation. The DNN uses the latent space representation to infer a virtual camera pose. In some implementations, a long-short-term memory (LSTM) unit can be used to infer the virtual camera pose. The virtual camera pose includes rotational and / or translational information. The DNN then uses the virtual camera pose to generate a warping grid for video stabilization. In some implementations, a long-short-term memory (LSTM) unit can be used to generate the warping grid. Also, to train the DNN, real camera pose history (including real camera poses for past, current, and future video frames) and virtual camera pose history (including virtual camera poses for past and current video frames) can be added to the latent space representation. In some embodiments, a warping grid can be applied to the predicted virtual camera pose to output a stabilized version. Thus, the trained neural network can process input video and predict stabilized video.

[0018] In one example, (a copy of) the trained neural network may reside on a mobile computing device. The mobile computing device may include a camera capable of capturing input video. A user of the mobile computing device may view the input video and determine that the input video should be stabilized. The user may then provide the input video and motion data to the trained neural network residing on the mobile computing device. In response, the trained neural network may generate a predicted output indicative of the stabilized video and then output the output video (e.g., provide the output video for display by the mobile computing device). In another example, the trained neural network does not reside on the mobile computing device; rather, the mobile computing device provides the input video and motion data to a remotely located trained neural network (e.g., via the Internet or another data network). The remotely located convolutional neural network may process the input video and motion data as described above to provide an output video indicative of the stabilized video. In another example, a non-mobile computing device may also use the trained neural network to stabilize video, including video not captured by the computing device's camera.

[0019] In some examples, the trained neural network may work in conjunction with other neural networks (or other software) and / or may be trained to recognize whether the input video is not stable and / or smooth. Then, upon determining that the input video is not stable and / or smooth, the trained neural network described herein may stabilize the input video.

[0020] Thus, the techniques described herein can enhance videos by stabilizing the image, thereby improving their actual and / or perceived quality. Improving the actual and / or perceived quality of videos can provide user experience benefits. These techniques are flexible and can be applied to a wide variety of videos in both indoor and outdoor settings.

[0021] II. Techniques for video stabilization using neural networks FIG. 1 illustrates a neural network 100 for video stabilization according to an example embodiment. The neural network 100 may include an encoder 115 and a decoder 130. A mobile computing device may receive one or more image parameters associated with a video frame of a plurality of video frames. For example, an input video 110 may include a plurality of video frames. Each video frame of the plurality of video frames may be associated with one or more image parameters. For example, each frame may be associated with image parameters such as frame metadata including exposure time, lens position, etc. In some embodiments, the image parameters of successive frames of the input video 110 may be utilized to generate an optical flow. For example, given a pair of video frames, a dense pixel-by-pixel optical flow may be generated. The optical flow provides a correspondence between two consecutive frames and indicates image motion from one frame to the next. The one or more image parameters and / or the optical flow may be input to the encoder 115.

[0022] In some embodiments, the mobile computing device may receive motion data 125 associated with the input video 110. For example, a motion sensor may maintain a log of timestamp data associated with each video frame. Also, for example, the motion sensor may capture motion data 125 that tracks the actual camera pose for each video frame. The term "pose," as used herein, generally refers to the rotation of an image capture device, such as a video camera. In some embodiments, the term "pose" may also include the lens offset of the image capture device. In some exemplary embodiments, the actual camera pose may be captured at a high frequency, such as 200 Hertz (Hz). The motion sensor may be a gyroscope device configured to capture a gyroscope signal associated with the input video 110. Therefore, the actual camera pose can be inferred with high accuracy based on the gyroscope signal. Also, each video frame may be associated with a timestamp. Therefore, past and future video frames and their respective rotations can be determined with reference to the current video frame.

[0023] The neural network 100 may be applied to one or more image parameters and motion data to predict a stabilized version of an input video 110. For example, the encoder 115 may generate a latent space representation 120 based on the one or more image parameters. Motion data (e.g., the actual camera pose) may also be input to the latent space representation 120. The decoder 130 utilizes the latent space representation 120 to predict the stabilized version. In this manner, a predicted output video 135 may be generated for each frame. Unlike the training phase, during the runtime phase, stabilization of video frames is performed in real time. Therefore, long sequences of video frames are not required during the runtime phase.

[0024] FIG. 2 illustrates another neural network 200 for video stabilization, according to an exemplary embodiment. Motion data 205 represents data from a motion sensor. In some embodiments, the motion sensor may be a gyroscope. Generally, a mobile device may be equipped with a gyroscope, and gyroscope signals may be captured from the mobile device. A gyro event handler in the mobile device may continuously fetch the gyroscope signals to estimate the real camera pose R(t). The gyroscope signals may be received at a high frequency (e.g., 200 Hz). The motion data 205 may include angular velocity and a timestamp, indicating the real camera rotation at a given time.

[0025] In some embodiments, the mobile device may be configured with an OIS lens shift handler that may be configured to read OIS movement along the horizontal x-axis or the vertical y-axis from the motion data 205. The OIS data may be sampled at a high frequency (e.g., 200 Hz), which may provide translation in the x and y directions. This may be modeled as an offset of the camera's principal axis. In some embodiments, the OIS readout information may not be included, such that only camera rotation is utilized to train the neural network 200. For example, each RGB frame includes motion data 205 that indicates rotation (e.g., hand movement) and translation (e.g., OIS movement). Therefore, the motion data 205 that indicates translation may be filtered out.

[0026] In other examples, both rotation and translation can be utilized. Translation occurs in the x and y axes. The OIS lens shift handler can be configured to continuously fetch the OIS readouts and convert them into 2D pixel offsets in pixels, as shown by:

[0027]

number

[0028] In the formula, O len (t) is the OIS lens offset at time t, which is the horizontal offset O along the x-axis. len (x,t) and the vertical offset O along the y axis len Includes (y,t).

[0029] In some embodiments, the mobile device may include a motion model constructor that constructs a projection matrix. Given an input video frame, the associated frame metadata 210 may include the exposure time and lens position for each scanline. The motion model constructor takes the exposure time, lens position, real camera pose, and OIS lens offset to construct a projection matrix P that maps the real-world scene to the image. i,j where i is the frame index and j is the scanline index.

[0030] In this description, the subscript "r" stands for "real" and the subscript "v" stands for "virtual." As described, a camera pose may generally include two components: rotation and translation. The real camera pose V at time T is r (T) can be expressed as follows:

[0031]

number

[0032] In the formula, R r (T) is the extrinsic matrix (rotation matrix) of the camera (for example, the camera on a mobile device), and O r (T) is the 2D lens offset of the principal point Pt, and T is the timestamp of the current video frame. The projection matrix is ​​P r (T)=K r (T)*R r (T), where K r (T) is the intrinsic matrix of the camera, given by:

[0033]

number

[0034] where f is the focal length of the camera lens, and Pt is a two-dimensional (2D) principal point that can be set at the center of the image of the current video frame at time T. Therefore, a three-dimensional (3D) point X can be calculated by x=P r (T)X, where x is a 2D homogeneous coordinate in image space. In some embodiments, OIS data indicating translation may not be used. In such cases, the camera intrinsic matrix can be determined as follows:

[0035]

number

[0036] Actual pose history 215 includes the actual camera poses in past, present and future video frames.

[0037]

number

[0038] where T is the timestamp of the current frame and N is the number of previous video frames. Also, the virtual pose history 230 includes the virtual camera poses for the past M video frames, as predicted by the deep neural network (DNN) 220.

[0039]

number

[0040] where M is the length of the virtual pose history. In some example implementations, a value of M=2 may be used. A fixed timestamp gap g (e.g., g=33 milliseconds) may be used to make the process invariant to the video frame rate measured in frames per second (FPS). In some example implementations, real camera pose history 215 may include real camera pose information for 21 video frames, including the current video frame, 10 previous video frames, and 10 future video frames. Virtual camera pose history 230 may include virtual camera pose information for the current video frame and one or more past video frames, because the virtual poses for future video frames are typically not known at runtime. In some implementations, the number of past video frames used in real camera pose history 215 and virtual camera pose history 230 may be the same. Real camera pose history 215 and virtual camera pose history 230 may be concatenated to generate a concatenated feature vector 235.

[0041] The DNN 220 takes as input the concatenated vector 235 and calculates the rotation R of the virtual camera pose corresponding to the video frame with timestamp T. v (T). The DNN 220 may generate a latent space representation as described below.

[0042] Actual camera pose V r (T) and the virtual camera pose V v Given (T), P r (T) and P v We can calculate two projection matrices, denoted as (T), for the 2D real camera domain x r to 2D virtual camera domain x v The mapping to can be found as follows:

[0043]

number

[0044] Real to virtual projection matrix P real to virtual is shown as follows:

[0045]

number

[0046] In the formula, A -1 represents the inverse of matrix A. Here, K v (T) is the intrinsic matrix of the camera corresponding to the virtual camera pose, and R v (T) is the predicted rotation of the virtual camera pose, and K r (T) is the intrinsic matrix of the camera corresponding to the real camera pose, and R r (T) is the rotation of the real camera pose. This is a 2D to 2D mapping that can be used to map the real camera image to the virtual camera image. As shown in Equation 8, we can calculate the inverse of the projection of the 2D real points to get the inverse projection map P of the real camera projection. r -1 (T) to obtain a point in 3D space, and then a projection map P of the virtual camera projection v (T) to project points in this 3D space back into 2D space. The rotation map R can be represented in several ways, for example, as a 3x3 matrix, a 1x4 quaternion, or a 1x3 axis angle. These different representations are equivalent and can be chosen based on the context. For example, a 3x3 matrix representation can be used to project points in this 3D space back into 2D space using a projection matrix P real to virtual However, a quaternion or axis-angle representation may be used to input the camera pose history into the DNN 220. These representations can be converted from one to the other and are equivalent.

[0047] In some embodiments, real pose history 215 and virtual pose history 230 may include OIS lens shift data, and the deep neural network may output a translation 225 corresponding to the predicted lens shift of the virtual camera. The predicted rotation and / or translation 225 may be added to virtual pose history 230. For example, the predicted rotation and / or translation 225 may also be provided to image warping grid 240. Image warping grid 240 may load the output from DNN 200 and generate output video 245 by mapping each pixel in the input frame to the output frame.

[0048] The DNN 220 may be trained based on a loss function L such as:

[0049]

number

[0050] In the training phase of the DNN 220, the virtual pose history 230 may be initialized with a virtual cue without rotation. N consecutive randomly selected video frames may be input along with the real pose history 215. A concatenated vector 235 for the real pose history 215 and the virtual pose history 230 may be input to the DNN 220. The output for each video frame of the input is a virtual rotation 225. The virtual rotation 225 may be fed back to the virtual pose history 230 to update the initial cue. The overall loss shown by Equation 9 may be backpropagated for each video frame. During the inference phase, a video frame sequence may be input, and a stabilized output video 245 corresponding to the input video may be obtained.

[0051] FIG. 3 illustrates a long-short-term memory (LSTM) network 300 for video stabilization, according to an example embodiment. One or more aspects of the architecture of network 300 may be similar to aspects of network 200 of FIG. 2. For example, motion data 205 and frame metadata 210 may be processed to generate real pose history 315. To initialize the process, a virtual pose history 330 with identity rotation may be generated. Real pose history 315 and virtual pose history 330 may be input to DNN 220 of FIG. 2.

[0052] As shown in FIG. 3, the DNN 220 may include an LSTM component 320. The LSTM component 320 is a recurrent neural network (RNN) that models long-range dependencies in a temporal sequence, such as a sequence of time-stamped video frames. Generally, the LSTM component 320 includes memory blocks in a recurrent hidden layer. Each memory block includes memory cells that store the temporal state of the network and one or more logic gates that control the flow of information. The LSTM component 320 computes a mapping from the input concatenated feature vector 335 to output a virtual pose 325.

[0053] In some embodiments, the method includes determining the relative rotation of the camera pose in the video frame relative to the reference camera pose in the reference video frame from the rotation data and the timestamp data, and the prediction of the stabilized version is based on this relative rotation. For example, instead of inputting absolute rotations into the LSTM component 320, the absolute rotations are converted into relative rotations or changes in rotation. This is based on the observation that absolute rotations may not be the same for similar types of motion because they may depend on when the rotation is initialized, i.e., where the origin is. Relative rotations, on the other hand, maintain similarity. For example, for illustrative purposes, consider 1D samples (1,2,3) and (4,5,6). These two samples are not the same. However, the relative changes of these samples can be determined. For example, consider the element-by-element differences in (1,2,3) relative to the first element "1." These differences are 1-1=0, 2-1=1, and 3-1=2. Therefore, the relative rotation vector can be determined to be (0,1,2). Similarly, consider the element-by-element differences in (4,5,6) relative to the first element, "4." These differences are 4 - 4 = 0, 5 - 4 = 1, and 6 - 4 = 2. Therefore, we can determine that the relative rotation vector is still (0,1,2). Thus, although the absolute rotations were different, the relative rotations are the same. In this way, the input more faithfully represents a similar set of movements, and much less training data is required.

[0054] The LSTM component 320 is the previous virtual pose R v Virtual rotation change dR relative to (T-1) v (T) and the frame center O v =(o′ x ,o′ y ) and the virtual lens offset for the camera pose (R v (T),O v (T)).

[0055]

number

[0056] As described herein, the LSTM component 320 outputs a rotation and / or translation 325. However, instead of an absolute rotation, a relative rotation is predicted. The output is the change in rotation multiplied by the previous rotation of the virtual camera pose of the previous frame. Thus, the virtual camera pose of the current frame can be determined as a multiple of the delta virtual pose of the current frame and the virtual pose of the previous frame. Thus, the LSTM component 320 calculates the dV t The virtual pose of the video frame corresponding to time t is calculated based on the virtual pose of the video frame corresponding to time t−1 and the relative rotation of the virtual pose output by the LSTM component 320, as follows: V t =dV t *V t-1 can be determined as:

[0057] Also, for the virtual lens offset or translation, this can be inferred from the LSTM component 320, or the lens offset can be set to (0,0). Thus, the lens position can be fixed at the primary center and only rotation can be used to stabilize the video frame.

[0058]

number

[0059] In some embodiments, a neural network may be trained to receive a particular video frame and output a stabilized version of the particular video frame based on one or more image parameters and motion data associated with the particular video frame. For example, during the training phase in network 300, virtual pose history 330 may be initialized with a virtual cue having an identity rotation. N consecutive randomly selected video frames may be input along with real pose history 315, which includes relative rotations. A concatenated vector 335 for real pose history 315 and virtual pose history 330 may be input to LSTM component 320. The output for each video frame of the input is a virtual relative rotation 325. The virtual relative rotation 325 may be added back to virtual pose history 330 to update the initial cue. During the inference phase, a video frame sequence may be input, and a stabilized output 345 corresponding to the input may be obtained. Image loss may also be determined, as described in more detail below. Furthermore, OIS data with translation may be used with lens offset, and relative rotation and translation 325 may include relative rotation and lens offset for the virtual camera. Also, multi-stage training may be performed, for example, as described in more detail below. In some example implementations, a modified version of the LSTM component 320 may be used. For example, the LSTM component 320 may be a deep LSTM RNN obtained by stacking multiple layers of the LSTM component 320.

[0060] 4 illustrates a deep neural network (DNN) 400 for video stabilization according to an exemplary embodiment. One or more aspects of the DNN 400 may be similar to those of the networks 200 and 300. An input video 405 may include multiple video frames. An optical flow 410 may be generated from the input video 405. For example, an optical flow extractor (e.g., on a mobile device) may be configured to extract the optical flow 410. In general, given a pair of consecutive video frames, a dense pixel-by-pixel optical flow 410 can be computed. The optical flow 410 provides a correspondence between the two frames and may be used as input to the DNN 400 for video stabilization.

[0061] 5 is a diagram illustrating an exemplary optical flow according to an exemplary embodiment. Two consecutive video frames are shown: a first frame 505 corresponding to a timestamp of time t and a second frame 510 corresponding to a timestamp of time t+1. For reference, an RGB spectrum 515 is shown. An optical flow 520 may be generated from the first frame 505 and the second frame 510. The optical flow 520 may be generated from the consecutive video frames in both forward and backward directions, e.g., from frame t to t+1 and from frame t+1 to t.

[0062] 4, the optical flow 410 may be input to an encoder 415 to generate a latent space representation 420. As noted above, the latent space representation 420 is a low-dimensional representation.

[0063] Also, for example, motion data 425 (e.g., similar to motion data 205) and frame metadata 430 (e.g., similar to frame metadata 210) may be utilized to generate actual pose history 435. For example, actual pose history 435 may consist of rotations and translations of video frames going back N frames in the past, N frames in the future, and the current frame.

[0064] Initially, the virtual pose history 460 may be set to an identity rotation and a lens offset of (0,0). The virtual pose history 460 may consist of predicted virtual poses for one or more past video frames, but not for future frames, which have not been predicted. Also, the lookback for the virtual camera may be N frames or may be different. In general, the frame rate of the video frames may vary. For example, the frame rate may be 30 fps or 60 fps. Therefore, a fixed timestamp gap (e.g., 33 ms) corresponding to the 30 fps setting may be set. Based on the actual pose history 435 and the virtual pose history 460, a concatenated vector 465 may be generated. The concatenated vector may be input to the latent space representation 420. The decoder 440 may consist of an LSTM component 445 (e.g., LSTM component 320) and a warping grid 450 (e.g., warping grid 240 or 340). The LSTM component 445 may use the latent space representation 420 to generate a virtual pose 455 (e.g., a predicted rotation and a predicted translation of the virtual camera).

[0065] The virtual poses 455 may be added to the virtual pose history 460 to update the virtual pose queue. For example, after the initial rotation values ​​are set to 0, these initial values ​​may be updated as each predicted virtual pose 455 is output by the LSTM component 445. Also, for example, instead of absolute rotations, relative rotations may be input for the actual pose history 435 and relative virtual poses 455 may be predicted. The warping grid 450 may stabilize each input video frame using the virtual poses 455, and a stabilized output video 470 may be generated.

[0066] An exemplary architecture of the DNN 400 for predicting the stabilized version may include a VGG-like convolutional neural network (CNN). For example, the convolutional neural network may be modeled as a U-Net. In some embodiments, the input to the encoder 415 may be an optical flow 410. A frame of such an optical flow (e.g., optical flow 520 in FIG. 5) may have a size (4×270×480). The encoder 415 converts the input optical flow 410 into a low-dimensional latent space representation L r (e.g., latent space representation 420). The real pose history 435, which includes the rotation and translation of the real camera, and the virtual pose history 460, which includes the predicted rotation and translation of the virtual camera, are concatenated to form vector 465.

[0067] The concatenated vector 465 is then converted into the latent space representation L r and concatenated to form the latent space representation L v =(L r ,dR r ,dR v ), where dR r represents the relative rotation of the real camera pose, and dR v represents the relative rotation of the virtual camera pose. The decoder 440 in the U-net may include an LSTM component 445 and a differentiable warping grid 450. Specifically, the LSTM component 445 outputs a virtual pose 455 that includes the relative rotation, which is then input to the differentiable warping grid 450 to generate a warped, stabilized frame of the output video 470.

[0068] In one example implementation of the DNN 400, the input size of the forward and backward optical flow 410 may be (4,270,480). There may be a total of five CNN hidden layers with sizes (8,270,480), (16,67,120), (32,16,30), (64,4,7), and (128,1,1). Each hidden layer may be generated by 2D operations using a rectified linear unit (ReLU) activation function. Features from the optical flow 410 may be resized to 64 by a fully connected (FC) layer before concatenation with the concatenated vector 465. The input data size of the latent space representation 420 may be (21+10)*4+64, which corresponds to 21 poses in the real pose history 435 (e.g., real poses from 10 past video frames, 10 future video frames, and the current video frame), 10 poses in the virtual pose history 460 (e.g., predicted virtual poses from 10 past video frames), and 64-dimensional optical flow 410. The latent space representation 420 may be input to a two-layer LSTM component 445 having sizes 512 and 512. The hidden states from the LSTM component 445 may be fed into an FC layer followed by a Softshrink activation function to generate an output image 470 (e.g., expressed as a 4D quaternion). In general, the Softshrink activation function can smooth the output and remove noise.

[0069] III. Training a Machine Learning Model with a Loss Function The neural networks described herein can be trained based on an optimization process that is based on one or more loss functions that can be designed to constrain the solution space. For example, the total loss function can be determined as follows:

[0070]

number

[0071] During the ceremony, w_ *are the respective weights assigned to each type of loss. These weights may be used to adjust the influence of each loss on the training process. In some embodiments, training the neural network involves adjusting, for a particular video frame, the difference between the virtual camera poses of successive video frames. For example, the C0 smoothness loss is adjusted using the weight w C0 and the loss can be determined as follows:

[0072]

number

[0073] In the formula, dR v (T) is the rotation of the reference frame R identity The C0 smoothness loss measures the relative rotation of the virtual camera pose with respect to the C0. The C0 smoothness loss ensures C0 continuity of virtual pose changes (i.e., changes in rotation) in a time domain. In general, C0 smoothness means that the current virtual pose is close to the previous virtual pose. In some embodiments, training the neural network includes adjusting, for a particular video frame, the first-order difference between the virtual camera poses of successive video frames.

[0074] Similarly, the C1 smoothness loss is weighted by w C1 and the loss can be determined as follows:

[0075]

number

[0076] The C1 smoothness loss ensures C1 continuity of virtual pose changes (i.e., rotation changes) in a certain time domain. In general, C1 smoothness is the measure of the change dR between the current and previous virtual camera poses. v (T) is the change dR between the previous virtual camera pose and the virtual camera pose before the previous virtual camera pose. vThis means that the loss function is identical to (Tg), i.e., their first order derivatives are close to each other. Therefore, this loss function provides a smoothly changing trajectory for the virtual camera pose. Both C0 smoothness and C1 smoothness ensure that the virtual camera pose is stable and smoothly changing.

[0077] In some embodiments, training the neural network involves adjusting the angular difference between the real and virtual camera poses for a particular video frame. For example, another loss that can be measured is the angular loss E, which indicates how closely the virtual camera pose tracks the real camera pose. angle The angle loss is calculated by the weight w angle and may be measured as the angular difference between the virtual camera pose and the real camera pose. The desired difference in angular difference may be 0, although some implementations may include a tolerance threshold.

[0078] Therefore, E angle =Logistic(θ,θ threshold ) measures the angular difference θ between the real and virtual camera rotations. θ is the threshold θ threshold A logistic function may be used to allow this angular difference to be valid when θ is greater than θ . In this way, the virtual camera is still free to move when the deviation of the virtual pose from the real pose is within a threshold, preventing rotation of the virtual camera from the real pose beyond the threshold. For example, in some implementations, θ threshold may be set to 8°, and the virtual camera pose may be allowed to deviate from the real camera pose by no more than 8°. In some embodiments, upon determining that the angular difference exceeds a threshold angle, the angular difference between the real camera pose and the virtual camera pose may be reduced. For example, if the virtual camera pose deviates from the real camera pose by more than 8°, the virtual camera pose may be adjusted to bring the difference between the real camera pose and the virtual camera pose to less than 8°.

[0079] In some embodiments, training the neural network includes adjusting for an area of ​​a distorted region that indicates undesired movement of the mobile computing device for a particular video frame. For example, another loss that may be measured is the area of ​​the distorted region that indicates undesired movement of the mobile computing device (alternatively referred to herein as "undefined region"). In some embodiments, the area of ​​the distorted region in one or more video frames that appear after a particular video frame may be determined. For example, the amount of undefined region from a current video frame to one or more future video frames, e.g., N look-ahead frames, may be measured as follows:

[0080]

number

[0081] where, for each i, w i is a preset weight that is large for frames close to the current frame and decreases with i. i (dR(T) is used to calculate the amount of undefined region using the current virtual pose dR(T) and the real camera pose at timestamp T+i*g. The output is a 1D normalized value measuring the maximum protrusion between the bounding box of the warped frame (e.g., the frame output by the warping grid) and the boundary of the real image. The loss E undefined is the weight w undefined can be associated with.

[0082] If only the undefined region of the current frame is taken into account, the resulting video may not be smooth. For example, there may be sudden movements in future video frames. Therefore, it may be necessary to take such sudden movements in future frames into account by adjusting the undefined region. One technique may include taking the undefined region of the current frame 0 and all N future frames. All current and future weights wi A difference between the distance of one or more video frames from a particular video frame may be determined. In some embodiments, the applied weights may be configured to decrease with the distance of the one or more video frames from the particular video frame. For example, such weights may be generally taken to be Gaussian weights. Thus, a large weight may be associated with the current frame (indicating the relative importance of the current frame with respect to future frames) and a small weight may be associated with future frames. The weights may be selected such that undefined regions in the current frame are treated as more important than undefined regions in future frames. If the virtual camera is vibrating due to hand movement, only the current frame may be used for undefined region loss. However, if the virtual camera is panning, the virtual camera is configured to follow the real camera as it is moving away from the current frame. Therefore, a number of look-ahead frames may be determined based on the type of camera motion. In some exemplary embodiments, 7 or 10 look-ahead video frames may provide excellent results. Also, increasing the number of look-ahead video frames may provide better output. However, as the number of look-ahead video frames increases, memory resource requirements also increase. In some mobile devices, up to 30 look-ahead video frames may be used.

[0083] In some embodiments, training the neural network includes adjusting the image loss for a particular video frame. Generally, the image loss E image measures the difference in optical flow between successive stabilized video frames. As mentioned above, optical flow connects pairs of corresponding points in successive video frames. If the camera motion is stable, the magnitude of optical flow will generally be close to 0. Specifically, for any point p in the previous frame at time t-1, f(t-1) For each point p in the frame at time t, we use forward optical flow to find the corresponding point p f(t)Similarly, for any point p in the current frame, b(t) , the corresponding point p in the previous frame at time t-1 b(t-1) can then be determined. image is the weight w image and can be determined as follows:

[0084]

number

[0085] Since optical flow includes both forward and backward flow, both directions of optical flow can be used in the image loss function. f(t) -p f(t-1) Similarly, for backward flow, p b(t) -p b(t-1) The feature difference between the current frame and the previous frame, denoted by , can be determined. The output video frame can be stabilized by minimizing the difference between these two. Since the optical flow is dense optical flow, the sum in Equation 15 spans all image pixels.

[0086] During the training phase, a training batch may be determined, for example, by randomly selecting a subsequence as a training batch. In some embodiments, the subsequence may include 400 video frames, and each of the 400 video frames may be processed by the neural network described herein. The loss functions may be combined into an overall loss function, which may then be backpropagated. By repeating this process, all of the parameters of the neural network may be trained. For example, due to the nature of LSTMs, long subsequences from a sequence of training video frames may be randomly selected as input, and the LSTM process may be applied to each such subsequence in a frame-by-frame manner. An overall loss for the entire sequence of training video frames may be determined and then backpropagated. In this way, the LSTM may be trained to learn to effectively represent various motion states (e.g., walking, running, panning) in the latent space representation.

[0087] In general, when a DNN is trained directly using the global loss, the training loss may not converge. This may be due to the complexity of the video stabilization problem and the large size of the feasible solution space. To overcome this problem, a multi-stage training process may be used as an offline training process to refine the solution space at each step.

[0088] For example, in the first stage of a multi-stage training process, the C0 and C1 smoothness losses and the angle loss can be optimized. Without the first stage, the undefined region loss can be substantially larger. In the first stage, the DNN is trained to track the virtual camera with the real camera, achieving C0 and C1 smoothness.

[0089] In the second stage of the multi-stage training process, the C0 and C1 smoothness losses and the undefined region loss may be optimized. Without the second stage, the image loss may be substantially larger. Typically, in the second stage, the angle loss from the first stage is replaced with the undefined region loss. In the second stage, the DNN is trained to have the virtual camera sometimes follow the real camera pose instead of always following it. However, as the size of the undefined region increases, the virtual camera is trained to more closely follow the real camera. Similarly, for example, C0 and C1 smoothness are achieved. This stage allows the DNN to learn to stabilize the input video frames when the camera is vibrating (e.g., from unintentional hand movements of the user holding the camera).

[0090] In the third stage of the multi-stage training process, the C0 and C1 smoothness losses and the undefined region loss may be optimized along with the image loss. Typically, the image loss is added in the second stage of training. By adding the image loss function, the camera is trained to distinguish outdoor scenes (e.g., objects far away) from indoor scenes (e.g., with a critical distance).

[0091] IV. Training a Machine Learning Model to Generate Inferences / Predictions FIG. 6 is a diagram 600 illustrating a training phase 602 and an inference phase 604 of a trained machine learning model 632, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input training dataset to recognize patterns in the training data and provide output inferences and / or predictions about the patterns in the training data. The resulting trained machine learning algorithms can be referred to as trained machine learning models. For example, FIG. 6 illustrates a training phase 602 in which one or more machine learning algorithms 620 are trained on training data 610 to become trained machine learning models 632. Then, during the inference phase 604, the trained machine learning model 632 can receive input data 630 and one or more inference / prediction requests 640 (perhaps as part of the input data 630) and, in response, provide one or more inferences and / or predictions 650 as output.

[0092] Thus, the trained machine learning model 632 may include one or more models of one or more machine learning algorithms 620. The machine learning algorithms 620 may include, but are not limited to, artificial neural networks (e.g., convolutional neural networks described herein, recurrent neural networks, Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, suitable statistical machine learning algorithms, and / or heuristic machine learning systems). The machine learning algorithms 620 may be supervised or unsupervised and may implement any suitable combination of online and offline learning.

[0093] In some examples, the machine learning algorithm 620 and / or the trained machine learning model 632 may be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application specific integrated circuit (ASIC). Such on-device coprocessors may be used to speed up the machine learning algorithm 620 and / or the trained machine learning model 632. In some examples, the trained machine learning model 632 may be trained to provide inference, reside, execute on, and / or perform inference for a particular computing device.

[0094] During the training phase 602, the machine learning algorithm 620 may be trained by providing at least the training data 610 as training inputs using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of the training data 610 to the machine learning algorithm 620, and the machine learning algorithm 620 determining one or more output inferences based on the portion (or all) of the provided training data 610. Supervised learning involves providing a portion (or all) of the training data 610 to the machine learning algorithm 620, and the machine learning algorithm 620 determining one or more output inferences based on the portion (or all) of the provided training data 610, where the output inferences are accepted or corrected based on the correct results associated with the training data 610. In some examples, the supervised learning of the machine learning algorithm 620 may be governed by a set of rules and / or a set of labels for the training inputs, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 620.

[0095] Semi-supervised learning involves having correct results for some, but not all, of the training data 610. During semi-supervised learning, supervised learning is used for the portions of the training data 610 that have correct results, and unsupervised learning is used for the portions of the training data 610 that do not have correct results. Reinforcement learning involves the machine learning algorithm 620 receiving a reward signal related to a prior inference, where the reward signal may be a numerical value. During reinforcement learning, the machine learning algorithm 620 can output an inference and receive a reward signal in response, where the machine learning algorithm 620 is configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum over time of the numerical values ​​provided by the reward signal. In some examples, the machine learning algorithm 620 and / or the trained machine learning model 632 may be trained using other machine learning techniques, including, but not limited to, incremental learning and curriculum learning.

[0096] In some examples, the machine learning algorithm 620 and / or the trained machine learning model 632 can use transfer learning techniques. For example, transfer learning techniques may include the trained machine learning model 632 being pre-trained on one set of data and further trained using training data 610. More particularly, the machine learning algorithm 620 may be pre-trained on data from one or more computing devices and the resulting trained machine learning model provided to the computing device CD1, which is intended to execute the trained machine learning model during the inference phase 604. Then, during the training phase 602, the pre-trained machine learning model may be further trained using training data 610, which may be derived from kernel data and non-kernel data of the computing device CD1. This further training of the machine learning algorithm 620 and / or the pre-trained machine learning model using the training data 610 of the data of CD1 may be performed using either supervised learning or unsupervised learning. Once the machine learning algorithm 620 and / or the pre-trained machine learning model have been trained on at least the training data 610, the training phase 602 may be complete. The resulting trained machine learning model can be utilized as at least one of the trained machine learning models 632.

[0097] In particular, once the training phase 602 is complete, the trained machine learning model 632 may be provided to the computing device if it is not already on the computing device. After the trained machine learning model 632 is provided to the computing device CD1, the inference phase 604 may begin.

[0098] During the inference phase 604, the trained machine learning model 632 can receive input data 630 and generate and output one or more corresponding inferences and / or predictions 650 for the input data 630. Thus, the input data 630 can be used as input to the trained machine learning model 632 to provide the corresponding inferences and / or predictions 650 to kernel and non-kernel components. For example, the trained machine learning model 632 can generate the inferences and / or predictions 650 in response to one or more inference / prediction requests 640. In some examples, the trained machine learning model 632 may be executed by another piece of software. For example, the trained machine learning model 632 may be executed by an inference or prediction daemon so that it is readily available to provide inferences and / or predictions upon request. The input data 630 may include data from the computing device CD1 executing the trained machine learning model 632 and / or input data from one or more computing devices other than CD1.

[0099] The input data 630 may include a collection of video frames provided by one or more sources. The collection of video frames may include video of objects under various motion conditions, such as camera shake, motion blur, rolling shutter, panning video, video captured while walking, running, or traveling in a vehicle, etc. Also, for example, the collection of video frames may include video of indoor and outdoor scenes. Other types of input data are possible.

[0100] The inferences and / or predictions 650 may include output images, output rotations of the virtual camera, output lens offsets of the virtual camera, and / or other output data generated by the trained machine learning model 632 operating on the input data 630 (and training data 610). In some examples, the trained machine learning model 632 can use the output inferences and / or predictions 650 as input feedback 660. The trained machine learning model 632 may also rely on past inferences as input to generate new inferences.

[0101] A convolutional neural network 220, 320, etc. may be an example of a machine learning algorithm 620. After training, a trained version of a convolutional neural network 220, 320, etc. may be an example of a trained machine learning model 632. In this approach, an example of an inference / prediction request 640 may be a request to stabilize an input video, and a corresponding example of inference and / or prediction 650 may be a stabilized output video.

[0102] In some examples, one computing device CD_SOLO may include a trained version of convolutional neural network 100, perhaps after training convolutional neural network 100. Computing device CD_SOLO may then receive a request to stabilize an input video and use the trained version of convolutional neural network 100 to generate the stabilized video.

[0103] In some examples, two or more computing devices CD_CLI and CD_SRV can be used to provide output images. For example, a first computing device CD_CLI can generate and send a request to stabilize an input video to a second computing device CD_SRV. CD_SRV can then generate the stabilized video using a trained version of convolutional neural network 100, perhaps after training convolutional neural network 100, and respond to a request from CD_CLI for the stabilized video. Then, upon receiving a response to the request, CD_CLI can provide the requested stabilized video (e.g., using a user interface and / or display).

[0104] V. Exemplary Data Network 7 illustrates a distributed computing architecture 700 according to an example embodiment. The distributed computing architecture 700 includes server devices 708 and 710 configured to communicate with programmable devices 704a, 704b, 704c, 704d, and 704e over a network 706. The network 706 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 706 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0105] While FIG. 7 shows only five programmable devices, the distributed application architecture may handle tens, hundreds, or thousands of programmable devices. Furthermore, programmable devices 704a, 704b, 704c, 704d, and 704e (or any additional programmable devices) may be any type of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted display (HMD), a network terminal, a mobile computing device, etc. In some examples, such as the example shown by programmable devices 704a, 704b, 704c, and 704e, the programmable devices may be directly connected to network 706. In other examples, such as the example shown by programmable device 704d, the programmable devices may be indirectly connected to network 706 through an associated computing device, such as programmable device 704c. In this example, programmable device 704c may act as an associated computing device to pass electronic communications between programmable device 704d and network 706. In other examples, such as that shown by programmable device 704e, the computing device may be part of and / or be within a vehicle, such as an automobile, truck, bus, boat or watercraft, airplane, etc. In other examples not shown in Figure 7, the programmable device may be directly or indirectly connected to network 706.

[0106] Server devices 708, 710 may be configured to perform one or more services requested by programmable devices 704a-704e. For example, server devices 708 and / or 710 may provide content to programmable devices 704a-704e. This content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are possible.

[0107] As another example, server devices 708 and / or 710 may provide software access for databases, search, computation, graphical, audio, video, World Wide Web / Internet usage, and / or other functionality to programmable devices 704a-704e. Many other examples of server devices are possible.

[0108] VI. Computing Device Architecture 8 is a block diagram of an exemplary computing device 800, according to an exemplary embodiment. In particular, the computing device 800 shown in FIG. 8 may be configured to perform at least one function of and / or associated with a convolutional neural network and / or method 1000 disclosed herein.

[0109] The computing device 800 may include a user interface module 801, a network communication module 802, one or more processors 803, data storage 804, one or more cameras 818, one or more sensors 820, and a power system 822, all of which may be linked together by a system bus, network, or other connection mechanism 805.

[0110] The user interface module 801 may be operable to transmit and / or receive data from external user input / output devices. For example, the user interface module 801 may be configured to transmit and / or receive data from user input devices such as a touchscreen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 801 may also be configured to provide output to a user display device such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices now known or later developed. The user interface module 801 may also be configured to generate audible output using devices such as a speaker, a speaker jack, an audio output port, an audio output device, an earphone, and / or other similar devices. Additionally, user interface module 801 may be configured with one or more haptic devices capable of generating haptic output, such as vibration and / or other output detectable by touch and / or physical contact with computing device 800. In some examples, user interface module 801 may be used to provide a graphical user interface (GUI) for utilizing computing device 800, such as, for example, the graphical user interface shown in FIG.

[0111] The network communication module 802 may include one or more devices providing one or more wireless interfaces 807 and / or one or more wireline interfaces 808 configurable to communicate over a network. The wireless interface 807 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth® transceiver, a Zigbee® transceiver, a Wi-Fi® transceiver, a WiMAX® transceiver, an LTE® transceiver, and / or other types of wireless transceivers configurable to communicate over a wireless network. The wireline interface 808 may include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet® transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate over twisted pair wire, coaxial cable, an optical fiber link, or a similar physical connection to a wireline network.

[0112] In some examples, the network communication module 802 may be configured to provide reliable, secure, and / or authenticated communications. For each communication described herein, information to facilitate reliable communications (e.g., guaranteed message delivery) may be provided, perhaps as part of the message header and / or footer (e.g., packet / message sequencing information, encapsulation header and / or footer, size / time information, and transmission verification information such as a Cyclic Redundancy Check (CRC) and / or parity check value). Communications may be secured (e.g., encoded or encrypted) and / or decrypted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, a secure sockets protocol such as the Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or the Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms may be used to secure (and subsequently decrypt / decode) communications, similar to or in addition to those listed herein.

[0113] The one or more processors 803 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits, etc.) The one or more processors 803 may be configured to execute computer-readable instructions 806 contained in data storage 804 and / or other instructions described herein.

[0114] Data storage 804 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 803. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, that may be integrated in whole or in part with at least one of the one or more processors 803. In some examples, data storage 804 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage 804 may be implemented using two or more physical devices.

[0115] Data storage 804 may include computer-readable instructions 806 and possibly additional data. In some examples, data storage 804 may include storage necessary to execute at least some of the methods, scenarios, and techniques described herein and / or at least some of the functionality of the devices and networks described herein. In some examples, data storage 804 may include storage for trained neural network model 812 (e.g., a trained convolutional neural network model). In particular, in these examples, computer-readable instructions 806 may include instructions that, when executed by processor 803, enable computing device 800 to provide some or all of the functionality of trained neural network model 812.

[0116] In some examples, computing device 800 may include one or more cameras 818. Camera 818 may include one or more image capture devices, such as a still camera and / or a video camera, equipped to capture light and record the captured light into one or more images. That is, camera 818 may generate images of the captured light. The one or more images may be one or more still images and / or one or more images utilized in a video image. Camera 818 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or one or more other frequencies of light.

[0117] In some examples, computing device 800 may include one or more sensors 820. Sensors 820 may be configured to measure conditions within computing device 800 and / or conditions in the environment of computing device 800 and provide data about these conditions.For example, the sensors 820 may include (i) sensors for obtaining data about the computing device 800 (such as, but not limited to, a thermometer for measuring the temperature of the computing device 800, a battery sensor for measuring the power of one or more batteries of the power system 822, and / or other sensors for measuring the condition of the computing device 800), (ii) identification sensors for identifying other objects and / or devices (such as, but not limited to, a Radio Frequency Identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode (e.g., a Quick Response (QR) (iii) sensors for measuring the location and / or movement of computing device 800 (such as, but not limited to, tilt sensors, gyroscopes, accelerometers, Doppler sensors, GPS devices, sonar sensors, radar devices, laser displacement sensors, and compasses); and (iv) sensors for obtaining data indicative of the environment of computing device 800 (such as, but not limited to, RFID tags, bar codes, QR codes, and other devices and / or objects configured to read and provide at least identification information). The computing device 800 may include one or more of the following: (i) environmental sensors (such as, but not limited to, infrared sensors, optical sensors, light sensors, biosensors, capacitive sensors, touch sensors, temperature sensors, wireless sensors, radio wave sensors, movement sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors); and / or (v) force sensors for measuring one or more forces (e.g., inertial forces and / or gravitational acceleration) acting on the computing device 800 (such as, but not limited to, one or more sensors measuring forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor for identifying the ZMP and / or the location of the ZMP).Many other examples of sensors 820 are possible.

[0118] The power system 822 may include one or more batteries 824 and / or one or more external power interfaces 826 for providing power to the computing device 800. Each battery of the one or more batteries 824, when electrically coupled to the computing device 800, can serve as a source of stored power for the computing device 800. The one or more batteries 824 of the power system 822 may be configured to be portable. Some or all of the one or more batteries 824 can be easily removable from the computing device 800. In other examples, some or all of the one or more batteries 824 may be internal to the computing device 800 and therefore not easily removable from the computing device 800. Some or all of the one or more batteries 824 may be rechargeable. For example, rechargeable batteries may be rechargeable via a wired connection between the battery and another power source, such as by one or more power sources external to the computing device 800 and connected to the computing device 800 via one or more external power interfaces. In other examples, some or all of the one or more batteries 824 may be non-rechargeable batteries.

[0119] The one or more external power interfaces 826 of power system 822 may include one or more wired power interfaces, such as a USB cable and / or a power cord, that enable a wired power connection to one or more power sources external to computing device 800. The one or more external power interfaces 826 may include one or more wireless power interfaces, such as a Qi wireless charger, that enable a wireless power connection to one or more external power sources via, for example, a Qi wireless charger. Once a power connection with an external power source is established using the one or more external power interfaces 826, computing device 800 can draw power from the external power source via the established power connection. In some examples, power system 822 may include associated sensors, such as battery sensors associated with one or more batteries or other types of power sensors.

[0120] VII. Cloud-Based Servers FIG. 9 illustrates a cloud-based server system according to an exemplary embodiment. In FIG. 9, functions of a convolutional neural network and / or computing devices can be distributed across computing clusters 909a, 909b, and 909c. Computing cluster 909a can include one or more computing devices 900a, a cluster storage array 910a, and a cluster router 911a, which are connected by a local cluster network 912a. Similarly, computing cluster 909b can include one or more computing devices 900b, a cluster storage array 910b, and a cluster router 911b, which are connected by a local cluster network 912b. Similarly, computing cluster 909c may include one or more computing devices 900c, a cluster storage array 910c, and a cluster router 911c, where the one or more computing devices 900c, the cluster storage array 910c, and the cluster router 911c are connected by a local cluster network 912c.

[0121] In some embodiments, each of computing clusters 909a, 909b, and 909c may have the same number of computing devices, the same number of cluster storage arrays, and the same number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster depends on the computing task or tasks assigned to each computing cluster.

[0122] For example, in computing cluster 909a, computing device 900a may be configured to perform various computing tasks of the convolutional neural network, confidence learning, and / or computing device. In one embodiment, the various functions of the convolutional neural network, confidence learning, and / or computing device may be distributed across one or more of computing devices 900a, 900b, and 900c. Computing devices 900b and 900c in each computing cluster 909b and 909c may be configured similarly to computing device 900a in computing cluster 909a. Meanwhile, in some embodiments, computing devices 900a, 900b, and 900c may be configured to perform different functions.

[0123] In some embodiments, computing tasks and stored data associated with the convolutional neural network and / or computing devices may be distributed across computing devices 900a, 900b, and 900c based at least in part on the processing requirements of the convolutional neural network and / or computing devices, the processing capabilities of computing devices 900a, 900b, and 900c, the latency of network links between computing devices within each computing cluster and between the computing clusters themselves, and / or other factors that may contribute to cost, speed, fault tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.

[0124] The cluster storage arrays 910a, 910b, 910c of the computing clusters 909a, 909b, 909c may be data storage arrays including disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, may be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0125] Similar to the way that the functionality and / or computing devices of a convolutional neural network can be distributed across computing devices 900a, 900b, and 900c of computing clusters 909a, 909b, and 909c, various active and / or backup portions of these components can be distributed across cluster storage arrays 910a, 910b, and 910c. For example, some cluster storage arrays can be configured to store portions of data for a convolutional neural network and / or computing device, while other cluster storage arrays can store other portions of data for a convolutional neural network and / or computing device. Also, for example, some cluster storage arrays can be configured to store data for a first convolutional neural network, while other cluster storage arrays can store data for a second and / or third convolutional neural network. Furthermore, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0126] Cluster routers 911a, 911b, and 911c in computing clusters 909a, 909b, and 909c may include networking equipment configured to provide internal and external communications for the computing clusters. For example, cluster router 911a in computing cluster 909a may include one or more Internet switching and routing devices configured to provide (i) local area network communications between computing device 900a and cluster storage array 910a via local cluster network 912a, and (ii) wide area network communications between computing cluster 909a and computing clusters 909b and 909c via wide area network link 913a to network 706. Cluster routers 911b and 911c may include networking equipment similar to cluster router 911a, and cluster routers 911b and 911c may perform similar networking functions for computing clusters 909b and 909b as cluster router 911a performs for computing cluster 909a.

[0127] In some embodiments, the configuration of the cluster routers 911a, 911b, 911c may be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communication capabilities of the network equipment within the cluster routers 911a, 911b, 911c, the latency and throughput of the local cluster networks 912a, 912b, 912c, the latency, throughput and cost of the wide area network links 913a, 913b, 913c, and / or other factors that may contribute to the cost, speed, fault tolerance, resiliency, efficiency and / or other design criteria of the moderation system architecture.

[0128] VIII. Exemplary Methods of Operation 10 illustrates a method 1000 according to an exemplary embodiment. Method 1000 may include various blocks or steps. These blocks or steps may be performed individually or in combination. These blocks or steps may be performed in any order and / or in series or parallel. Additionally, blocks or steps may be omitted or added to method 1000.

[0129] The blocks of the method 1000 may be performed by various elements of the computing device 800 illustrated and described with reference to FIG.

[0130] Block 1010 includes the mobile computing device receiving one or more image parameters associated with a video frame of the plurality of video frames.

[0131] Block 1020 includes receiving motion data associated with the video frame from a motion sensor of the mobile computing device.

[0132] Block 1030 includes predicting a stabilized version of the video frame by applying a neural network to the one or more image parameters and the motion data.

[0133] In some embodiments, the neural network may include an encoder and a decoder, and applying the neural network may include applying the encoder to the one or more image parameters to generate a latent space representation, adjusting the latent space representation based on the motion data, and applying the decoder to the adjusted latent space representation to output the stabilized version.

[0134] Some embodiments include generating a real camera pose associated with the video frame from the motion data, wherein the latent space representation may be based on the real camera pose.

[0135] In some embodiments, the decoder may include a long short-term memory (LSTM) component, and applying the decoder may include applying the LSTM component to predict a virtual camera pose.

[0136] In some embodiments, the decoder may include a warping grid, and applying the decoder may include applying the warping grid to the predicted virtual camera pose to output the stabilized version.

[0137] Some embodiments include determining a history of real camera poses and a history of virtual camera poses, and the latent space representation may be based on the history of the real camera poses and the history of the virtual camera poses.

[0138] In some embodiments, the motion data includes rotation data and timestamp data. Such embodiments may include determining from the rotation data and the timestamp data a relative rotation of a camera pose in the video frame relative to a reference camera pose in a reference video frame. The prediction of the stabilized version may be based on the relative rotation.

[0139] In some embodiments, applying the encoder may include generating an optical flow from a pair of consecutive video frames of the plurality of video frames, the optical flow indicating a correspondence between the pair of consecutive video frames, and the method may further include generating the latent space representation based on the optical flow.

[0140] Some embodiments include training the neural network to receive a particular video frame and output a stabilized version of the particular video frame based on one or more image parameters and motion data associated with the particular video frame.

[0141] In some embodiments, training the neural network may include adjusting for the difference between the real camera pose and the virtual camera pose for the particular video frame.

[0142] In some embodiments, training the neural network may include adjusting for a first order difference between the real and virtual camera poses for the particular video frame.

[0143] In some embodiments, training the neural network may include adjusting an angular difference between the real camera pose and the virtual camera pose for the particular video frame, hi some embodiments, adjusting the angular difference includes decreasing the angular difference between the real camera pose and the virtual camera pose upon determining that the angular difference exceeds a threshold angle.

[0144] In some embodiments, training the neural network may include adjusting, for the particular video frame, an area of ​​a distorted region indicative of undesired movement of the mobile computing device. In some embodiments, adjusting the area of ​​the distorted region includes determining an area of ​​a distorted region in one or more video frames that appear after the particular video frame. The method further includes applying a weight to the area of ​​the distorted region. The applied weight may be configured to decrease with distance of a video frame of the one or more video frames from the particular video frame.

[0145] In some embodiments, training the neural network may include adjusting for image loss for the particular video frame.

[0146] In some embodiments, the one or more image parameters may include Optical Image Stabilization (OIS) data indicative of a lens position, and applying the neural network includes predicting a lens offset of a virtual camera based on the lens position.

[0147] In some embodiments, predicting the stabilized version of the video frame includes obtaining the trained neural network at the mobile computing device, and the method further includes applying the obtained trained neural network to the prediction of the stabilized version.

[0148] The particular arrangements shown in the drawings should not be considered limiting. It should be understood that other embodiments may include more or fewer elements than each element shown in a given drawing. Furthermore, some of the elements shown may be combined or omitted. Furthermore, example embodiments may include elements not shown in the drawings.

[0149] The steps or blocks representing the processing of information may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the steps or blocks representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to perform specific logical functions or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a disk, hard drive, or other storage medium.

[0150] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media that store data for short periods of time, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for long periods of time. Thus, computer-readable media may include secondary or permanent long-term storage, such as, for example, read-only memory (ROM), optical or magnetic disk, compact-disc read-only memory (CD-ROM). Computer-readable media may also be other volatile or non-volatile storage systems. Computer-readable media may be considered, for example, to be computer-readable storage media or tangible storage devices.

[0151] While various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are for purposes of illustration and are not intended to be limiting, the true scope being indicated by the following claims.

Claims

1. 1. A computer-implemented method comprising: receiving, via a camera at a mobile computing device, a plurality of video frames; receiving, by the mobile computing device, one or more image parameters associated with each of the plurality of video frames; receiving motion data associated with each of the plurality of video frames from a motion sensor of the mobile computing device; predicting a stabilized version of each of the plurality of video frames by applying a neural network including an encoder and a decoder to the one or more image parameters and the motion data; said applying said neural network applying the encoder to the one or more image parameters and the motion data to generate a latent space representation including real camera pose information associated with each of the plurality of video frames, wherein generating the latent space representation comprises: generating the actual camera pose associated with each of the plurality of video frames from the motion data; adjusting the latent space representation based on a first history of the real camera poses and a second history of virtual camera poses for the real camera poses; The applying the neural network further comprises: applying the decoder to the latent space representation to predict the virtual camera pose; generating a warping grid based on the virtual camera pose to stabilize the plurality of video frames; A computer-implemented method, wherein predicting a stabilized version of each of the plurality of video frames includes applying the warping grid to each of the plurality of video frames.

2. The decoder includes a long short-term memory (LSTM) component, and applying the decoder utilizes the LSTM to predict a virtual camera pose. The computer-implemented method of claim 1 , further comprising applying the component.

3. The motion data includes rotation data and timestamp data, and the method further comprises: determining a relative rotation of a camera pose in the video frame relative to a reference camera pose in a reference video frame from the rotation data and the timestamp data; The computer-implemented method of claim 1 , wherein the prediction of the stabilized version is based on the relative rotation.

4. Applying the encoder comprises: generating an optical flow indicating a correspondence relationship between a pair of consecutive video frames from the plurality of video frames; and generating the latent space representation based on the optical flow.

5. 10. The computer-implemented method of claim 1, further comprising training the neural network to receive successive video frames and output stabilized versions of each of the successive video frames based on one or more image parameters and motion data associated with each of the successive video frames.

6. The computer-implemented method of claim 5 , wherein the training of the neural network further comprises adjusting for a difference between the virtual camera pose for each of the successive video frames.

7. 7. The computer-implemented method of claim 5 or 6, wherein the training of the neural network further comprises adjusting, for each of the successive video frames, an angular difference between the real camera pose and the virtual camera pose.

8. 8. The computer-implemented method of claim 7, wherein the adjusting the angular difference further comprises decreasing the angular difference between the real camera pose and the virtual camera pose upon determining that the angular difference exceeds a threshold angle.

9. 9. The computer-implemented method of claim 5, wherein the training of the neural network further comprises adjusting, for each of the successive video frames, an area of ​​distorted regions indicative of undesired movement of the mobile computing device.

10. The adjustment of the area of ​​the distorted region comprises: determining an area of ​​a distorted region in one or more video frames that occur after a video frame in the sequence of video frames; and applying weights to the areas of the distorted regions, the applied weights configured to decrease with distance of a video frame of the one or more video frames from the one video frame.

11. The computer-implemented method of any one of claims 5 to 10, wherein the training of the neural network further comprises adjusting for image loss for each of the successive video frames.

12. The one or more image parameters include Optical Image Stabilization (OIS) data indicating lens position, and the adaptive neural network The computer-implemented method of claim 1 , wherein the step of predicting includes predicting a lens offset of a virtual camera based on the lens position.

13. Predicting the stabilized version of each of the plurality of video frames includes: obtaining the trained neural network on the mobile computing device; and applying the obtained trained neural network to the prediction of the stabilized version.

14. 1. A computing device comprising: one or more processors; and a data storage having computer-executable instructions stored thereon that, when executed by the one or more processors, cause the computing device to perform functions including the computer-implemented method of any one of claims 1 to 13.

15. A program that, when executed by one or more processors of a computing device, causes the computing device to perform functions including the computer-implemented method of any one of claims 1 to 13.

Images