Video stabilization
The method uses homography transformation matrices to stabilize video across multiple degrees of freedom, addressing shake in handheld devices with limited processing power by filtering out high-frequency movements, ensuring effective stabilization without excessive distortion.
Patent Information
- Application Number
- DE112016004730
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2015-10-14
- Filing Date
- 2016-09-23
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2036-09-23
AI Technical Summary
Handheld video capture devices suffer from video shake due to user movements, which existing optical stabilization mechanisms fail to address effectively, especially in devices with limited processing power and increased cost or size constraints.
A computer-implemented method using homography transformation matrices to stabilize video by compensating for movements across eight degrees of freedom, applying transformations based on past frames without requiring future frame information, and employing low-pass filtering to reduce high-frequency camera movements.
The method provides effective video stabilization with low computational complexity, suitable for devices with limited processing power, while maintaining video quality by reducing high-frequency camera shake and avoiding unrealistic distortions.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
TECHNICAL FIELD
[0001] This document refers generally to video stabilization. BACKGROUND
[0002] Video recording has traditionally been confined to dedicated video recording devices, but everyday devices such as mobile phones and tablet computers that can record video are becoming more common. One problem with handheld recording devices is video shake, with unintentional movements of the user while holding the recording device affecting the video quality.
[0003] Shake in the capture device can result in equally shaky video unless that shake is compensated for, for example, by a video stabilization mechanism. Optical video stabilization can reduce the shake present in video caused by the mechanical movement of capture device components, such as a lens or the image sensor. However, optical video stabilization devices can increase the material and manufacturing costs of a capture device. Furthermore, optical video stabilization devices can increase the size of a capture device, and there is often a desire to design capture devices with small dimensions.
[0004] From the state of the art, solutions relating to video stabilisation mechanisms are known, for example from US 2011 / 0 085 049 A1, US 2009 / 0 066 800 A1, US 8 385 732 B2 and US 2013 / 0 121 597 A1. SUMMARY
[0005] This document describes techniques, procedures, systems, and other mechanisms for video stabilization.
[0006] As an additional description to the embodiments described below, the present disclosure describes the following embodiments.
[0007] The proposed solution relates in particular to a computer-implemented method according to claim 1.
[0008] The proposed solution further relates to one or more non-transitory computer-readable devices according to claim 9.
[0009] Certain implementations may achieve one or more of the following advantages in certain cases. The video stabilization techniques described herein may compensate for motion across more than two degrees of freedom (e.g., more than just horizontal and vertical movement), for example, by compensating for motion across eight degrees of freedom (e.g., translation aspects, rotation aspects, zooming aspects, and non-rigid rolling shutter distortion). The video stabilization techniques described herein may operate while video is being captured by a device and do not require information from future frames. In other words, the video stabilization techniques may use information from only past frames, allowing the system to store a stabilized video stream while that video stream is being captured (e.g.,: without storing multiple unstabilized video frames, such as without storing more than 1, 100, 500, 1000, or 5000 unstabilized video frames in a video that is currently being recorded or has been recorded). Accordingly, the system may not need to wait until after the entire video has been recorded to stabilize a video. The described video stabilization techniques may have low complexity and therefore may work on devices that have low processing power (e.g., some smartphones). Furthermore, the video stabilization techniques described herein may work in situations where frame-by-frame motion estimation fails in the first step.
[0010] The details of one or more implementations are set forth in the accompanying drawings and description below. Other features, objects, and advantages will be apparent from the description and drawings, as well as from the claims. DESCRIPTION OF THE DRAWINGS Fig. Figure 1 shows a diagram of a video stream stabilized by a video stabilization process. Fig. 2A - 2B shows a flowchart of a video stabilization process. Fig. Figure 3 is a block diagram of computer devices that can be used to implement the systems and methods described in this document as either a client or as one or more servers.
[0011] The same reference numerals in the different drawings refer to the same elements. DETAILED DESCRIPTION
[0012] This document generally describes video stabilization. Video stabilization can be performed by identifying a transformation between a most recently received frame of video and a previously received frame of video (where the transformation indicates movement of the camera relative to a scene from frame to frame), modifying this transformation based on information from past frames, creating a second transformation based on the transformation and the modified transformation, and applying the second transformation to the currently received frame to create a stabilized version of the currently received frame. This process is generally described in terms of Fig. 1 and in greater detail with regard to Fig. 2 described.
[0013] Fig. 1 illustrates a diagram of a video stream being stabilized by a video stabilization process. The figure includes three frames of video 110a-c. These frames may be consecutive, such that frame 110b may be the frame captured immediately after frame 110a was captured, and frame 110c may be the frame captured immediately after frame 110b was captured. This document may occasionally refer to two frames of video as a first frame of video and a second frame of video; however, the "first" notation does not necessarily mean that the first frame is the introductory frame of the entire video.
[0014] Frames 110a-c are shown positioned between or near lines 112a-b, which indicate a position of the scenes depicted by the frames relative to one another. The lines are provided in this figure to show that the camera was moving while capturing frames 110a-c. For example, the camera was pointing more downward while capturing frame 110b than while capturing frame 110a, and was pointing more upward while capturing frame 110c than while capturing frames 110a-b.
[0015] A computing system identifies a mathematical transformation (box 120) that indicates a movement of the camera from the first frame 110b to the second frame 110c. The identification may be performed using frames 110b-c (as illustrated by the arrows in the figure), where frame 110c may be the most recently captured frame. These two frames 110b-c may be received from a camera sensor or camera module connected to a computing system or may be received from a remote device that captured the video frames 110b-c. Identifying the mathematical transformation may include creating the mathematical transformation. The mathematical transformation may be a homography transformation matrix, as described in more detail with respect to box 210 in Fig. 2A-B.
[0016] A computational system then creates a modified transform (box 125) by modifying the initial transform (box 120) such that the modified transform (box 125) is less representative of recently initiated motion than the initial transform. In other words, creating the modified transform involves modifying the initial transform based on information from one or more video frames prior to the first and second frames 110b-c. The modified transform may be a low-pass filtered version of the initial transform. This results in a modified mathematical transform (box 125) that is more representative of motion that has occurred over a long period of time than the initial transformation (box 120), as opposed to motion that began recently.
[0017] As an example, the modified mathematical transformation (Box 125) can more accurately represent a panning motion that occurred over several seconds than an oscillation that began a fraction of a second ago. Modifying the transformation in this way takes previous frames of the video into account, as indicated by the arrows in Fig. 1, which point from frames 110a-b to box 122. Transformations calculated using, for example, the previous frames can be used to identify which movements have occurred over a longer period of time and which movements began only recently. An exemplary way in which previous frames can be used to calculate the modified transformation can be the application of a low-pass filter to the homography transformation matrix, as described in more detail with respect to box 220 in Fig. 2AB.
[0018] Box 130 shows a second transformation created from the initial transformation (Box 120) and the modified transformation (Box 125). The second transformation can be a difference between the initial transformation and the modified transformation. Creating the second transformation from the initial transformation and the modified transformation is described in more detail with reference to Box 230 in Fig. 2A-B.
[0019] Box 132 shows how a computing system identifies an expected distortion in a stabilized version of the second image frame 110c resulting from the application of the second transformation to the second image frame 110c based on a difference between (i) a measure of the distortion in the horizontal direction that would result from the application of the second mathematical transformation to the second image frame 110c and (ii) a measure of the distortion in the vertical direction that would result from the application of the second mathematical transformation to the second image frame 110c. The calculation of the expected distortion is described in more detail with respect to Box 250 in Fig. 2A-B.
[0020] Box 134 shows how a computing system determines an amount by which a stabilization effect resulting from the application of the second mathematical transformation to the second frame 110c should be reduced, based on a degree by which the expected distortion exceeds an acceptable change in distortion. The acceptable change in distortion may be calculated using several frames of video prior to the second frame 110c, as indicated by the arrows in Fig. 1, which point from frames 110a-b to box 134. As an example, the distortion in several previous frames can be analyzed, and if the distortion resulting from the stabilization of the current frame 110c deviates significantly from a degree by which the distortion changes from frame to frame, the computing system can reduce the stabilization of the current frame 110c to avoid the distortion being too obvious to a viewer of the video. Determining the degree by which the video stabilization should be reduced is described in more detail with respect to boxes 250 and 260 in Fig. 2. The use by the computing system of the determined amount by which the video stabilization is to be reduced may include the creation of a modified second transform (Box 140) using the determined amount.
[0021] The computing system creates the stabilized version of the second frame 110c (box 150) by applying the modified second transform (box 140) to the second frame 110c. Since the modified second transform (box 140) was modified based on the determined amount by which the stabilization is to be reduced, the computing system considers the creation of the stabilized version of the second frame (box 150) to be reduced based on the determined amount by which the stabilization effect is to be reduced.
[0022] In some implementations, the determination of the amount by which the stabilization effect should be reduced is further or alternatively based on a particular movement speed of the camera relative to the scene from the first frame 110b to the second frame 110c that exceeds an acceptable change in camera speed relative to the scene. The acceptable change in camera speed may be calculated from several frames of the video preceding the second frame 110c, as described in more detail with respect to boxes 240 and 260 in Fig. 2A-B.
[0023] In some implementations, creating the stabilized version of the second frame includes zooming into a version of the second frame created by applying the second mathematical transformation to the second frame. The computing system may translate a zoomed-in region horizontally and / or vertically to prevent the zoomed-in region from representing an invalid region that may appear at the edges of the stabilized second frame. This is described in more detail with respect to boxes 280 and 290 in Fig. 2A-B.
[0024] Fig. 2A-B show a flowchart of a video stabilization process. This process is represented by boxes 210 to 290, which are described below. The operations described in connection with these boxes may not have been described in the following or in Fig. 2AB listed order.
[0025] In box 210, the computing system evaluates a matrix representing frame-to-frame motion ("H_Interframe") using two video frames as input. This frame-to-frame motion matrix may be a homography transformation matrix. A homography transformation matrix may be a matrix representing motion of a scene or a camera that captured a scene between two frames of a video. As an example, each frame of a video may represent a two-dimensional photographic image. Suppose that a first frame took a photographic image of a square from just in front of the square, such that the square had equal sides with ninety-degree angles in the video frame (in other words, it appeared square).Now suppose the camera was moved sideways (or the square itself was moved), so that a subsequent frame of video showed the square as distorted, with some sides longer than others and angles that were not 90 degrees. The location of the square's four vertices in the first frame can be mapped to the location of the four vertices in the second frame to identify how the camera or scene moved from one frame to the next.
[0026] The mapping of these four vertices to each other in the individual images can be used to create a homography transformation matrix, which represents the movement of the camera's viewpoint relative to the scene it is capturing. With such a homography transformation matrix, a first image can be used with the created homography transformation matrix to reconstruct the second image, for example, by moving pixels in the first image to different locations according to known homography transformation methods.
[0027] The homography transformation matrix described above can represent not only translational motion, but also rotation, zoom, and non-rigid rolling shutter distortion. Thus, the application of the homography transformation matrix can be used to stabilize video with respect to motion across eight degrees of freedom. For comparison, some video stabilization mechanisms only stabilize images to compensate for translational motion (e.g., up / down and left / right motion).
[0028] The homography transformation matrix described above can be a 3x3 homography transformation matrix, although other types of homography matrices can be used (and other mathematical representations of motion from one frame to the next can be used, even if they are not homography or matrix). The 3x3 matrix (referred to as the H_interface) can be determined in the following way. First, a computational system finds a set of feature points (usually vertices) in the current frame, where these points are denoted as [x'_i, y'_i], i = 1....N (N is the number of feature points). Then, corresponding feature points are found in the previous frame, where the corresponding feature points are denoted as [x_i, y_i]. Note that the points are described as being in the GL coordinate system (i.e., the x and y range between -1 and 1, with the frame center as the origin).If these points are in the image pixel coordinate system, where x extends from 0 to the image width and y extends from 0 to the image height, the points can be converted to the GL coordinate system or the resulting matrix can be converted to compensate.
[0029] The H_Interframematrix described above is a 3x3 matrix containing 9 elements: h1, h2, h3 • H_Interframe = h4, h5, h6 h7, h8, h9 H_Interfame is the transformation matrix that transforms [x_i, y_i] into [x'_i, y'_i], as described below. • z_i*[x'_i,y'_i,1]'=H_Interframe *[x_i,y_i, 1]' [x'_i, y'_i, 1 ]' is a 3x1 vector that is the transpose of the [x'_i, y'_i, 1] vector. [x_i, y_i, 1]' is a 3x1 vector that is the transpose of the [x_i, y_i, 1] vector. z_i' is a scaling factor.
[0030] For a set of corresponding feature points, an example algorithm for estimating the matrix is described in the following computer vision book in Algorithm 4.1 (page 91) and Algorithm 4.6 (page 123): "Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press (2000), 2nd edition, first published in print format 2004, ISBN-13: 978-0-521-54051-3".
[0031] In box 220, the computing system estimates a low-pass transform matrix (H_LowPass). This low-pass transform matrix can later be combined with the H_Interframe matrix to create a new matrix (H_Compensation), which can be used to remove the results of the unintentional "high-frequency" movement of the video camera. If the system has attempted to remove all motion (in other words, has not performed the low-pass filtering described herein), the user may not be able to intentionally move the camera and cause the scene represented by the video to also move. As such, the computing system creates the low-pass transform to filter out the high-frequency motion. High-frequency motion can be motion that is irregular and not represented by any individual frames, such as back-and-forth movements within a short period of time.In contrast, low-frequency movements can be movements represented by many individual images, such as a user panning a video camera for several seconds.
[0032] To perform this filtering, the computing system creates a low-pass transform matrix (H_LowPass) that includes values weighted to emphasize low-frequency motion that has occurred over a long time series. The low-pass transform matrix may be the result of applying a low-pass filter to the H_Interframe matrix. Each element in the low-pass transform matrix is generated individually on an element-by-element basis from (1) its own time series of the low-pass transform matrix from the previous frame, (2) the H_Interframe matrix representing motion between the previous frame and the current frame, and (3) a damping ratio specified by the user. In other words, the elements in the matrix that are weighted with appreciable values may be those elements that represent motion that was present in the H_Interframe matrix across multiple frames.The equation for creating the H_low pass can be represented as follows:. • H_lowpass = H_previous_lowpass * transform_damping_ratio + H_interfame * (1−transform_damping_ratio) This equation is an example of an IIR (infinite impulse response) filter with two taps.
[0033] In box 230, the computing system calculates a compensation transformation matrix (H_compensation). The compensation matrix can be a combination of the low-pass matrix (H_lowpass) and the frame-to-frame motion matrix (H_interframe). The combination of these two matrices creates a matrix (H_compensation) needed to preserve the motion from one frame to the next, but only those movements that have occurred over a reasonable period of time, excluding recent "unintended" movements.The H_compensation matrix can represent the difference in motion between H_interframe and the H_lowpass filter, so while the H_lowpass filter could be applied to the last frame to create a modified version of the last frame representing the intended motion that occurred between the last frame and the current frame, the H_compensation can be applied to the current frame to create a modified (and stabilized) version of the current frame representing the intended motion that occurred between the last frame and the current frame. Roughly speaking, applying H_compensation to the current frame removes the unintended motion from that frame.Specifically, with this calculated H_compensation matrix, the system should be able to take the current frame of the video, apply the H_compensation matrix to this frame using a transformation process, and obtain a newly created frame that resembles the current frame but excludes any sudden and small movements. In other words, the system attempts to keep the current frame as close as possible to the previous frame, while allowing for long-term "intentional" movements.
[0034] The compensation transformation matrix can be created using the following equation: • H_Compensation = Normalizer(Normalizer(Normalizer(H_Lowpass)*H_previous_compensation)*Inverse(H_Interframe)) The previous H_previous_compensation_matrix is the H_constrained_compensation_matrix, which is calculated later in this process, but was calculated for the previous frame. Inverse() is the matrix inverse operation used to create the original version of the last frame by inverting the transformation. Combining the original version of the last frame with the low-pass filter matrix enables the intended movements. Combining it with H_previous_compensation compensates for the previous compensation value.
[0035] Normalize() is an operation that normalizes the 3x3 matrix by its second singular value. The normalization process is performed because some of the steps in the process might result in a transformation that—for lack of a better word—wouldn't make much sense in the real world. As such, a normalization process can ensure that a reasonable result is obtained at each step of the process. Normalization was performed for each step of the process so that an odd output from one step doesn't corrupt the remaining steps of the process (e.g., imagine if the odd output yields a value close to zero, which would also pull the output of the remaining steps close to zero). For the reasons discussed below, additional processing can potentially improve the results of the video stabilization process.
[0036] In box 240, the computing system calculates a speed reduction value. The speed reduction value can be a value used to determine how much the video stabilization is reduced when the camera is moving very quickly and the video stabilization becomes disruptive, as frame-to-frame motion can be unreliable. To calculate the amount by which the video stabilization can be reduced, the speed of the movement between frames is initially calculated. In this example, the computing system generates the speed of the frame center. The X-direction speed is taken from row 1, column 3, element in the H_lnterframe as follows (box 242): • Speed_x = H_Interframe[1, 3]*Aspect_Ratio The velocity in the Y direction is taken from row 2, column 3, element in the H_Interframe matrix as follows (Box 242): • Speed_y = H_Interframe[2, 3] The aspect ratio described above is the frame width divided by the frame height. These speed determinations can only account for translational motion between two frames; however, speed can account for rotation, zooming, or other types of motion in other examples.
[0037] The system can then determine the low-pass motion speed, which takes into account the long-term speed of the camera (or scene) and eliminates sudden and rapid "unintended" movements. This is done by combining the current speed with the previously calculated low-pass speed and further applying a damping ratio that inversely weights the current speed with respect to the previously calculated low-pass speed, for example, as follows: • Low-pass_speed_x = Low-pass_speed_x_previous *Speed_damping_ratio + Speed_x * (1 -Speed_damping_ratio) This equation actually generates a low-pass velocity by reducing the previously calculated velocity by an amount specified by a damping ratio. The reduction is offset by the current velocity. Thus, the current video velocity affects the overall low-pass velocity value, but is not an exclusive factor of the low-pass velocity value. The preceding equation represents a filter with infinite impulse response. The same process can be performed for the Y velocity to generate the low-pass Y velocity, for example, as follows: • Low-pass_speed_y = Low-pass_speed_y_previous *Speed_damping_ratio + Speed_y * (1−Speed_damping_ratio) The damping ratio in this process is set by a user and an example value is 0.99.
[0038] The process then combines these values to generate a unified representation of the low-pass velocity that takes into account motion in the X and Y directions, for example, using the following equation (Box 244): • Low-pass speed = sqrt(Low-pass speed_x ∗Low-pass speed_x + Low-pass speed_y ∗Low-pass speed_y) This calculated low-pass velocity essentially represents the long-term velocity of movement between frames. In other words, the low-pass velocity takes less account of recent changes in velocity and gives more weight to the long-term velocity trend.
[0039] Once the low-pass speed has been calculated, the system can calculate the speed reduction value. In some examples, the speed reduction value is a value between 0 and 1 (other boundary values are possible), and the system can generate the speed reduction value based on how the low-pass speed compares to a lower and upper boundary value. If the low-pass speed is below the lower boundary value, the speed reduction value can be set to the 0 boundary value. If the low-pass speed is above the upper boundary value, the speed reduction value can be set to the 1 boundary value. If the low-pass speed is between the two boundaries, a computing system can select a speed reduction value that represents the scaled value of the low-pass speed between the boundaries, e.g.: where the speed reduction value lies between the 0 and 1 boundary values. The calculation of the speed control value can be represented by the following algorithm (Box 246):. • If lowpass_speed < lower_speed_limit, then speed_reduction = 0 • Otherwise, if lowpass_speed >upper_speed_limit, then speed_reduction = maximum_speed_reduction • Otherwise, speed_reduction = maximum_speed_reduction * (low_pass_speed - lower_speed_limit) / (upper_speed_limit−lower_speed_limit)
[0040] With this algorithm, the lower speed limit, high speed threshold, and maximum speed reduction are specified by the user. Example values include a lower_speed_limit = 0.008; upper_speed_limit = 0.016; and maximum_speed_reduction = 1.0.
[0041] In box 250, the computing system calculates a distortion reduction value. The computing system may calculate a distortion reduction value because the compensation transformation may produce excessive non-rigid distortion when applied to a single video frame. In other words, the video stabilization may appear unrealistic, for example, because the distortion caused by stretching the image more in one direction than the other may occur too quickly and appear unusual to a user.
[0042] To calculate the distortion reduction value, the computing system can first calculate the compensation zoom factor by looking for the values for the zoom factors in the H_ compensation matrix as follows: • zoom_x = H_Compensation[1,1]which is the row 1 column 1 element in the H_Compensation matrix • zoom_y = H_Compensation[2,2]which is the row 2 column 2 element in the H_Compensation matrix The zoom factors can be these factors that identify how the transformation stretches the image into one dimension.
[0043] The computer system can then determine the difference between the two zoom factors to determine the degree to which the image is distorted by stretching in one direction more than the other, as follows (Box 252): • Distortion = abs(zoom_x - zoom_y)
[0044] A low-pass filter is applied to the distortion to limit the degree to which the distortion is allowed to change and thus ensure that sudden changes in distortion are minimized using the following formula (Box 254): • Low-pass distortion = Previous low-pass distortion * Distortion damping ratio + Distortion * (1 - Distortion damping ratio) In other words, the algorithm is designed to allow the distortion level to change gradually. In the previous formula, the distortion attenuation ratio is the attenuation ratio for the distortion IIR filter, which is specified by the user. An example value is 0.99.
[0045] By calculating the low-pass distortion, the computing system can calculate the distortion reduction value. In some examples, the distortion reduction value is a value between 0 and 1 (other boundary values are possible), and the system can generate the distortion reduction value based on how the low-pass distortion compares to a lower boundary value and an upper boundary value. If the low-pass distortion is below the lower boundary value, the low-pass reduction value can be set to the 0 boundary value. If the low-pass distortion value is above the upper boundary value, the distortion reduction value can be set to the 1 boundary value. If the low-pass distortion value lies between the two boundaries, a value is selected that represents the scaled value of the low-pass distortion between the boundaries (e.g., where the resulting distortion reduction value lies between the 0 and 1 boundary values).The calculation of the distortion reduction value can be represented by the following algorithm (Box 256):. • If low-pass distortion <Unterer_Tiefpass_Grenzwert, dann Verzerrungs_Reduktion=0 • Otherwise, if Lowpass_Distortion>Upper_Distortion_Limit, then Maximum_Distortion_Reduction • Otherwise, distortion_reduction=maximum_distortion_reduction *(low_pass_distortion−low_distortion_limit) / (upper_distortion_limit−low_distortion_limit) With this algorithm, the lower distortion limit, upper distortion limit, and maximum distortion reduction are all user-specified. Example values include lower distortion limit = 0.001, upper distortion limit = 0.01, and maximum distortion reduction = 0.3.
[0046] In box 260, the computing system reduces the strength of the video stabilization based on the determined speed reduction value and distortion reduction value. To do so, the computing system calculates a reduction value, which in this example is determined as a maximum of the speed reduction value and the distortion reduction value as follows (box 262): • Reduction=maximum(speed_reduction, distortion_reduction) In other examples, the reduction value may be a combination of these two values, taking a portion of each value into account (e.g., the values may be added together or multiplied together and then possibly multiplied by a predefined number such as 0.5). The reduction value may fall below or between the marginal values of 0 and 1, and the closer this reduction value is to 1, the more the computing system can reduce the strength of the image stabilization.
[0047] The computing system can then modify the compensation transformation matrix to create a reduced compensation transformation matrix (Box 264). The computing system can do this by multiplying the compensation transformation matrix by a subtraction of the reduction value from one. In other words, if the reduction value is very close to one (indicating that a large reduction in image stabilization will occur), the values in the compensation matrix can be significantly reduced, as they would be multiplied by a number close to zero. The numbers in the modified compensation transformation matrix are then added to a unity matrix multiplied by the reduction value. An example equation follows: • H_reduced_compensation=one element*reduction+H_compensation*(1−reduction)
[0048] In box 270, the computing system may restrict the compensation so that the resulting video stabilization does not display invalid areas of the output frame (e.g., those areas that lie outside the frame). As parts of the background, since the compensation may deform the image resulting from the image stabilization process, this image may display invalid areas that are substantially outside the image at its edges. To ensure that these invalid areas are not displayed, the computing system may zoom in on the image and crop out sides of the image that might include the invalid areas.
[0049] With respect to the processes of box 270, if the camera is moved rapidly and significantly, the stabilization might lock on displaying an old location because the rapid and significant movement might be filtered out, which could introduce the previously described invalid regions into the display of a stabilized frame. In such a case, the constraint process described below can ensure that video stabilization essentially stops, with the region of the frame that would be displayed if the video were to begin displaying an invalid region fully controlled. This determination regarding whether the stabilization must relinquish control of the frame can begin by initially determining the vertices of the source frame and determining whether these vertices are outside the predefined crop region.A maximum amount of compensation and zooming can be defined as twice the crop ratio, where the crop ratio can be specified by a user (e.g.: to 15% on each side or 0.15 in the following equation): . • Maximum_compensation=crop_ratio*2
[0050] The computing system can then use the H_reduced_compensation matrix to convert the four corners of a unit square in the GL coordinate (x01, y01) = (-1, -1), (x02, y02) = (1, -1), (x03, y03) = (-1, 1), (x04, y04) = (1, 1) into the four vertices (x1, y1), (x2, y2), (x3, y3), (x4, y4). (Note that the video frame does not have to be a unit square; the dimensions of the unit image are mapped to a unit square in the GL coordinate.) More specifically, we use the following formula to convert (x0i, y0i) to (xi, yi): • dzi*[xi,yi,1]'=H_reduced_compensation*[x0i,y0i,1]' In this example, [x0i, y0i, 1]' is a 3x1 vector that is a transpose of the [x0i, y0i, 1] vector. [xi, yi, 1]' is a 3x1 vector that is the transpose of the [xi, yi, 1] vector. zi is a scaling factor.
[0051] The computing system can then determine the maximum amount of displacement in each direction (left, right, up, and down) from the corners of each converted video frame to the edge of the unit square as follows: • maximum_left_shift=1+maximum(x1,x3) • maximum_right_shift=1−minimal(x2,x4) • maximum_upper_displacement=1+maximum(y1,y2) • maximum_lower_displacement=1−minimal(y3,y4) If any of the detected shifts exceed the maximum amount of compensation (which, as described above, is twice the crop ratio and would indicate that the invalid regions are located within the displayed area of the zoomed-in area of the unit square), then the vertices of the unit image are shifted an equal amount away from the edge of the unit square, so that the invalid regions are not displayed. The equations for shifting the vertices follow accordingly: • If maximum_left_shift>maximum_compensation, shift the 4 vertices to the left by the maximum_left_shift−maximum_compensation • If maximum_right_shift>maximum_compensation, shift the 4 vertices to the right by a maximum_right_shift−maximum_compensation • If maximum_upper_displacement>maximum_compensation, move the 4 vertices upwards by a maximum_upper_displacement−maximum_compensation • If maximum_lower_displacement>maximum_compensation, move the 4 vertices down by a maximum_lower_displacement−maximum_compensation. Moving the vertices is a provision that invalid areas would have been displayed even if the display was cropped (Box 272).
[0052] After all the previous translation operations, the four vertices can be named (x1', y1'), (x2', y2'), (x3', y3'), (x4', y4'). The computing system then calculates the constrained compensation transformation matrix H_constrained_compensation, which assigns the four corners of a unit square of the GL coordinate (x01, y01) = (-1, -1), (x02, y02) = (1, -1), (x03, y03) = (-1,1), (x04, y04) = (1,1) to the four constrained vertices (x1', y1'), (x2', y2'), (x3', y3'), (x4', y4'), as follows: • zi'[xi',yi',1]'=H_limited_compensation*[x0i,y0i,1]' In this example, [x0i, y0i, 1]' is a 3x1 vector that is the transpose of the [x0i, y0i, 1] vector. [xi', yi', 1]' is a 3x1 vector that is the transpose of the [xi', yi', 1] vector. zi' is a scaling factor. Given four pairs of points [x0i, y0i, 1]' and [xi, yi', 1]', an example algorithm for estimating the matrix is described in the following computer vision textbook in Algorithm 4.1 (page 91): "Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press (2000), 2nd edition, first published in print 2004, ISBN-13: 978-0-521-54051-3."
[0053] The H_constrained_compensation is then stored as H_previous_compensation, which is used in calculations to stabilize the next frame, as described above with respect to box 230.
[0054] In box 280, the computing system modifies the limited compensation matrix so that the stabilized image is zoomed in to crop the edge. In some examples, the computing system first determines a zoom factor as follows: • Zoom_Factor=1 / (1−2*Crop_Ratio) For example, the calculation system doubles the crop ratio (e.g., by doubling the 15% value of 0.15 to 0.3), subtracts the resulting value from 1 (e.g., to obtain 0.7), and then divides 1 by this result to obtain the zoom factor (e.g., 1 divided by 0.7 to equal a zoom factor of 1.42). The calculation system can then divide certain functions of the limited compensation matrix to zoom the display by a specific amount, as follows: • H_limited_compensation[3,1]=H_limited_compensation[3,1] / Z oom_factor • H_limited_compensation[3,2]=H_limited_compensation[3,2] / Z oom_factor • H_limited_compensation[3,3]=H_limited_compensation[3,3] / Z oom_factor
[0055] In Box 290, the computing system applies the modified constrained compensation matrix to the current frame to create a cropped and stabilized version of the current frame. An exemplary way of applying the constrained compensation matrix (H_constrained_compensation) to the input frame to produce an output image can be described as follows. • z'*[x',y',1]'=H_limited_compensation*[x,y,1]' • [x,y,1]' is a 3×1 vector representing a coordinate in the input frame • [x',y',1]' is a 3×1 vector representing a coordinate in the output frame • z' is a scaling factor • H_constrained_compensation is a 3×3 matrix containing 9 elements: h1, h2, h3 • H_limited_compensation=h4,h5,h6 h7,h8,h9
[0056] In more detail, for each pixel [x, y] in the output frame, find the position [x', y'] in the output frame using the previous transformation and copy the pixel value from [x, y] in the input frame to [x', y'] in the output frame. Another approach is to find the position [x, y] in the input frame for each pixel [x', y'] in the output frame using the inverse transformation and copy the pixel value from [x, y] in the input frame to [x', y'] in the output frame. These operations can be efficiently performed in the computing systems' Graphics Processing Unit (GPT).
[0057] The process described herein for boxes 210 to 290 can then be repeated for the next frame with some of the values from the current frame's processing used for the next frame.
[0058] In various implementations, operations that are performed "in response to" or "as a consequence of" another operation (e.g., a determination or a determination) are not performed if the previous operation was not successful (e.g., if the determination was not carried out). Operations that are performed "automatically" are operations that are performed without user intervention (e.g., intrusive user input). Functions described in this document using the option form may describe optional implementations. In some examples, "sending" from a first device to a second device includes placing, by the first device, data on a network for receipt by the second device, but may not include receiving the data by the second device.Conversely, “receiving” from a first device may include receiving the data from a network, but not sending the data by the first device.
[0059] "Determining" by a computing system may include the computing system requesting another device to perform the determination and provide the results to the computing system. Furthermore, "displaying" or "presenting" by a computing system may include the computing system sending data to cause another device to display or present the referenced information.
[0060] In some implementations, operations are described as being performed on a matrix, which means the operations performed on that matrix or a version of that matrix modified by an operation described in this disclosure or an equivalent thereof.
[0061] Fig.Figure 3 shows a block diagram of computing devices 300, 350 that can be used to implement the systems and methods described in this document, either as a client or server, or as a plurality of servers. Computing device 300 is intended to represent various forms of digital computers such as laptops, desktops, workstations, PDAs, servers, blade servers, mainframes, and other suitable computers. Computing device 350 is intended to represent various forms of mobile devices such as PDAs, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be examples only and are not intended to limit implementations of the inventions described and / or claimed in this document.
[0062] Computing device 300 includes a processor 302, a memory 304, a storage device 306, a high-speed interface 308 connected to memory 304 and high-speed expansion ports 310, and a low-speed interface 312 for connecting to the low-speed bus 314 and the storage device 306. Each of components 302, 304, 306, 308, 310, and 312 are interconnected using various buses and may be mounted on a standard motherboard or in other manner as appropriate. The processor 302 may process instructions for execution within the computing device 300, including instructions stored in the memory 304 or on the storage device 306 for displaying graphical information for a GUI on an external input / output device, such as a display 316, connected to the high-speed interface 308.In other implementations, multiple processors and / or multiple buses may be used, as needed, along with multiple memories and storage types. Multiple computing devices 300 may also be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).
[0063] Memory 304 stores information in computing device 300. In one implementation, memory 304 is a volatile storage unit or units. In another implementation, memory 304 is a non-volatile storage unit or units. Memory 304 may also be in another form of computer-readable medium, such as a magnetic or optical disk.
[0064] Storage device 306 is capable of providing mass storage for computing devices 300. In one implementation, storage device 306 may be or include a computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical drive, a tape device, flash memory, or other similar solid-state storage device, or any number of devices, including devices in a storage network or other configurations. A computer program product may be physically embodied in an information carrier. The computer program product may also include instructions, the execution of which performs one or more methods such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 304, storage device 306, or processor memory 302.
[0065] The high-speed controller 308 manages bandwidth-intensive operations for the computing device 300, while the low-speed controller 312 manages less bandwidth-intensive operations. However, this assignment of functions is only exemplary. In one implementation, the high-speed controller 308 is coupled to memory 304, display 316 (e.g., via a graphics processor or accelerator), and high-speed expansion ports 310 that can accommodate various expansion cards (not shown). In the implementation, the low-speed controller 312 is coupled to the storage device 306 and the low-speed expansion port 314. The low-speed expansion port, which can accommodate various communication ports (e.g.,: USB, Bluetooth, Ethernet, wireless Ethernet) can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner or a network device such as a switch or router e.g. through a network adapter.
[0066] Computing device 300 may, as shown in the figure, be implemented in a number of different forms. For example, it may be implemented as a standard server 320 or multiple times within a group of those servers. It may also be implemented as part of a rack server system 324. Furthermore, it may be implemented in a personal computer (PC), such as a laptop computer 322. Alternatively, components of computing device 300 may be combined with other components in a mobile device (not shown), such as device 350. Each of these devices may include one or more of computing devices 300, 350, and an entire system may be composed of multiple computing devices 300, 350 communicating with each other.
[0067] Computing device 350 includes, among other components, a processor 352, a memory 364, an input / output device such as a display 354, a communications interface 366, and a transceiver 368. Device 350 may also be equipped with a storage device, such as a microdrive or other device, to provide additional storage. All components 350, 352, 364, 354, 366, and 368 are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other ways, as appropriate.
[0068] Processor 352 may execute instructions in computing device 350, including instructions stored in memory 364. The processor may be implemented as a chipset of chips including separate and multiple analog and digital processors. In addition, the processor may be implemented using any number of architectures. For example, the processor may be a Complex Instruction Set Computer (CISC) processor, a Reduced Instruction Set Computer (RISC) processor, or a Minimal Instruction Set Computer (MISC) processor. For example, the processor may provide coordination of the other components of device 350, such as control of user interfaces, applications executed by device 350, and wireless communications by device 350.
[0069] Processor 352 may communicate with a user via control interface 358 and display interface 356 coupled to a display 354. Display 354 may be, for example, a TFT-LCD (thin-film transistor liquid crystal display) or an OLED (organic light-emitting diode) display, or other suitable display technology. Display interface 356 may include suitable circuitry for driving display 354 to display graphical and other information to a user. Control interface 358 may receive commands from a user and translate them for transmission to processor 352. Additionally, an external interface 362 may provide communication with processor 352 to enable short-range communication of device 350 with other devices.For example, the external interface 362 may be a wired connection in some implementations or a wireless connection in other implementations, and multiple interfaces may also be used.
[0070] Memory 364 stores information within computing device 350. Memory 364 may be implemented as one or more of computer-readable media, volatile memory unit(s) or device, or non-volatile memory unit(s). Expanded memory 374 may also be provided and coupled to device 350 via expansion interface 372, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expanded memory 374 may provide additional storage space for device 350 or may also store applications or other information for device 350. In particular, expanded memory 374 may include instructions for performing or supplementing the processes described above, and may also include secure information.Thus, expanded memory 374 may be provided as a security module for device 350 and programmed with instructions that ensure the secure use of device 350. Additionally, secure applications may be provided via the SIMM cards along with additional information, such as storing identifying information on the SIMM card in a manner that cannot be hacked.
[0071] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is physically embodied in an information carrier. The computer program product includes instructions, the execution of which performs one or more methods such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 364, extended memory 374, or memory in processor 352, which may be received, for example, via transceiver 368 or external interface 362.
[0072] Device 350 can communicate wirelessly via communications interface 366, which may include digital signal processing circuitry if desired. Communications interface 366 can provide connections using various communication types or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, through radio frequency transceivers 368. Additionally, short-range communication can occur, such as using Bluetooth, WLAN, or other such transceivers (not shown). In addition, GPS (Global Positioning System) receiver module 370 can provide additional navigation and location-related wireless data to device 350, which may be used by applications executing on device 350, if desired.
[0073] Device 350 may also communicate audibly using audio codec 360, which may receive spoken information from a user and convert it into usable digital information. Audio codec 360 may also generate audible sound for a user, such as through a speaker, for example, in a handset of device 350. Such sound may include sound from voice calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 350.
[0074] Computing device 350 may be implemented in a number of different forms, as illustrated in the figure. For example, it may be implemented as a mobile phone 380. It may also be implemented as part of a smartphone 382, PDA, or other similar mobile device.
[0075] Additionally, the computing devices may include 300 or 350 USB (Universal Serial Bus) flash drives. The USB flash drives can store operating systems and other applications. The USB flash drives may include input / output components, such as a wireless transmitter or USB connector, that can be connected to a USB port on another computer.
[0076] Various implementations of the systems and techniques described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (Application Oriented Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a memory system, at least one input device, and at least one output device.
[0077] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” means any signal used to provide machine instructions and / or data to a programmable processor.
[0078] To enable interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (such as a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including auditory, voice, or tactile input.
[0079] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., a data server), a middleware component (e.g., an application server), or a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with the implementations of the systems and techniques described herein), or a combination of those back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), peer-to-peer (P2P) networks (with ad hoc members and permanent members), network computing infrastructures, and the Internet.
[0080] The computing system may include clients and servers. A client and a server are generally located remotely from each other and typically interact via a communications network. The relationship between client and server arises due to computer programs running on the respective computers, which have a client-server relationship with each other.
[0081] Although some implementations have been described in detail above, other modifications are possible. Furthermore, other mechanisms may be used to carry out the systems and methods described in this document. Furthermore, the logical flows depicted in the figures do not require the particular order or sequential sequence shown to achieve desirable results. Other steps may be provided, or steps may be eliminated from the described processes, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.
Claims
[1] Computer-implemented method for video stabilization, comprising: Receiving, by a computing system (300, 350), first and second individual images (110b, 110c) of a video recorded by a camera, Creating, by the computing system using the first and second frames of the video, a first mathematical transformation (120) indicating movement of the camera with respect to a scene captured by the video from a time at which the first frame (110a) was captured to a time at which the second frame (110b) was captured, comprising movement that began recently and movement that has occurred over a longer period of time, wherein the computing system creates the first mathematical transformation by identifying how feature points present in the first frame and the second frame have moved between the first frame and the second frame, Filtering, by the computing system, the movement of the camera relative to the scene to highlight low-frequency movement that has occurred for the longer period of time, Creating, by the computing system using the first mathematical transformation and the filtered movement of the camera relative to the scene that has occurred over a longer period of time, a second mathematical transformation (130), wherein the second mathematical transformation, compared to the first mathematical transformation, is more representative of movement that has occurred over the longer period of time and is less representative of movement that began recently, and Creating, by the computing system, a stabilized version of the second frame (150) without taking into account information from future frames of the video, the method further comprising: Identifying, by the computing system, an expected distortion that would be present in a stabilized version of the second frame as a result of applying the second mathematical transformation to the second frame based on a difference between: (i) a measure of the distortion in the horizontal direction resulting from the application of the second mathematical transformation to the second frame, and (ii) a measure of the distortion in the vertical direction resulting from the application of the second mathematical transformation to the second frame, Determining, by the computing system, an amount by which a stabilization effect resulting from the application of the second mathematical transformation to the second frame is to be reduced based on a degree to which the expected distortion exceeds an acceptable change in distortion, wherein the acceptable change in distortion was calculated from distortion in several frames of the video preceding the second frame, wherein creating the stabilized version of the second frame comprises reducing a stabilization effect of applying the second mathematical transformation to the second frame based on the determined amount by which the stabilization effect is to be reduced. [2] The computer-implemented method of claim 1, wherein the second frame is a frame of the video immediately following the first frame of the video. [3] The computer-implemented method of claim 1, wherein the mathematical transformation indicating movement of the camera includes a homography transformation matrix. [4] The computer-implemented method of claim 1, wherein creating the stabilized version of the second image frame includes zooming into a version of the second image frame created by applying the second mathematical transformation to the second image frame. [5] The computer-implemented method of claim 4, further comprising shifting a zoomed-in region of the version of the second frame horizontally or vertically to avoid the zoomed-in region of the second frame representing an invalid region. [6] The computer-implemented method of claim 1, wherein filtering the motion of the camera relative to the scene to highlight low-frequency motion that occurred for the longer period of time comprises analyzing previous frames (110a) of video that the camera captured prior to the first and second frames. [7] The computer-implemented method of claim 6, wherein filtering the motion of the camera relative to the scene to emphasize low-frequency motion that occurred for the longer period of time comprises applying a low-pass filter to the first mathematical transformation based on analysis of previous frames (110a) of the video that the camera captured prior to the first and second frames. [8] The computer-implemented method of claim 1, wherein determining the amount by which to reduce the stabilization effect resulting from applying the second mathematical transformation to the second frame is further based on a particular speed of movement of the camera from the first frame to the second frame that exceeds an acceptable change in the speed of movement of the camera calculated based on the speed of movement of the camera between multiple frames of the video preceding the second frame. [9] One or more non-transitory computer-readable devices comprising instructions that, when executed by one or more processors, result in the performance of operations including: Receiving, by a computing system (300, 350), first and second individual images (110b, 110c) of a video recorded by a camera, Creating, by the computing system using the first and second frames of the video, a first mathematical transformation (120) indicating movement of the camera with respect to a scene captured by the video from a time at which the first frame (110b) was captured to a time at which the second frame (110c) was captured, comprising movement that began recently and movement that has occurred over a longer period of time, wherein the computing system creates the first mathematical transformation by identifying how feature points present in the first frame and the second frame have moved between the first frame and the second frame, Filtering, by the computing system, the movement of the camera relative to the scene to highlight low-frequency movement that has occurred for the longer period of time, Creating, by the computing system using the first mathematical transformation and the filtered movement of the camera relative to the scene that has occurred over a longer period of time, a second mathematical transformation (130), wherein the second mathematical transformation, compared to the first mathematical transformation, is more representative of movement that has occurred over the longer period of time and is less representative of movement that began recently, and Creating, by the computing system, a stabilized version of the second frame (150) without taking into account information from future frames of the video, the operations further comprising: Identifying, by the computing system, an expected distortion that would be present in a stabilized version of the second frame as a result of applying the second mathematical transformation to the second frame based on a difference between: (i) a measure of the distortion in the horizontal direction resulting from the application of the second mathematical transformation to the second frame, and (ii) a measure of the distortion in the vertical direction resulting from the application of the second mathematical transformation to the second frame, Determining, by the computing system, an amount by which a stabilization effect resulting from the application of the second mathematical transformation to the second frame is to be reduced based on a degree to which the expected distortion exceeds an acceptable change in distortion, wherein the acceptable change in distortion was calculated from distortion in several frames of the video preceding the second frame, wherein creating the stabilized version of the second frame comprises reducing a stabilization effect of applying the second mathematical transformation to the second frame based on the determined amount by which the stabilization effect is to be reduced. [10] One or more non-transitory computer-readable devices according to claim 9, wherein the second frame is a frame of the video immediately following the first frame of the video. [11] One or more non-transitory computer-readable devices according to claim 9, wherein the mathematical transformation indicating movement of the camera includes a homography transformation matrix. [12] One or more non-transitory computer-readable devices according to claim 9, wherein creating the stabilized version of the second image frame includes zooming into a version of the second image frame created by applying the second mathematical transformation to the second image frame. [13] One or more non-transitory computer-readable devices according to claim 12, wherein the operations further comprise horizontally or vertically shifting a zoomed-in area of the version of the second frame to avoid the zoomed-in area of the second frame representing an invalid area. [14] One or more non-transitory computer-readable devices according to claim 9, wherein filtering the movement of the camera with respect to the scene to highlight low-frequency motion that occurred for the longer period of time comprises analyzing previous frames (110a) of the video that the camera captured before the first and second frames. [15] One or more non-transitory computer-readable devices according to claim 14, wherein filtering the motion of the camera with respect to the scene to emphasize low-frequency motion that occurred for the longer period of time comprises applying a low-pass filter to the first mathematical transformation based on the analysis of previous frames (110a) of the video that the camera captured before the first and second frames. [16] One or more non-transitory computer-readable devices according to claim 9, wherein determining the amount by which to reduce the stabilization effect resulting from applying the second mathematical transformation to the second frame is further based on a particular speed of movement of the camera from the first frame to the second frame that exceeds an acceptable change in the speed of movement of the camera calculated based on the speed of movement of the camera between multiple frames of the video preceding the second frame.
Citation Information
Patent Citations
Method and apparatus for image or video stabilization
US20090066800A1
Method and apparatus for image stabilization
US20110085049A1
Image stabilization method and image stabilization device
US20130121597A1
Image stabilization
US8385732B2
