Digital image stabilization with adaptive filtering

The adaptive filtering technique addresses over-deflections in digital image stabilization by combining strong and weak compensation filters, effectively stabilizing video frames by adjusting filter strength based on deflection history, ensuring smooth and stable output.

DE102011056982B4Active Publication Date: 2024-11-07SAMSUNG ELECTRONICS CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
DE102011056982
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2010-12-23
Filing Date
2011-12-23
Publication Date
2024-11-07
Estimated Expiration
2031-12-23

AI Technical Summary

Technical Problem

Existing digital image stabilization methods often result in over-deflections of compensation windows in captured video frames, leading to jerky and unstable video output, particularly when dealing with high-frequency tremors and unintentional camera movements.

Method used

An adaptive filtering method using a combination of strong and weak compensation filters to balance video stabilization and prevent over-excursions, based on the history of compensation window deflections, employing a strong compensation filter for high-frequency jitter and a weak compensation filter for larger deflections.

Benefits of technology

The method effectively stabilizes video by reducing over-deflections while maintaining smooth output, ensuring that the compensation window remains within the boundaries of the input frame, even with fast-moving scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000018_0000
    Figure 00000018_0000
  • Figure 00000018_0001
    Figure 00000018_0001
  • Figure 00000019_0000
    Figure 00000019_0000
Patent Text Reader

Abstract

A method for processing video data, comprising: Identifying Transformations T i (n) of motion vectors in a video frame recorded by a camera, wherein a compensation transformation C(n) represents a camera motion to be compensated; and Adaptive filtering of the compensation transformation C(n) based on non-linear normalized, time-integrated deflections to prevent excessive over-deflection of a compensation window.
Need to check novelty before this filing date? Find Prior Art

Description

1. Technical area

[0001] The concept of the present invention relates to digital image stabilization (DIS), and more particularly to a method for reducing over-excursions of the compensation windows in recorded video frames.

[0002] From the publication US 2010 / 0 208 087 A1, a video camera with an image stabilization function is known, which distinguishes between intentional and unintentional camera movements from motion vectors identified in the image data in order to compensate only for unintentional camera movements.

[0003] From the document US 2006 / 0 153 472 A1 a method for image stabilization is known which determines several partial motion vectors from different image areas, weights the partial motion vectors with respective confidence levels and combines them into an overall motion vector in order to carry out motion compensation. 2. Description of the state of the art

[0004] Digital cameras, digital video cameras, and portable devices incorporating such cameras are often used to capture images or videos while the camera is operated in the hand of a human operator. Thus, the video camera may experience shake or jitter while capturing the image or video in the user's hand. The jitter may be caused by hand shake or surface vibration, and it may include a horizontal component, a vertical component, a scaling component, and a rotational component. The jitter can make hand-shot video appear jarring or confusing to the viewer, and thus it is desirable to use digital circuitry to digitally estimate a camera trajectory (i.e., the jitter as detected between each pair of consecutive frames) and to filter the jitter from a sequence of video frames of the same scene.

[0005] The video produced by a still video camera, whether stationary or moving, contains primarily smooth movements (translation, rotation) in the recorded video. On the other hand, a shaky video camera produces a video with consistently high-frequency jitter (translational and / or rotational) in the video frames. The video produced by a still video camera, whether stationary or moving, contains primarily smooth movements in the video. On the other hand, a shaky video camera produces videos with consistently high-frequency jitter in the video frames.

[0006] A digital image stabilization (DIS) system first estimates unwanted (unintentional) movements and then applies corrections to the image sequences. Accurate, comprehensive motion estimation (camera trajectory estimation) is key to any video stabilization algorithm. The visual effect of a stabilized video depends largely on the quality of the camera trajectory estimation. The comprehensive motion of the camera / scene is analyzed to distinguish between intended (e.g., panning) and unwanted (shake) comprehensive motion, and a compensation transformation P(n) is created to compensate for the unintentional, jittery motion. The video stabilization algorithm eliminates the jittery motion while maintaining the camera motion intended by the user. Generally, jitter is caused by hand shaking and surface vibration, which can occur faster (i.e.,with higher frequency) and nonlinear, while camera motion is slower and linear or monotonic. The comprehensive motion (camera trajectory) vector is included in the affine transformation parameters of the compensation transformation P(n), which are estimated between adjacent frames using matching feature point pairs.

[0007] The compensating transformation P(n) can be an affine transformation. In mathematics, affine geometry is the study of geometric properties that remain unchanged by an affine transformation, i.e., non-singular linear transformations and translations. A mathematical system of equations defined in terms of numerical coefficients is called an affine matrix and is designed to characterize lateral (up / down), rotational, and scalar (e.g., zooming in or out) motion detected between any pair of consecutive frames, or between portions of them (e.g., moving objects within the frame). Thus, the compensating transformation P(n) for compensating camera shake with respect to any truly stationary object (e.g., rocks, tables, parked cars, mountains, the sun) in the scene can be characterized as the first affine transformation matrix.In almost all cases, hand shaking and surface vibration can lead to translation, rotation, and scaling of video images. To model all of this, a six-parameter affine transformation is required.

[0008] Even if the compensation transform P(n) is correctly constructed to compensate for the unintended jittery motion, the resulting compensation frame may exhibit significant oscillating motion relative to the captured input video frames, and it may exceed the available image data in some of the captured input video frames. This may lead to over-excursion of the compensation window.

[0009] To remove jerky motion in the video, the compensation unit clips some boundary areas from each input video frame. The amount of removed boundary areas can be quantified as the clipping ratio. A large clipping ratio means that more area is removed at the boundaries. An output video frame can be modeled as a compensation window overlaid on the input video frame (see, for example, Fig. 1). The compensation window can be rotated, offset, scaled, etc., relative to the input video frame.

[0010] For a given clipping ratio, the amount of movement of the compensation window is called a compensation window excursion. Movement of the compensation window beyond the boundary of the input video frame is called a compensation window overexcursion.

[0011] If no jitter (no unintended camera trajectory) is present, the compensation transformation P(n) (based on feature points of actually stationary objects) assumes identical positions in two or more consecutive frames (e.g., UNITY). If high-frequency jitter is present, it is desirable to generate a spatially stabilized video with a reduced degree or frequency of over-excursions of the compensation window.

[0012] Therefore, there is a need for a filtering method that creates an adaptive balance between insufficient video stabilization and over-excursion. SUMMARY

[0013] One aspect of the inventive concept provides a digital image stabilization method comprising adaptive filtering of a main / compensation transform P(n) representing a stationary / background object in the scene of a video frame based on a history of over-excursions of the compensation window.

[0014] One aspect of the inventive concept provides a highly effective and predictable method for eliminating jitter by using a strong compensation (SC) filter. The SC filter is a highly frequency-selective, linear, time-invariant, or lag-free, higher-order digital filter. Effective filtering of a very jerky input video using the SC filter implies significant movement of the compensation window through the captured input video frame. For a given clipping ratio, the amount of movement of the compensation window is referred to as the compensation window excursion. Movement of the compensation window beyond the boundary of the captured input video frame is referred to as the compensation window overexcursion.A strict application of the SC filter to a highly moving input video will produce a very balanced output video at the expense of many over-excursions of the compensation window. On the other hand, a weak compensation (WC) filter, which has a lower frequency-selective characteristic, may produce fewer over-excursions of the compensation window at the expense of a less balanced output video.

[0015] One aspect of the inventive concept provides an adaptive compensation AC filter configured to prevent excessive over-excursion with a high-motion input video while maintaining excellent video stabilization characteristics.

[0016] In an exemplary embodiment of the inventive concept, a linear time-invariant filter comprising a WC filter complementing an SC filter is caused to generate a predictable characteristic. The combined WC / SC filter can be controlled based on the history of excursions of the compensation window over a plurality of K frames. Small excursions from the history allow a strong influence of the SC filter for the current frame, whereas large excursions from the history justify a strong influence of the WC filter for the current frame n. Medium excursions from the history determine proportional influences of the SC filter and the WC filter.

[0017] Another aspect of the invention provides a digital image stabilization circuit adapted to perform the DIS methods disclosed herein. The circuit may be incorporated into a video camera itself, and may be used to correct the jitter prior to storing the captured video frames in real time (e.g., before or during MPEG encoding if the video camera includes a real-time MPEG encoder).Alternatively, the DIS circuit used to estimate the camera trajectory between successive video frames and to filter out jitter from a stored sequence of video frames may be a general-purpose microcomputer controlled by software implementing a digital image stabilization (DIS) technique, or it may be dedicated hardware, such as an MPEG video encoder implemented in an ASIC (application-specific integrated circuit) optimized to perform a digital image stabilization (DIS) technique.

[0018] The methods for calculating the main / compensation P(n) transform from observed feature points in the scene are described in detail in co-pending and currently filed applications entitled "DIGITAL IMAGE STABILIZATION DEVICE AND METHOD" Application No. 8729-357, and entitled "DIGITAL IMAGE STABILIZATION" Application No. 8729-358, and the disclosures of these co-pending applications are incorporated herein by reference. A feature of the exemplary embodiments of the inventive concept is the continuous storage of the compensation window over-excursion history data for evaluating the calculated main / compensation transform P(n) and selecting a filtering strength for outputting a filtered main reference / compensation P'(n) optimized to reduce over-excursions.The method simultaneously measures vertical-top, vertical-bottom, horizontal-right and horizontal-left over-deflections and records the peak values ​​among them to select the filtering coefficient E(n).

[0019] Hereinafter, exemplary embodiments of the inventive concept will be described in detail with reference to the accompanying drawings. The inventive concept may, however, be embodied in various forms and should not be construed as limited to the illustrated embodiments. Rather, these embodiments are provided so that this disclosure will be clear and complete, and will fully convey the inventive concept to those skilled in the art. Like numerals refer to like components throughout. BRIEF DESCRIPTION OF THE DRAWING

[0020] The accompanying drawings are included to facilitate a better understanding of the inventive concept and are incorporated in and constitute a part of this description. The drawings illustrate exemplary embodiments of the inventive concept and, together with the description, serve to explain the principles of the inventive concept. In the figures: Fig. 1 is a block diagram of a digital image stabilization (DIS) circuit performing a DIS method in accordance with an exemplary embodiment of the inventive concept; Fig. 2 a block diagram of a detection unit in the DIS circuit of Fig. 1, which is adapted to compute affine transformations of tile vector groups; Fig. 3 a block diagram of a trajectory unit (TU) of the DIS circuit from Fig. 1, which is adapted based on scores of the tile group transformations and feature group transformations T i (n) select the main (stationary / background) transformation P(n); Fig. 4A is a block diagram of an exemplary implementation of a group transformation scoring and selection circuit configured to perform a step in the DIS method of the DIS circuit of Fig. 1 to be carried out; Fig. 4B is a block diagram of an exemplary implementation of a history scoring calculation unit; Fig. 5 is a block diagram of an exemplary implementation of a collective transformation evaluation and selection circuit; Fig. 6 is a block diagram illustrating an exemplary embodiment of a moving object exclusion circuit; Fig. 7 is a flowchart illustrating processing steps according to an embodiment of the present inventive concept; Fig. 8 is a view of a captured video frame and a compensation window calculated therein in a step of a digital image stabilization (DIS) method according to an exemplary embodiment of the inventive concept; Fig. 9 is a block diagram of a digital image stabilization (DIS) module that performs DIS; Fig. 10 is a block diagram of a method of an adaptive compensation filter module in accordance with an exemplary embodiment of the inventive concept; and Fig. 11 is a schematic block diagram of an adaptive compensation filter module in accordance with an exemplary embodiment of the inventive concept. DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

[0021] According to one embodiment of the present inventive concept, each captured video frame is divided into a small number of non-overlapping tiles (typically 4x4 tiles for standard resolution and 6x6 tiles for high resolution) with the objective of algorithmically selecting feature points that provide a good feature point distribution and are suitable for digital image stabilization. Different regions of the image may have a different density of suitable feature points. In extreme cases, a region of the frame may not have a single suitable feature point, for example, in the case of a blue sky without a single cloud. In other regions, the potential feature points may be present in high density.

[0022] The obtained feature point distribution is based on small regions of the video frame (e.g., non-overlapping tiles), where the number of feature points in each tile increases with the deviation σ 2 the luminance of the image data of the tile increases linearly. Tiles with more interesting image data and with an associated requirement of more feature points exhibit, as expected, a higher deviation σ 2 See the co-pending application, which describes a procedure that sets a minimum distance (MIN_DIST) between feature points in each tile while requiring little local state information, thereby reducing the cost of the hardware implementation. The disclosure of 8729-357 is incorporated herein by reference.

[0023] If the scene was captured in low-light conditions, it will have relatively more noise, and the noise effect will affect the feature points more than the tiles, because the number of pixels in the feature points is much smaller than the number of pixels in the tile. In this case, the larger the number of pixels in the tile that provide the noise cancellation effect, the more accurate the downsampled tile-based motion vector will be.

[0024] Even if the scene is not captured under low light conditions, the tile-based motion vector may be more accurate if the scene is too flat. When the tiled scene is very flat, such as a cloudy sky or a blue sky, there may be some feature points, and these feature points in the flat tile may find similar levels of matching points at many locations in the next frame. However, tile-based matching does not rely solely on a small feature point search area, and any pattern in the tile can contribute to the tile matching process. Therefore, tile-based motion vectors are more reliable when the scene is flat.

[0025] If the best score of the feature point motion vector groups is smaller than a given threshold, it is decided to use tile-based motion vectors instead of feature point-based motion vectors, and this strategy works well for scenes with high noise or flat scenes.

[0026] According to one embodiment of the inventive concept, the estimated motion vectors are chosen to represent the motion of the background and large objects, whereas smaller objects must have an accurate motion vector associated with them. Any inaccurate vector of a smaller object can be filtered out in a later section of the DIS algorithm.

[0027] The feature points of large, stationary significant objects are assumed to move along a coherent path due to either extensive motion or camera movement. It has been found that sufficiently large objects moving independently cover the majority of at least one tile, so their motion can be considered the dominant motion of the tile itself, while the motion of small objects has little effect on the motion vector of the tile itself.

[0028] Using lowest-resolution block matching, a motion vector was derived per tile. The tile-based motion vectors can be used for camera trajectory decisions in some special cases, such as high-noise or flat scene videos.

[0029] The motion vector for a given tile is the one that minimizes the sum of absolute differences (SAD). The procedure for calculating the motion vectors of the feature points in each tile can be modified to simplify the computation by using a hierarchical motion estimation algorithm and favoring tile motion over local motion using the tile's motion vector as a starting vector. Since a sufficiently large object that covers most of at least one tile may extend into neighboring tiles, it is likely that some feature points in each tile will be more strongly related to the motion vector of a neighboring tile than to the motion vector of the tile within which they are found.Thus, it would be effective to use the motion vectors of all neighboring tiles as multiple seed vectors in the block-matching search for the motion vector of the feature points of any given tile. Thus, the seed vectors used to obtain motion vectors of the selected feature points are both those of the tile to which the feature point belongs and those belonging to the four directly neighboring ones (top tile, left tile, right tile, bottom tile), if any. For each seed vector used, only a very small area is used for the local search for feature point motion vectors. The goal here is not so much to determine an exact vector for each feature point (poor motion vectors are discarded later in the DIS sequence).Rather, the interesting feature points are merely those that belong to the background or large objects. For such feature points, one of the tile motion vectors should be good, or close to the motion vector of the interesting feature points, and therefore a small local search around each of the selected tile motion vectors is sufficient. A small local block matching search is performed in a higher resolution field (this can be the original video resolution or one enhanced by a factor f). s3 or 2 or 4 downsampled) around each set of seed vectors, for each selected feature point in the tile.

[0030] Fig. Figure 1 is a block diagram of a digital image stabilization (DIS) circuit that performs a DIS method in accordance with an exemplary embodiment of the inventive concept. The DIS circuit includes a detection unit (DU) that analyzes a received jerky video and performs cross-frame transformations T i (n), a trajectory unit (TU) that outputs a selected main / compensation transformation P(n) selected from the cross-frame transformations T i (n) is selected, and a compensation unit (CU) that outputs a stabilized video by modifying the jerky video using the selected main / compensation transform P(n).

[0031] The detection unit (DU) estimates cross-frame motion vectors of feature points (FP) and cross-frame motion vectors of non-overlapping tiles (tile vectors) in a received frame of video data. The detection unit further outputs FP motion vector group transforms and tile vector group transforms T i (n) out.

[0032] The trajectory unit (TU) selects one of the cross-frame transformations T i (n) (or a uniformity transformation in a case where a large moving object covers the scene) as the main transformation P(n), thus excluding cross-frame transformations of small moving objects and a large moving object that can move into the frame and completely cover it.

[0033] Fig. 2 is a block diagram of the detection unit 2000 in the DIS circuit of Fig. 1, which is adapted to perform affine transformations of tile vector groups as steps of the DIS method of the DIS circuit from Fig. 1. The detection unit 2000 includes a feature point circuit 3000, a motion vector (MV) group circuit 1300, and a motion vector (MV) group affine transform calculator 2010.

[0034] The feature point circuit 3000 receives each frame of video data and preferably divides each video frame into a smaller number jxk of non-overlapping tiles. The number jxk of tiles can range from 4x4 for SD videos to 6x6 for HD videos; other numbers in the range (4...8)x(4...8) are also possible and may be advantageous. The tile size is chosen such that sufficiently large objects moving independently cover the majority of at least one tile so that their motion can be captured for DIS purposes, while the motion of small objects can be ignored. A feature point circuit 3000 identifies and selects feature points (SFPs) in a received video frame and outputs the motion vectors of the feature points and the tiles (SFP MVs and tile MVs).

[0035] The feature point circuit 3000 includes a feature point selector and a motion vector calculator, and a shared RAM 350. The feature point selector 300 may further include a Harris Corner feature point candidate identifier and a feature point candidate sorter. To save computational power and reduce the number of required operations, the feature point circuit 3000 operates only on luminance data and includes one or more downsamplers and a hierarchical block matching search unit.

[0036] The feature point circuit 3000 estimates a motion vector for each tile. The tile motion vector (tile MV) estimation is performed based on non-overlapping tiles that cover the center of the input image (e.g., the same tiles that might be used in a feature point sorting algorithm). For each of the tiles, a full-search block match is performed on a heavily downsampled image. A full-search block match is performed for each tile, and the tile motion vector (tile MV) is stored (356) for later use, e.g., as a seed vector in the hierarchical block match search unit for deriving the feature point motion vector (SFP MV stored at 352) and for stationary object detection.

[0037] The feature point circuit 3000 preferably provides a list of feature points 352 having a distribution based on small regions (tiles) of the video frame called tiles, with the maximum number of feature points per tile having the deviation σ 2 of the tile's luminance image data increases linearly. Good feature points for the DIS method are points that yield non-ambiguous motion vectors when a suitable motion estimation algorithm is applied. To identify feature points in an image, a Harris Corner detection algorithm is applied to the pixels of a video frame to measure how well that pixel qualifies as a feature point. Different areas (tiles) of the image may have a different density of identified feature point candidates.

[0038] The feature point circuit 3000 preferably includes a motion vector calculator that performs the function of a tile calculator for calculating a motion vector of each tile, and a hierarchical block match search unit for determining and outputting the motion vector of each selected feature point (SFP). The tile vector calculator calculates the motion vector of each tile using heavily downsampled luminance data of the current frame Ft. The hierarchical block match search unit determines the motion vector of each selected feature point using the fully resolved or downsampled luminance data of two consecutive frames, and it may use the tile vectors as seed vectors.

[0039] All feature point and tile related data are passed to the next DIS block, specifically the motion vector group circuit 1300.

[0040] The motion vector grouping circuit 1300 is configured to perform the grouping algorithm on the FP motion vectors and the tile motion vectors. The feature point grouping circuit 1300 includes a motion vector comparator 1310 configured to make grouping decisions by comparing each pair of vectors selected by the pairing algorithm controller 1302.

[0041] Feature point grouping circuitry 1300 groups FP motion vectors to associate the motion vectors of the selected feature points (SFPs) with objects in the scene based on the perceived relative motion for an object between consecutive video frames. Feature point grouping circuitry 1300 also groups tile motion vectors to associate the tile vectors with objects in the scene based on the perceived relative motion for an object between consecutive video frames.

[0042] Feature point grouping circuit 1300 shares RAM 350 with feature point circuit 3000. SPF MV list section 352-FP of memory 350 contains the list of positions and motion vectors of the selected feature points (SFPs). Tile MV list section 352-TMV of memory 350 contains the list of positions and motion vectors of the non-overlapping tiles.

[0043] The pairing algorithm controller 1302 tracks which feature points and tiles (motion vectors) have already paired with which others, which remain unpaired, and which are completely excluded from grouping. The pairing algorithm repeatedly provides pairs of motion vectors (vector A and vector B) as inputs to the MV comparator 1310.

[0044] The pairing algorithm controller 1302 in the feature point grouping circuit 1300 accesses the SPF MV list (352-MV) and the tile MV list (352-TMV) and selects vectors A and vectors B for comparison in the motion vector comparator 1310. When series of vector A-vector B comparisons result in one or more groups of vectors (e.g., groups of selected feature points and groups of tiles), the pairing algorithm controller 302 writes the grouped motion vectors, or a representative list thereof, to the FP-MV group inventory section 354 and the tile MV group inventory section 358 of the memory 350.

[0045] The Motion Vector (MV) Group Affiner Transformation Calculator 2010 calculates the cross-frame transformation of each group of feature point motion vectors, and it calculates the cross-frame transformation of each group of tile motion vectors and outputs all of them as Ti (n) out.

[0046] Fig. 3 is a block diagram of the trajectory unit (TU) 4000 of the DIS circuit of Fig. 1, which is adapted to calculate the main (stationary / background) transformation P(n) based on a method of scoring tile group transformations and feature group transformations T i (n) in accordance with steps of the DIS method of the DIS circuit Fig. 1 to select.

[0047] The trajectory unit (TU) 4000 ( Fig. 3) has a tile group transformation evaluation and selection circuit 4100-1 ( Fig. 4A), a feature group transformation evaluation and selection circuit 4100-2 ( Fig. 4A), a collective group selection circuit 4200 ( Fig. 5), a moving object exclusion circuit 4400 ( Fig. 6) and an adaptive compensation filter.

[0048] The trajectory unit (TU) 4000 identifies the main motion P(n) caused by a statically held camera while ignoring moving objects in the scene, filters and selects a main transformation P(n), and outputs the compensation transformation C(n). The trajectory unit (TU) 4000 employs a plurality of continuous weighting functions to compute the main transformation P(n) from received cross-frame transformations T i (n) to select.

[0049] Fig. 4A is a block diagram of the group transformation evaluation and selection circuit 4100 of the trajectory unit (TU) 4000 of the DIS circuit of Fig. 1, comprising a transform evaluation calculator 4150, a transform quality calculator 4160, and a group transform and quality selector 4170, which are designed to perform a step in the DIS method of the DIS circuit of Fig. 1. The group transformation evaluation and selection circuit 4110 is adapted to perform the tile group main transformation GP tile (n) from the tile group cross-frame transformations T tile,i (n) (4100-1) and is adapted to output the feature group main transformation GP FP (n) from the FP cross-frame transformations T FP, i (n) to be issued (4100-2).

[0050] Fig. 4B is a block diagram of an exemplary implementation of the history scoring calculation unit 4110-1 in the group transformation scoring and selection circuit 4100 of Fig. 4A, which in the implementation 4100-1 of the same from Fig. 4 is shown.

[0051] Regarding the Fig. 4A and Fig. 4B, the group transformation evaluation and selection circuit 4100 comprises a history evaluation calculation unit 4110 (e.g., 4110-1), a motion evaluation calculation unit 4120, a feature evaluation calculation unit 4130, and an extension evaluation calculation unit 4140, as well as an overall transformation evaluation SI i (n) calculator 4150, a transformation quality Q i (n) calculator 4160 and a group transformation and quality selector 4170 (e.g. 4170-1).

[0052] The group transformation and quality selector 4170 of the group transformation evaluation and selection circuit 4110 of Fig. 4A selects one of the cross-frame transformations T i (n) as the group main transformation GP(n) (by discarding the cross-frame transformations of small moving objects) based on the total transformation score S i (n) from (from each cross-frame transformation Ti (n)) received from the overall transformation evaluation calculator 4150, and outputs the group principal transformation GP(n) and the associated quality Q(n).

[0053] T i (n) be the i th Transform from all received transform candidates received by the detection unit (DU) 2000, where n denotes a frame and the time sequence type. Let GP(n) be the selected group principal transform at frame time n, i.e., GP(n) = T i (n) for a selected i.

[0054] For each T i (n), the overall transformation score Si(n) calculator 4150 receives from the history score calculation unit 4110 (e.g., 4110-1) a history score H i (n), it receives a movement score M from the movement score calculation unit 4120 i (n), it receives a feature score F from the feature score calculation unit 4130 i(n) and receives an extension score E from the extension score calculation unit 4140 i (n), and it calculates the overall transformation score S i (n) based on the following equation: Si(n)=Hi(n)∗Mi(n)∗Fi(n)∗Ei(n).

[0055] For each T i (n) receives the transformation quality Q i (n) calculator 4160 from the feature evaluation calculation unit 4130 a feature evaluation F i (n), and receives an extension score E from the extension score calculation unit 4140 i (n), and it calculates the transformation quality Q i (n) based on the following equation: Qi(n)=Fi(n)∗Ei(n).

[0056] The T i (n) which has the largest value S i (n) is to be selected by the group transformation selector 4170 of the group transformation evaluation and selection circuit 4100 from Fig. 4A as the group main transformation GP(n). Thus, in the exemplary embodiment, the cross-frame transformation candidate T i (n) with the highest rating S i (n) is selected as the group main transformation GP(n) and then in the DIS compensation unit (CU) 6000 of the DIS circuit from Fig. 1 to generate the compensation transform C(n) to compensate for shaky camera movements.

[0057] The history evaluation calculation unit 4110 (e.g. 4110-1) stores the history of the group main transformation GP(n) and calculates a history evaluation H i (n) for each T i (n), e.g. in a row, if each T i(n) is obtained from the acquisition unit (TU) 2000 based on a predetermined length HL of the stored history of the group main transform GP(n), where HL is an integer indicating a predetermined number of previous frames. The incoming T i (n) is mathematically compared with each of the previously selected group principal transformations GP(n-1)...GP(nk) stored in HL, where k is the integer frame time index ranging from one (denotes the immediately preceding frame: n-1) to HK (the frame further away in time: n-HK). Transformations from T i (n) that have a higher correlation with the HL stored, previously selected group principal transformations GP (n-1) to GP (n-HL) have a higher trend score H i (n).

[0058] The correlation H i,k (n) between T i (n) and each GP (nk) is one unit minus the normalized norm (1 - |T i(n) - GP(nk)|) in the range of [0,1], where a value of H i,k (n) = 1 indicates the highest correlation.

[0059] The contribution of each correlation H i,k (n) (1 - |Ti(n) - GP(nk)|) is weighted by corresponding gradient weights HW(k).

[0060] The progress rating H i (n) is the total correlation and it is the HW(nk) weighted sum of H i,k (n) for 1 < k < HL, where HL is the length (number of last frames) of the history. Thus, Hi(n)=∑[1−|Ti(n)−GP(n−k)|∗HW(k)], for 1 <k<HL.

[0061] The weights from HW(n-HL) to W(n-1) are preferably selected so that their sum is equal to 1 and that the output history score H i (n) is non-linearly normalized and has a continuous range of [0,1].

[0062] The exemplary hardware implementation 4110-1 of the history evaluation calculation unit 4110 shown in Fig. 4B, comprises a FIFO (First-In-First-Out) memory buffer for storing HL previously selected group main transformations GP(n-1) to GP(n-HL), which have HL punctures (for n-1 to n-HL) for outputting their stored contents to a comparator 4114. The comparator 4114 compares the current T i (n) with each of the HL stored, previously selected group principal transformations GP(n-1) to GP(n-HL) and outputs each comparison weighted by history weights HW(n-1) to HW(n-HL) to the overall history score calculator 4116, which calculates the overall correlation as the overall history score H i (n) in the continuous range [0,1].

[0063] The movement score calculation unit 4120 receives each T i (n) and calculates their movement rating N i (n) based solely on the T i(n). In alternative embodiments, the motion score calculation unit 4120 may be configured to receive stored information from the detection unit 2000 for the purpose of calculating the motion score M i (n). Transformations with small movements have a higher movement score M i (n) and are more likely to be the group principal transformation GP(n). The movement score calculation unit 4120 calculates the movement score N i (n) for each cross-frame transformation from T i (n).

[0064] An M i (n), which has a large value, corresponds to a small movement and vice versa. A movement M i (n) can be based on the horizontal, vertical, or overall linear displacement of the transformation. The movement rating M i(n) is inversely related to the linear displacement and is preferably non-linearly normalized to have a continuous range [0,1].

[0065] The feature evaluation calculation unit 4130 receives each T i (n) and calculates their feature value F i (n) based only on T i (n). In an alternative embodiment, the feature score calculation unit 4130 may be configured to use stored information from the acquisition unit 2000 for the purpose of calculating the feature score F i (n). The feature score calculation unit 4130 calculates a feature score F i (n) for each cross-frame transformation from T i (n). A characteristic value F i (n) correlates with the number of feature points grouped together to form the feature point group obtained by each cross-frame transformation from T i(n). Transformations from T i (n) that have more feature points per group have a higher feature score F i (n). A characteristic value F i (n) is preferably non-linearly normalized and has a continuous range of [0,1].

[0066] The extent evaluation calculation unit 4140 receives each T i (n) and calculates their extension value E i (n) based only on T i (n). In alternative embodiments, the extent score calculation unit 4140 may be configured to use stored information from the detection unit 2000 for the purpose of calculating the extent score E i (n). The extent score calculation unit 4140 calculates an extent score E i (n) for each cross-frame transformation from T i (n). Transformations from T i(n) that have feature points that cover larger areas (distributed) have a higher score. An extent score E i (n), which has a larger value, corresponds to a larger covered area and vice versa. An extension rating E i (n) correlates with the height times width of the rectangular area containing all feature points of the transformation group. The extent score E i (n) is preferably non-linearly normalized and has a continuous range of [0,1].

[0067] Various exemplary embodiments of the inventive concept use scene history analyses to exclude large objects moving throughout the scene, which would otherwise cause undesirable results in video stabilization. Without clear scene history analyses, it is likely that a main transformation selector will select the transformation candidate that corresponds to the size of the moving object, especially if it covers the entire scene. It has been recognized that the transformation candidates T i (n) do not include a principal transformation P(n) corresponding to the camera being held still when a large object moves through and fills the entire scene.

[0068] Fig. 5 is a block diagram of an exemplary implementation of the collective transformation evaluation and selection circuit 4200 of the trajectory unit (TU) 4000 of the DIS circuit of Fig. 1, which comprises a collective decision calculator 4250 configured to calculate a collective decision CD(n), and a collective transformation selector 4260 configured to select a collective main transformation CP(n) as steps of the DIS method of the DIS circuit of Fig. 1 to be issued.

[0069] The collective decision calculator 4250 in Fig. 5 calculates a collective decision CD(n) from the feature group transformation quality Q FP (n), the tile group transformation quality Q tile (n), and the number of feature group transformation candidates K FG (n) obtained from the registration unit (DU) 2000.

[0070] An exemplary implementation of the collective decision calculator 4250 includes calculating a non-linearly normalized apportionment measure θ F (n) from the number of feature groups K FG (n), such that θ F(n) is 0 if K FG (n) is small, and θ F (n) is 1 if K FG (n) is large. Thus, a θ F (n) value close to 1 indicates that all feature points in the video scene are divided into many feature groups and vice versa.

[0071] The collective decision calculator 4250 outputs a collective decision CD(n) by comparing Q F (n) and θ F (n)*Q T (n), and if Q F (n) > Θ r (n)*Q T (n), the collective decision CD(n) is set to select a feature group. And if Q F (n) <= Θ F (n)*Q T (n), the collective decision CD(n) is set to select a tile group. In this formulation, Θ F (n) is close to 0 if the feature groups are not partitioned, and the feature group is more likely to be selected. On the other hand, if the feature groups are partitioned, OF (n) close to 1 and the tile group transformation quality Q Tile (n) is calculated on the same basis with the feature group transformation quality Q FP (n) compared.

[0072] The collective transformation selector 4260 performs a selection between the feature group main transformation GP FP (n) and the tile group main transformation GP Tile (n). The collective transformation selector 4260 is controlled by the collective decision CD(n) such that the output collective main transformation CP(n) is mapped to a feature group main transformation GP FP (n) is set when CD(n) is set to a feature group, and on the other hand to a tile group main transformation GP Tile (n).

[0073] In this embodiment, the collective transformation evaluation and selection circuit 4200 performs selections based on a feature group transformation quality QFP (n) and a tile group transformation quality Q Tile (n). The group transformation qualities are calculated by the transformation quality calculator 4160 from Fig. 4A, which receives inputs from the feature score calculation unit 4130 and the extent score calculation unit 4140.

[0074] The feature score calculation unit 4130 calculates its feature score F i (n) from feature-based and tile-based transformations T i (n). In this embodiment, the feature score calculation unit 4130 is configured to receive stored information of the detection unit 2000 for the purpose of calculating the feature score F i (n). For each cross-frame transformation from T i (n) the feature score calculation unit 4130 calculates a feature score F i (n). Transformations T i(n) with more feature points in a group or more tiles in a group have a higher feature score F i (n) and each result in a higher feature group transformation quality Q FP (n) or tile group transformation quality Q Tile (n). In some embodiments, the number of feature points per tile for the feature score F i (n) of the feature-based transformations T i (n) may be decisive. In other embodiments, the number of tiles for the feature score F i (n) of the tile-based transformations T i (n) may be relevant. The number of feature points per tile and / or the number of tiles in each group of tile vectors may be obtained directly from the detection unit 2000.

[0075] The extension score calculation unit 4140 calculates the extension score E i(n) of the feature-based and tile-based transformations T i (n). In this embodiment, the feature score calculation unit 4130 is configured to receive stored information from the detection unit 2000 for the purpose of calculating the feature score F i (n). Transformations with feature points or tiles that cover a larger area are scored higher. The number of features and the dimensions of the tiles in each group of tile vectors can be obtained directly from the acquisition unit 2000. Similarly, the horizontal and vertical extent of each group of feature-based motion factors can be obtained directly from the acquisition unit 2000. Feature groups that cover a larger area, or a tile group that covers a larger area, have a higher extent score E i (n) and results in a higher feature group transformation quality Q FP(n) or a tile group transformation quality Q Tile (n). In this embodiment, the extent score calculation unit 4140 is configured to receive stored extent information from the detection unit 2000 for the purpose of calculating the extent score E i (n) to be calculated.

[0076] After the collective main transform CP(n) has been selected by the collective transform evaluation and selection circuit 4200, the large object exclusion hardware decides whether or not the selected collective main transform CP(n) is a large moving object moving into and completely covering the scene. If such exclusion occurs, a uniformity transform (UT) is created as a substitute and to serve as the selected main transform P(n) for the compensation circuit of the DIS system, so that the stabilized video does not erroneously or unnecessarily follow the transformation of the large moving object.

[0077] According to an embodiment of the present inventive concept, the moving object exclusion method is activated based on two observations: a previously existing stationary background (indicated by the history of P(n)); and a duration of co-existence of the stationary background and the large moving object.

[0078] The moving object exclusion technique can be highly effective in dealing with the following scenario: The scene has a predominantly stationary background with or without moving objects; a large moving object enters the scene and progressively covers a larger area; the large moving object covers the entire scene; the large moving object begins to leave the scene and the background begins to reappear; the large moving object may have moved away.

[0079] The Moving Object Analyzer detects the exclusion scenario if: the existence of a consecutive stationary MV group indicates that a scene with a predominantly stationary background existed; an increasing number of consecutive MV groups with similar speed indicates that an object is moving into the scene; the tendency continues, and at the time n at which the successive MV groups with similar speed cover the entire scene and the existence of the stationary MV group disappears, the exclusion scenario is detected.

[0080] The exclusion decision ED(n) is sent to the exclusion transform selector. The exclusion transform selector selects the collective principal transform CP(n) until ED(n) indicates the exclusion scenario, upon which the principal transform P(n) is set to the uniformity transform. Thus, the stabilized video does not erroneously track a large moving object, even if it covers the entire scene.

[0081] Fig. 6 is a block diagram of an exemplary implementation of the moving object exclusion circuit 4400 of the trajectory unit (TU) 4000 of the DIS circuit of Fig. 1, comprising a moving object analyzer 4470 and an exclusion transform selector 4480 configured to perform a step in the DIS method of the DIS circuit of Fig. 1 to be carried out.

[0082] The moving object exclusion circuit 4400 comprises a plurality of group history circuits 4410, 4420, 4430, 4440 for storing a history of a scene and a moving object analyzer 4470. At any given time, only a specific stationary group G 0 However, 0 or more movement groups G k where k>0. There may also be a new movement group GN which, during the next frame, replaces one of the k (e.g., k(n+1)=k(n)+1) existing movement groups G k becomes.

[0083] The inpatient group G 0 has an assigned group history GH 0 . Each of the k existing motion groups G k has both an assigned group history GH k as well as an associated motion vector M k Each standing movement group G K has a motion vector M K, which over a time up to a frame n for each similar speed T i (n) is essentially low-pass filtered |T i (n)|.

[0084] Each new movement group G N has an assigned group history GH N (n), which is preset at the time of creation. The moving object analyzer 4470 receives the scene history, which is composed of the plurality of group history GH 0 (n), GH 1 (n), ... GH J (n), and GH K (n) and GH N (n) and calculates an exclusion decision ED(n) from these.

[0085] The exclusion transformation selector 4480 performs selections between the uniformity transformation (UT) and the collective main transformation CP(n). The exclusion transformation selector 4480 is controlled by the selection decision ED(n) such that the output main transformation P(n) is set to the uniformity transformation (UT) when the ED(n) is activated, and otherwise to the collective main transformation CP(n). A uniformity transformation (UT) causes the compensation unit to perform nothing during compensation. Thus, when the moving object analyzer 4470 detects the "large moving object" scenario and activates the exclusion decision ED(n), the large moving object transformation, which could otherwise be selected as the main transformation P(n), is excluded from being selected as the main transformation P(n).The transformation of a large moving object is practically excluded from the compensation performed by the compensation unit 6000 when it is detected.

[0086] Fig. 7 is a hybrid block diagram flowchart showing details of the moving object exclusion circuit 4400 of Fig. 6, which is designed to carry out steps in the DIS method from the DIS circuit of Fig. 1 to be carried out. Fig. Figure 7 shows details of illustrative group history circuits 4410, 4430 and 4440, each corresponding to the stationary group G 0 , the existing movement group G K and the newly formed movement group G N+1 are equivalent to.

[0087] Each of the group courses (e.g. H 0 (n)) which are determined by the moving object analyzer 4470 of the moving object exclusion circuit 4400 from Fig. 6 received from the group history circuits (e.g. 4410) includes two types of history data, selection history (e.g. SH 0 (n)) and course of existence (e.g. EH 0 (n)) corresponding to each group.

[0088] The Moving Object Analyzer 4470 detects the exclusion scenario as follows: Consecutive existence and selections occurring in the group history GH 0 of the stationary transformation G0(n) indicate that a scene with a predominantly stationary background has existed for a number of frames; a progressively increasing number of consecutive existences in the group history GH K a specific movement group G K indicates that an object is moving into the scene; if this tendency of existences and movement persists, and if at time (n) no stationary transformation unites with G0, but the selected transformation P(n) with G Kunited, the large moving object scenario is detected and the exclusion decision ED(n) made is sent to the main transformation selector 4160-2. If the ED(n) indicates the large object exclusion scenario, the main transformation P(n) is set as the uniformity transformation; otherwise, the main transformation P(n) is selected according to some of the evaluation functions from T i (n) selected.

[0089] Each of the group history circuits 4410, 4420, 4430, 4440 performs storage and processing of three types of history information for each group, which is associated with one of the received inter-frame transforms T i (n) is related. The three types of group trajectories are selection trajectories, existence trajectories, and movement trajectories. The stationary group G 0 is created at the beginning of a video stabilization with an empty gradient. The motion gradient of the stationary group G 0can be omitted and assumed to be 0.

[0090] The movement groups (G 1 , ..., G K , ... , G N ) are dynamically created or deleted during the course of DIS video processing.

[0091] In relation to Fig. 7 represent the group progression circuits 4410, 4430 and 4440, each of which corresponds to the stationary group G 0 , N existing movement group G K and the newly formed movement group G N+1 correspond to the group progressions GH 0 , GH K and GH N+1 ready.

[0092] The group progression circuit 4410 of the movement group G 0 includes a course 0 -Memory for saving the selection history SH 0 and the course of existence EH 0 . The course of existence EH 0 is a value of one bit per last frame that indicates whether a cross-frame transformation T i(n) with the movement group G 0 in the previous framework or not. The selection process SH 0 is a value with one bit per last frame that indicates whether the cross-frame transformation T i (n) who are involved with the movement group G 0 was selected as the main transformation P(n) in the previous frame or not.

[0093] The group progression circuit 4410 of the stationary group G 0 allows a movement pattern M 0 because the decision (decision step dS4418) whether any T i (n), which includes the selected principal transformation P(n), with the stationary group G 0 united, rather from a comparison of the T i (n) with a threshold thd 0 , than with a variable gradient-based motion vector M 0 depends, since the group G 0is considered inpatient. The inpatient group G 0 is created as an empty gradient at the beginning of a video stabilization.

[0094] If during a frame n a T i (n) the condition |T i (n)| < thd 0 fulfilled, (YES branch of the decision at step dS4418), then follows: this T i (n) unites with G 0 ; History of EH 0 is updated to indicate the existence of a stationary transformation at frame n; and if P(n) = this T i (n), the selection history SH 0 updated to select this T i (n) to display.

[0095] Otherwise (NO branch of the decision at step dS4418), such T i (n), the |T i (n)| < thd 0 not fulfill, while a frame with a group progression in each of the existing movement groups G 1to G N compared.

[0096] The group progression circuit 4430 of the movement group G K includes a course K -Memory for saving the selection history SH K , a course of existence EH K and a movement pattern M K . The course of existence EH K is a value with one bit per last frame that indicates whether a cross-frame transformation T i (n) in the previous frame with the movement group G K united or not. The course SH K is a value with one bit per last frame that indicates whether the cross-frame transformation T i (n) who are involved with the movement group G K was selected as the main transformation P(n) in the previous frame or not.

[0097] The movement pattern M K stores information that defines the vector M Kfrom the overall movement of Group G K Show. Every T i (n) also represents a motion vector M. Each motion group G K forms a motion vector M K . off. |T i (n)| be the magnitude of the motion vector of T i (n) and |T i (n) - M K | the deviation from T i (n) of the motion vector M K the existing movement group G K for 1 ≤ K ≤ N, where N is the number of currently existing motion groups. A motion group G J among the N existing motion groups that have a minimum of |T i (n) - M J | have, shows a best fitting group G J for T i (n). This union decision can be made by comparing |T i (n) - M J | with a predetermined threshold thd 1 be determined. If |T i (n) - M J | ≤ |T i (n) - MK | for a certain J and all K between 1 and n, and |T i (n) - M J | < thd 1 , (YES branch of the decision at step ds4438) thus combines, for example, in the decision at step ds4438 this T i (n) with the existing movement group G J .

[0098] If |T i (n) - M J | ≤ |T i (n) - M K | for all K and |T i (n) - M J | < thd 1 (YES branch of the decision at dS4438), then follows: T i (n) unites with G J ; Movement pattern M J will be adjusted to the newly united T i (n) to reproduce; the course of existence EH J is updated to reflect the existence of the movement group G J to display at frame n; if P(n) = this T i (n), the selection history SH Jupdated to select this T i (n) = P(n).

[0099] On the other hand, if after the decision step dS4438 for a T i (n) and for all existing movement groups (G 1 to G N ) has been repeated and none of the M K |T i (n) - M K | < thd 1 fulfilled (NO branch of the decision at step dS4438), this T i (n) with the newly formed movement group G N+1 (Step S4449). If this T i (n) with the newly formed movement group G N+1 combined (step S4449), then follows: T i (n) unites with the newly formed movement group G N+1 ; Movement pattern M N+1 is applied to the motion vector of this T i (n) discontinued; the course of existence EH N+1is preset to indicate the existence of the new movement group G N+1 to display at frame n; and if P(n) = this T i (n), the selection history SH N+1 updated to select this T i (n,) = P(n).

[0100] Each movement group (from G 0 to G J ) without a T i (n) that join for an extended period of time (frame) is deleted.

[0101] Fig. 8 is a view of a captured video frame recorded at time n and a compensation window corresponding to a compensation transformation C(n) calculated from the main transformation P(n), and illustrates a vertical over-excursion v1 to be reduced. The vertical over-excursion of the compensation window is measured as v1 in a step of a digital image stabilization (DIS) method according to an exemplary embodiment of the inventive concept.

[0102] As in Fig. As shown in Figure 8, the compensation window corresponding to a compensation transformation C(n) of the captured video frame may have a vertical over-excursion (v0 or v1), a horizontal over-excursion (u0 or u1), or both a vertical and horizontal over-excursion (v0 or v1) and (u0 or u1). Each of the potential over-excursions (v0, v1, u0, and u1) may be caused by a translational component of the compensation transformation C(n), by a rotational component of the compensation transformation C(n), or by a combination of translational and rotational components of the compensation transformation C(n).

[0103] It is desirable to minimize the over-deflections (of v0, v1, u0 and u1) by adaptively filtering the main transform P(n) to output a filtered compensation transform C(n) for each recorded video frame based on the history of the displacements.

[0104] Fig. Figure 9 is a block diagram of the digital image stabilization (DIS) circuit that performs a DIS method in accordance with another exemplary embodiment of the inventive concept. The DIS circuit includes a detection unit (DU) 2000 that analyzes a received jerky video and performs cross-frame transformations T i (n), a motion spell unit (TU) 4000 having main transformation selection circuits (4100, 4200, 4400) which output a main transformation P(n) from T i (n), and an adaptive compensation filter 8000 that filters P(n) to a compensation transform C(n), and a compensation unit (CU) 6000 that outputs a stabilized video by modifying the jerky video frames using C(n).

[0105] The main transform selection circuits (4100, 4200, 4400) select by identifying the cross-frame transformation T i (n) the comprehensive movement caused by a shaky camera, one of the cross-frame transformations T i (n) as the main transformation P(n), while the cross-frame transformations T i (n) of the moving objects in the scene are ignored, and they output their selection as the computed main transformation P(n). Thus, the main transformation selection circuits (4100, 4200, 4400) of the DIS circuit select one of the cross-frame transformations T i(n) as the calculated main transformation P(n) and output it. The compensation transformation C(n) is obtained by adaptive filtering of the main transformation P(n). The compensation transformation C(n) is a description of the geometry of a stabilized video image (compensation window) with respect to the corresponding input video image. The description may include a position, an angle, a scale, etc. Some commonly used compensation transformations are the similarity transformation and the affine transformation, and although the inventive concept is not limited to these transformations, the affine transformation is used below to illustrate the exemplary method according to the present inventive concept.

[0106] The main transformation selection circuits (4100, 4200, 4400) successively output the selected main transformations P(n-∞), ... , P(n-1), P(n) of a continuous sequence of frames to the adaptive compensation filter 8000, where the main transformation P(n-∞) indicates the use of the recursive (infinitive impulse feedback IIR) filter. The adaptive compensation filter 8000 estimates the intended camera trajectory from the jittery motion, represented by the sequence of main transformations P(n-∞), ... , P(n-1), P(n), and outputs the compensation transformation C(n) consistent with the estimated camera trajectory.

[0107] The visual effect of the stabilized video depends heavily on the quality of the adaptive compensation filter 8000. Conventional trajectory estimation methods include motion vector integration, a Kalman filter, etc. However, these and other conventional trajectory estimation methods do not perform well over a wide range of jerky video characteristics. In exemplary embodiments of the inventive concept, an adaptive compensation filter is used to filter out the jerky movements and generate a stabilized video.

[0108] Fig. 10 is a block diagram of an adaptive compensation filter 8000 in the trajectory unit (TU) 4000 of the DIS circuit of Fig. 9, which is configured to adaptively filter the main transform P(n) based on the history of the compensation window excursions. The adaptive compensation filter 8000 filters the main transform P(n) and outputs the adaptively filtered compensation transform C(n) based on the received main transforms P(n-∞), ... , P(n-1), P(n) of a continuous sequence of frames.

[0109] The adaptive compensation filter 8000 includes a strong compensation filter (SC) 8700, a weak compensation filter (WC) 8600, an adaptive filter control circuit 8500 for outputting a control signal E(n), and an excursion-modulated mixer 8200. The SC filter is a highly frequency-selective, high-order linear time-invariant digital filter effective for filtering very jerky input video. On the other hand, the weak compensation (WC) filter has a lower frequency-selective characteristic, producing fewer over-excursions of the compensation window at the expense of a less smooth output video.

[0110] The adaptive compensation filter 8000 is an effective combination of the SC filter and the WC filter. The excursion-modulated mixer 8200 performs the mixing of the SC filter and WC filter outputs based on the control signal E(n) generated and output by the adaptive filter controller 8500 based on the excursion profile of the compensation window.

[0111] Fig. 11 is a block diagram of a first exemplary implementation 8000-1 of the adaptive compensation filter 8000 of the trajectory unit (TU) 4000 of the DIS circuit of Fig. 9. The exemplary adaptive compensation filter 8000-1 has a strong compensation filter 8700 and a weak compensation filter 8600 and a feedback loop into the excursion calculator 8510 of the adaptive filter controller 8500-1.

[0112] In relation to Fig. 11, the strong compensation filter (SC) 8700 is a linear time-invariant recursive high-order digital filter that has a cutoff frequency of about 1.0 Hz and a sharp rolloff to obtain a visually well-stabilized video and has a highly frequency-selective output F(n).

[0113] The Weak Compensation Filter (WC) 8600 is a linear, time-invariant, high-order or low-order recursive digital filter. The WC 8600 features a low frequency selective output G(n) with a cutoff frequency slightly higher than 1 Hz (e.g., at 1.2 Hz) and a smooth rolloff to reduce over-excursions.

[0114] The displacement-modulated mixer 8200-1 of the adaptive compensation filter 8000-1 performs displacement-modulated adaptive filtering that combines F(n) and G(n) in accordance with the scalar control signal E(n). The output of the SC filter F(n) and the output of the WC filter G(n) are both intermediate compensation transforms, and the output of the displacement-modulated mixer 8200-1 C(n) is also a compensation transform. The displacement modulated mixer 8200-1 outputs C(n) = (1 - E(n)) * F(n) + E(n) * G(n), where E(n) is a non-linear normalized scalar control signal in the range [0,1], “*” is a multiplication operation between a scalar and a transform that maps to a transform, and “+” is an addition operation between two transforms that maps to a transform.Thus, in this exemplary embodiment, the adaptive compensation filter 8000-1 is a linear combination of the SC filter and the WC filter. Thus, the adaptive compensation filter 8000-1 is effectively a linear, time-varying, recursive higher-order digital filter that possesses a well-known stabilization characteristic through the basic principle of linear superposition.

[0115] The linearly combined compensation transformation C(n) is controlled by a scalar control signal E(n) based on the displacement profile of the compensation window. Small displacements in the profile produce a small E(n), thus increasing the impact of the SC filter on the current frame m, whereas large displacements in the profile produce an E(n) close to 1, thus increasing the impact of the WC filter on the current frame n. Medium displacements in the stored profile result in proportional impacts of the SC filter and the WC filter.

[0116] Thus, the SC filter provides the primary contribution at small displacements and is very effective at filtering out high-frequency jitter. And because the WC filter contributes more strongly at large displacements, the occurrence of over-deflection is significantly reduced. The 8000-1 adaptive compensation filter prevents excessive over-deflection in high-motion input video while maintaining excellent video stabilization characteristics.

[0117] With reference to Fig. 11, the adaptive filter controller 8500-1 includes a displacement calculator 8510, four displacement history integrators 8520, and a modulation factor calculator 8530-1. The adaptive filter controller 8500-1 is part of a feedback loop. The displacement calculator 8510 outputs E(n), which is derived from previous adaptive compensation filter outputs C(n-∞), ... , C(n-2), C(n-1), where n denotes its time sequence type, such that E(n) and C(n) do not form an unrealizable delay-free loop. Thus, the exemplary embodiment is suitable for real-time video stabilization and includes a causal linear time-varying filter with predictable characteristics.

[0118] The displacement calculator 8510 receives a feedback loop of a compensation transformation C(n) output by the displacement-modeled mixer 8200-1. The displacement calculator 8510 includes a u0 calculator, a u1 calculator, a v0 calculator, and a v1 calculator to individually calculate the left, right, bottom, and top side displacements per frame based on the position of the four corners of the compensation window (see Fig. 8).

[0119] The adaptive compensation filter 8500-1 maintains a displacement profile through the use of the recursive filter. The output from each lateral displacement calculator of the displacement calculator 8510 is then individually integrated over time by the displacement profile integrator, which is necessary for a recursive low-pass filter. The output from each recursive-pass filter (Hu0, Hu1, Hv0, Hv1) ​​is then input to the modulation factor calculator 8530-1. The modulation factor calculator 8530-1 selects the maximum among the four time-integrated displacement quantities (Hu0, Hu1, Hv0, Hv1) ​​and generates a nonlinear normalized scalar control signal E(n) with a continuous range [0, 1].

[0120] The 8530-1 Modulation Factor Calculator outputs the nonlinear normalized scalar control signal E(n) to model the mixture of F(n) and G(n). A small value of E(n) indicates a small displacement profile, and a large value of E(n) indicates a large displacement profile.

[0121] Thus, the mixing of F(n) and G(n) under the control of the scalar control signal E(n) to generate and output a compensation transform C(n) is based on the displacement profile of the compensation window. This exemplary embodiment provides good stabilization without frequent over-deflections and has a known frequency response and predictable stabilization characteristics suitable for real-time video stabilization.

[0122] The above-disclosed subject matter is intended to be illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, improvements, and other embodiments as fall within the true spirit and scope of the inventive concept. Thus, to the fullest extent permitted by law, the scope of the inventive concept is to be determined by the broadest allowable interpretation of the following claims, and their equivalents, and is not restricted or limited to the foregoing detailed description.

Claims

[1] A method for processing video data, comprising: Identifying Transformations T i (n) of motion vectors in a video frame recorded by a camera, wherein a compensation transformation C(n) represents a camera motion to be compensated; and Adaptive filtering of the compensation transformation C(n) based on non-linear normalized, time-integrated deflections to prevent excessive over-deflection of a compensation window. [2] The method of claim 1, wherein the adaptive filtering comprises a scalar combination of: a highly frequency-selective linear time-invariant recursive higher-order filter (8700) that performs strong compensation (SC) filtering from the compensation transform C(n); and a lower frequency selective linear time-invariant recursive filter of higher order or lower order (8600) that performs weak compensation (WC) filtering from the compensation transform C(n). [3] The method of claim 2, wherein the adaptive filtering comprises: Achieving a large impact of the SC filter (8700) on the current frame based on a history of small deflections; and Achieving a large impact of the WC filter (8600) on the current frame during a course of large deflections. [4] The method of claim 3, further comprising achieving proportional effects of the SC filter (8700) and the WC filter (8600) at a displacement profile of mean displacements. [5] The method of claim 1, wherein the non-linear normalized time-integrated displacements comprise: Selecting a maximum value M(n) from at least four time-integrated deflection values ​​of the deflection curve. [6] The method of claim 5, wherein E(n) is a non-linear normalized value of M(n) within a range of [0,1] from a range from small deflections to large deflections. [7] The method of claim 2, wherein the SC filtering (8700) is set to a cutoff frequency lower than a cutoff frequency set for the WC filtering (8600). [8] The method of claim 7, wherein the SC filtering (8700) comprises using a rolloff that is sharper than the rolloff in the WC filtering (8600). [9] The method of claim 7, wherein the SC filtering (8700) has a cutoff frequency of approximately 1Hz. [10] The method of claim 3, wherein achieving an action of the SC filter (8700) and the WC filter (8600) is based on a non-linear normalized value E(n) within a continuous range of 0 to 1. [11] Image processing circuit, comprising: a receiver configured to receive frames of image data; a memory (350) configured to store a first group of motion vectors having a first motion characteristic and a second group of motion vectors having a second motion characteristic; a transformation selector (4000) configured to select from transformations T i (n) to identify from the first and second groups of motion vectors a compensation transformation C(n) representing a camera movement to be compensated; and an adaptive filter (8000) configured to prevent excessive over-excursion of a compensation window based on non-linear normalized time-integrated excursions. [12] The circuit of claim 11, further comprising a mixer (8200, 8200-1) configured to combine a highly frequency-selective higher-order linear time-invariant filter (8700) performing strong compensation (SC) filtering of the compensation transform C(n) and a lower frequency-selective lower-order linear time-invariant filter (8600) performing weak compensation (WC) filtering of the compensation transform C(n). [13] The circuit of claim 12, wherein the adaptive filter (8000) is further configured to achieve a large influence of the SC filter (8700) on the current frame based on a deflection history of small deflections, and to achieve a large influence of the WC filter (8600) on the current frame based on a deflection history of large deflections. [14] The circuit of claim 13, further comprising achieving proportional effects of the SC filter (8700) and the WC filter (8600) at a displacement profile of mean displacements. [15] The circuit of claim 12, further comprising a modulation factor calculator (8530-1) for calculating a maximum magnitude from four time-integrated displacement magnitudes of the displacement curve to output a non-linear normalized signal E(n). [16] The circuit of claim 15, wherein the non-linear normalized signal E(n) is input to the mixer (8200-1) to model the mixing of the SC filter (8700) and the WC filter (8600) with a scalar value within a range of 0 to 1. [17] The circuit of claim 13, wherein the SC filter (8700) is set to a cutoff frequency lower than a cutoff frequency set for the WC filter (8600). [18] The circuit of claim 17, wherein the SC filter (8700) comprises using a rolloff that is sharper than the rolloff in the WC filter (8600). [19] The circuit of claim 11, wherein the adaptive filter (8000) is configured to maintain a deflection profile by using recursive filters. [20] Circuit according to claim 11, wherein transformations T FP,i(n) of the first group of motion vectors represent a movement of selected feature points, and transformations T tile,i (n) the second group of motion vectors represent a movement of non-overlapping tiles into which a video frame is divided. [21] A camera comprising: an image sensor configured to capture an image; an image data circuit configured to convert the captured image into a frame of image data; an image processing circuit comprising: a receiver configured to receive frames of image data; a motion vector detector (2000) configured to detect movements of objects and generate motion vectors; a transformation selector (4000) configured to select from transformations T i(n) of the motion vectors to identify a compensation transformation C(n) representing a camera movement to be compensated; an adaptive filter (8000) configured to filter over-deflections of a compensation window; and a compensation unit (6000) configured to adjust the captured image based on the compensation transformation C(n) and the output of the adaptive filter (8000). [22] The camera of claim 21, further comprising a grouping circuit (1300) configured to group the motion vectors (MV) into one of at least two groups comprising a feature point group and a tile group, the tile group comprising non-overlapping tiles into which a video frame is divided. [23] The camera of claim 21, wherein the transformation selector (4000, 4100-1, 4100-2) is configured to identify the compensation transformation C(n) based on a plurality of scoring functions consisting of gradient, motion, feature, and extent scores (H i (n), M i (n), F i (n), Ei(n)) can be selected. [24] The camera of claim 21, wherein the adaptive filter (8000) is configured to prevent excessive over-excursion of a compensation window based on non-linear normalized time-integrated excursions. [25] A camera according to claim 21, wherein the adaptive filter (8000) is configured to maintain a displacement history by using recursive filters.

Citation Information

Patent Citations

  • Blurring correction method and imaging device

    US20060153472A1

  • Image processing device, camera motion component calculation method, image processing program, and recording medium

    US20100208087A1

  • WO8729357A1

  • WO8729358A1