A Fast and Robust Electronic Image Stabilization Method

Abstract

The present invention relates to a fast and robust electronic image stabilization method, and the method is as follows: For any frame of image, a region with a size of m×n is extracted with the origin of the image coordinate as the center point as the image stabilization template; m is the number of pixel columns of the image stabilization template, and n is the number of pixel rows of the image stabilization template; m = 30%M to 40%M, n = 30%N to 40%N; M is the number of pixel columns of the image, and N is the number of pixel rows of the image; the row and column rough offsets are obtained based on the offset calculation method of gray projection; the final row and column offsets are obtained based on the diamond search method; the center of the image stabilization template is offset according to the final row and column offsets as the center coordinates of the image stabilization template in the next frame of image, and at the same time, the rows and columns of this frame of image are stabilized respectively according to the final row and column offsets. The present invention reduces the amount of calculation and improves the operation efficiency on the premise of ensuring high accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of video image processing, and particularly relates to a fast and robust electronic image stabilization method that takes into account both accuracy and real-time performance. Background Art

[0002] Electronic image stabilization is to solve the problem of instability between frames of a sequence of images through image processing related technologies. That is, by detecting the motion vectors between image frames, motion compensation is performed on the subsequent image sequence to output a stable video sequence.

[0003] Relatively mature methods in this field include gray projection, block matching, feature matching, etc. Among them, the gray projection method has a smaller amount of calculation and is easier to implement compared to other methods. However, it cannot eliminate the interference caused by local motion and has a lower image stabilization accuracy. Traditional block matching and feature matching have higher image stabilization accuracy, but have a larger amount of calculation, and it is difficult to ensure real-time performance when the hardware configuration is low. Summary of the Invention

[0004] The present invention provides a fast and robust electronic image stabilization method that combines gray projection and diamond search. This method can solve the problem of poor anti-interference ability of the gray projection method, and has a small amount of calculation and high efficiency.

[0005] To solve the above technical problems, the fast and robust electronic image stabilization method of the present invention is as follows:

[0006] For any frame of image, an m×n-sized area is extracted with the origin of the image coordinates as the center point as the image stabilization template; m is the number of pixel columns of the image stabilization template, and n is the number of pixel rows of the image stabilization template; m = 30%M to 40%M, n = 30%N to 40%N; M is the number of pixel columns of the image, and N is the number of pixel rows of the image; the rough row and column offsets are obtained based on the offset calculation method of gray projection; the final row and column offsets are obtained based on the diamond search method; the center of the image stabilization template is offset according to the final row and column offsets as the center coordinates of the image stabilization template in the next frame of image, and at the same time, image stabilization is performed on the rows and columns of this frame of image according to the final row and column offsets.

[0007] Further, the calculation method of the rough row and column offsets is as follows:

[0008] Calculate the gray projection values of each row and column of the image stabilization template;

[0009] An (m + 2p)×(n + 2q)-sized image area is extracted with the origin of the coordinates as the center point as the specified area, and the rough offset is obtained through the following relevant functions:

[0010]

[0011] Where d x 、d yrespectively represent the rough offsets of rows and columns; Y k (y) respectively represent the gray projection values of the x-th column and the y-th row of the k-th frame image stabilization template; X k (x + p), Y k (y + q) respectively represent the gray projection values of the (x + p)-th row and the (y + q)-th column of the specified area of the k-th frame image.

[0012] Furthermore, calculate the gray projection values of each row and column of the image stabilization template according to the following formula:

[0013]

[0014] where f k (x, y) is the gray value of the pixel at the x-th column and the y-th row of the image stabilization template.

[0015] Furthermore, the gray projection values of each row and column of the image stabilization template can also be calculated according to the following formula:

[0016]

[0017] where X kAvg , Y kAvg respectively represent the average gray projection value of columns and the average gray projection value of rows of the k-th frame image stabilization template; f k (x, y) is the gray value of the pixel at the x-th column and the y-th row of the image stabilization template.

[0018] Furthermore, the method for obtaining the final row and column offsets based on diamond search is as follows: First, perform a rough search through the large diamond template, and then perform a fine search through the small diamond template to obtain the final row and column offsets.

[0019] The method for performing a rough search through the large diamond template is as follows:

[0020] Take the area after the overall horizontal offset of the image stabilization template by d x and the overall vertical offset by d y as the initial large diamond search area; Take the center of the large diamond search area as the center of the large diamond template, and search the 9 search points within the large diamond template through the following relevant function to obtain the large diamond search result i.e., MBD C ;

[0021]

[0022] where s C , t Care respectively the row number and column number where the minimum error point of the large diamond search is located within the large diamond search area, s is the column number where the pixels in the large diamond template are located, t is the row number where the pixels in the large diamond template are located; f c (x, y) is the gray value of the pixel at the x-th column and y-th row within the large diamond search area; f c (x + s, y + t) is the gray value of the pixel at the (x + s)-th column and (y + t)-th row with the center of the large diamond search area as the origin.

[0023] The method for fine search through the small diamond template is as follows:

[0024] Take the center of the image after translating the large diamond template search as the center of the small diamond template;

[0025] If MBD C is located at the center of the large diamond template, search the 5 search points within the positive small diamond template through the following relevant function to obtain the final row and column offsets:

[0026]

[0027] Among them, represents the small diamond search result, u D is the final column offset, v D is the final row offset, u is the column number where the pixels in the positive small diamond template are located, v is the row number where the pixels in the positive small diamond template are located; f c (x, y) is the gray value of the pixel at the x-th column and y-th row within the large diamond search area; f c (x + u, y + v) is the gray value of the pixel at the (x + u)-th column and (y + v)-th row with the center of the large diamond search area as the origin;

[0028] If MBD C is located above the center of the large diamond template, move the search point at the upper corner of the positive small diamond template up one row and jointly form an upper - small diamond template with other search points, and search through the upper - small diamond template to obtain the final row and column offsets; the relevant function is as follows:

[0029]

[0030] Among them, u is the column number where the pixels in the upper - small diamond template are located, v is the row number where the pixels in the upper - small diamond template are located;

[0031] If MBD C is located below the center of the large diamond template, move the search point at the lower corner of the positive small diamond template down one row and jointly form a lower - small diamond template with other search points, and search through the lower - small diamond template to obtain the final row and column offsets; the relevant function is as follows:

[0032]

[0033] Among them, u is the column number where the pixel is located in the lower-small diamond template, and v is the row number where the pixel is located in the lower-small diamond template;

[0034] If the MBD C is located on the left side of the center of the large diamond template, then move the search point at the left corner of the positive small diamond template one column to the left and jointly form a left-small diamond template with other search points, and perform a search through the left-small diamond template to obtain the final row and column offset; the relevant function is as follows:

[0035]

[0036] Among them, u is the column number where the pixel is located in the left-small diamond template, and v is the row number where the pixel is located in the left-small diamond template;

[0037] If the MBD C is located on the right side of the center of the large diamond template, then move the search point at the right corner of the positive small diamond template one column to the right and jointly form a right-small diamond template with other search points, and perform a search through the right-small diamond template to obtain the final row and column offset; the relevant function is as follows:

[0038]

[0039] Among them, u is the column number where the pixel is located in the right-small diamond template, and v is the row number where the pixel is located in the right-small diamond template.

[0040] The present invention has the following beneficial effects:

[0041] By using the gray projection method, the calculation amount is shortened, a rough search range is provided for the subsequent diamond search method, and the operation efficiency is effectively improved; by the improved diamond search method, on the premise of ensuring the operation efficiency, the image stabilization accuracy is further improved, and the interference caused by local movement is suppressed. The present invention reduces the calculation amount and improves the operation efficiency while ensuring a relatively high accuracy. The operation efficiency is increased by about 60%, and the image stabilization accuracy is 1 pixel. Brief Description of the Drawings

[0042] Figure 1 is a flowchart of the electronic image stabilization method of the present invention.

[0043] Figure 2 is a large diamond search template diagram.

[0044] Figure 3 is a small diamond search template diagram.

[0045] Figure 4 is an improved diamond search - upper-diamond search template diagram.

[0046] Figure 5Improved diamond search - down - diamond search template diagram.

[0047] Figure 6 Improved diamond search - left - diamond search template diagram.

[0048] Figure 7 Improved diamond search - right - diamond search template diagram. Detailed implementation manners

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. In addition, it should be noted that for the convenience of description, only the parts related to the present invention rather than all the structures are shown in the drawings.

[0050] In the description of the present invention, unless otherwise clearly defined and limited, the terms "connected", "connected to", and "fixed" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the communication inside two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0051] In the present invention, unless otherwise clearly defined and limited, the first feature being "above" or "below" the second feature may include the direct contact between the first and second features, or may include the situation where the first and second features are not in direct contact but in contact through other features therebetween. Moreover, the first feature being "above", "above and over", and "on top of" the second feature includes that the first feature is directly above and obliquely above the second feature, or merely means that the horizontal height of the first feature is higher than that of the second feature. The first feature being "below", "below and under", and "beneath" the second feature includes that the first feature is directly below and obliquely below the second feature, or merely means that the horizontal height of the first feature is lower than that of the second feature.

[0052] In the description of this embodiment, the orientation or positional relationships such as "above", "below", "left", and "right" are based on the orientation or positional relationships shown in the drawings. They are only for the convenience of description and simplifying the operation, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as a limitation to the present invention. In addition, the terms "first" and "second" are only used for distinction in description and have no special meanings.

[0053] Such as Figure 1As shown in the figure, the fast and robust electronic image stabilization method of the present invention mainly includes the following four parts: image preprocessing, offset calculation based on gray projection, offset calculation based on diamond search, and motion compensation.

[0054] I. Image preprocessing

[0055] In order to meet the real-time requirement without affecting the image stabilization accuracy, an image region of size m×n is extracted with the coordinate origin as the center point as the image stabilization template; m is the number of pixel columns of the image stabilization template, and n is the number of pixel rows of the image stabilization template; m = 400, n = 300; the gray values of each pixel of the image stabilization template are obtained and Gaussian filtering is performed to remove noise for use in subsequent operations.

[0056] II. Offset calculation based on gray projection

[0057] a. Perform gray projection calculations on the rows and columns of the image stabilization template respectively;

[0058] According to the conventional gray projection algorithm, the two-dimensional gray information of the rows and columns of the image stabilization template is mapped into independent one-dimensional projection sequences. The formula is as follows:

[0059]

[0060] Among them, X k (x), Y k (y) respectively represent the gray projection values of the x-th column and the y-th row of the k-th frame image.

[0061] In order to reduce the influence of factors such as background brightness on the result of image stabilization based on the gray projection method, it is preferably to perform gray projection calculations on the rows and columns of the image stabilization template respectively through the following formula;

[0062]

[0063] Among them, X kAvg , Y kAvg respectively represent the average column gray projection value and the average row gray projection value of the image stabilization template of the k-th frame image; X k (x), Y k (y) respectively represent the gray projection value of the x-th column and the gray projection value of the y-th row of the image stabilization template of the k-th frame image; f k (x, y) is the gray value of the pixel at the x-th column and the y-th row of the image stabilization template.

[0064] b. In order to ensure the operation efficiency, the present invention extracts an image region of size (m + 2p)×(n + 2q) with the coordinate origin as the center point as the specified region, 2p = 40 - 60; 2q = 30 - 40, in this embodiment, p = 20, q = 20 is selected; the following correlation function is used to calculate the offset to obtain the rough offset:

[0065]

[0066] Among them, d x and d y respectively represent the rough offsets of rows and columns; X k (x + p) and Y k (y + q) respectively represent the gray - level projection values of the (x + p)-th column and the (y + q)-th row in the specified area of the k-th frame image; 2P = 40, 2Q = 40.

[0067] III. Offset Calculation Based on Diamond Search

[0068] The diamond search algorithm has two search templates, namely the large diamond and the small diamond, as shown in Figure 2 and 3 . First, a rough search is performed through the large diamond template with a larger step size, and then a fine search is performed through the small diamond template. This method has a fast calculation speed and high accuracy.

[0069] Since the motion vector has the following characteristics: central offset, correlation, and directionality. Therefore, the present invention adopts an improved diamond search method to perform a fine search on the calculation result of the gray - level projection offset, minimizing the diamond search calculation amount and improving the image stabilization accuracy. The improved large diamond template and small diamond template are shown in Figures 4 to 7 .

[0070] The diamond search process is as follows:

[0071] a. Search through the large diamond template

[0072] Take the center of the image after the overall horizontal offset of the image stabilization template by d x and the overall vertical offset by d y as the center of the initial large diamond template, and search for 9 search points within the large diamond template, as shown in Figure 2 . Among them, the correlation function is as follows:

[0073]

[0074] Among them, represents the large diamond search result, that is, MBD C (the minimum error point of the large diamond search), s C , t C are respectively the row number and column number of the minimum error point of the large diamond search within the large diamond template, s is the column number of the pixel within the large diamond template, t is the row number of the pixel within the large diamond template; f c (x, y) is the gray - level value of the pixel at the x - th column and y - th row within the large diamond search area; f c (x + s, y + t) is the gray - level value of the pixel at the (x + s)-th column and (y + t)-th row with the center of the large diamond search area as the origin.

[0075] b. Search through an improved small diamond template

[0076] The MBD obtained according to step a C Position, and process them separately.

[0077] Since the large diamond template moves continuously during the search process, Converge to the center of the large diamond template, and use the center of the image after the large diamond template searches and translates (the center point of the entire image) as the center of the small diamond template;

[0078] If the MBD C Position is at the center of the large diamond template, then search through the positive small diamond template as shown in Figure 3 ;

[0079] The search correlation functions for the 5 search points in the positive small diamond template are as follows:

[0080]

[0081] Among them, Represents the small diamond search result, that is, MBD D (The minimum error point of the small diamond search), u D Is the final column offset, v D Is the final row offset; u is the column number of the pixel in the positive small diamond template, v is the row number of the pixel in the positive small diamond template; f c (x, y) is the gray value of the pixel at the x-th column and y-th row in the large diamond search area; L c (x + u, y + v) is the gray value of the pixel at the (x + u)-th column and (y + v)-th row with the center of the large diamond search area as the origin.

[0082] If the MBD C Position is above the center of the large diamond template, then move the search point at the upper corner of the positive small diamond template up one row and jointly form an upper-small diamond template with other search points, and search through the upper-small diamond template, as shown in Figure 4 ;

[0083] The search correlation functions for the 5 search points in the upper-small diamond template are as follows:

[0084]

[0085] Among them, Represents the small diamond search result, that is, MBD D (The minimum error point of the small diamond search), u is the column number of the pixel in the upper-small diamond template, v is the row number of the pixel in the upper-small diamond template; f c (x, y) is the gray value of the pixel at the x-th column and y-th row in the large diamond search area; fc (x + u, y + v) is the grayscale value of the pixel at the (x + u)-th column and (y + v)-th row with the center of the large diamond search area as the origin.

[0086] If MBD C is located below the center of the large diamond template, then the search point at the lower corner of the positive small diamond template is moved down one row and together with other search points forms the lower - small diamond template, and search is performed through the lower - small diamond template, as Figure 5 shown;

[0087] The search - related function for the 5 search points within the lower - small diamond template is as follows:

[0088]

[0089] Among them, represents the small diamond search result, that is, MBD D (the minimum error point of small diamond search), u is the column number of the pixel within the lower - small diamond template, v is the row number of the pixel within the lower - small diamond template; f c (x, y) is the grayscale value of the pixel at the x - th column and y - th row within the large diamond search area; f c (x + u, y + v) is the grayscale value of the pixel at the (x + u)-th column and (y + v)-th row with the center of the large diamond search area as the origin.

[0090] If MBD C is located to the left of the center of the large diamond template, then the search point at the left corner of the positive small diamond template is moved left one column and together with other search points forms the left - small diamond template, and search is performed through the left - small diamond template, as Figure 6 shown;

[0091] The search - related function for the 5 search points within the left - small diamond template is as follows:

[0092]

[0093] Among them, represents the small diamond search result, that is, MBD D (the minimum error point of small diamond search), u is the column number of the pixel within the left - small diamond template, v is the row number of the pixel within the left - small diamond template; f c (x, y) is the grayscale value of the pixel at the x - th column and y - th row within the large diamond search area; f c (x + u, y + v) is the grayscale value of the pixel at the (x + u)-th column and (y + v)-th row with the center of the large diamond search area as the origin.

[0094] If MBD CIf the position is to the right of the center of the large diamond template, then move the search point at the right corner of the positive small diamond template one column to the right and jointly form a right - small diamond template with other search points, and perform a search through the right - small diamond template, as Figure 7 shown.

[0095] The relevant functions for searching the 5 search points within the right - small diamond template are as follows:

[0096]

[0097] Among them, represents the small diamond search result, that is, MBD D (the minimum error point of the small diamond search), u is the column number where the pixel is located within the right - small diamond template, v is the row number where the pixel is located within the right - small diamond template; f c (x, y) is the gray value of the pixel at the x - th column and y - th row within the large diamond search area; f c (x + u, y + v) is the gray value of the pixel at the (x + u)-th column and (y + v)-th row with the center of the large diamond search area as the origin.

[0098] Fourth, take the small diamond search result as the optimal solution for the image stabilization operation, that is, the final row - column offset. Offset the center of the image stabilization template of the k - th frame according to the final row - column offset as the center coordinates of the image stabilization template in the next frame.

[0099] Fifth, image motion compensation

[0100] Move the rows and columns of the current frame image respectively according to the small diamond search result. For the pixels in the unknown area that appear after moving the image, in order to improve the operation efficiency, the present invention adopts the method of filling the black edges by cropping according to a percentage.

[0101] The above - mentioned embodiments are the preferred embodiments of the present invention. Compared with the prior art, the operation efficiency is increased by about 60%, and the image stabilization accuracy is 1 pixel. When step three uses the conventional large diamond and small diamond search templates to calculate the gray - scale projection offset, compared with the prior art, the operation efficiency can only be increased by about 25% on the premise of achieving the same image stabilization accuracy.

[0102] The present invention is not limited to the above - mentioned embodiments. When m is in the range of 30%M to 40%M, n is in the range of 30%N to 40%N, 2P is in the range, and 2Q is in the range, it can achieve the effects of improving the image stabilization accuracy, reducing the calculation amount, and improving the operation efficiency.

Claims

1. A fast and robust electronic image stabilization method, characterized in that The method is as follows: For any frame of image, an m×n sized area is extracted with the origin of the image coordinates as the center point as the image stabilization template; m is the number of pixel columns of the image stabilization template, and n is the number of pixel rows of the image stabilization template; m = 30%M to 40%M, n = 30%N to 40%N; M is the number of pixel columns of the image, and N is the number of pixel rows of the image; the rough row and column offsets are obtained based on the offset calculation method of gray projection; the final row and column offsets are obtained based on the diamond search method; The center of the image stabilization template is offset according to the final row and column offsets as the center coordinates of the image stabilization template in the next frame of image. At the same time, the rows and columns of this frame of image are stabilized respectively according to the final row and column offsets; the method for obtaining the final row and column offsets based on the diamond search is as follows: First, a rough search is performed through a large diamond template, and then a fine search is performed through a small diamond template to obtain the final row and column offsets; The method for performing a fine search through the small diamond template is as follows: The center of the image after the translation of the large diamond template search is used as the center of the small diamond template; If MBD C is located at the center of the large diamond template, the following relevant function is used to search for 5 search points within the positive small diamond template to obtain the final row and column offsets: Among them, represents the small diamond search result, u D is the final column offset, v D is the final row offset, u is the number of columns where the pixels are located within the positive small diamond template, and v is the number of rows where the pixels are located within the positive small diamond template; f c (x, y) is the gray value of the pixel at the x-th column and the y-th row within the large diamond search area; f c (x + u, y + v) is the gray value of the pixel at the (x + u)-th column and the (y + v)-th row with the center of the large diamond search area as the origin; If MBD C is located above the center of the large diamond template, the search point at the upper corner of the positive small diamond template is moved up one row and together with other search points forms an upper - small diamond template. Search is performed through the upper - small diamond template to obtain the final row - column offset. The relevant function is as follows: Where u is the number of pixel columns where the pixels in the upper-small diamond template are located, and v is the number of pixel rows where the pixels in the upper-small diamond template are located; If MBD C is located below the center of the large diamond template, then move the search point at the lower corner of the positive small diamond template down one row and jointly form the next small diamond template with other search points, and perform a search through the next small diamond template to obtain the final row and column offset; the relevant function is as follows: Where u is the number of pixel columns where the pixels in the lower-small diamond template are located, and v is the number of pixel rows where the pixels in the lower-small diamond template are located; If MBD C is located to the left of the center of the large diamond template, then move the search point at the left corner of the positive small diamond template one column to the left and jointly form a left - small diamond template with other search points, and perform a search through the left - small diamond template to obtain the final row - column offset; the relevant function is as follows: Where u is the number of pixel columns where the pixels in the left-small diamond template are located, and v is the number of pixel rows where the pixels in the left-small diamond template are located; If MBD C is located to the right of the center of the large diamond template, then the search point at the right corner of the positive small diamond template is shifted one column to the right and together with other search points forms a right - small diamond template. Search is performed through the right - small diamond template to obtain the final row - column offset. The relevant function is as follows: Where u is the number of pixel columns where the pixels in the right-small diamond template are located, and v is the number of pixel rows where the pixels in the right-small diamond template are located.

2. The fast and robust electronic image stabilization method according to claim 1, characterized in that The calculation method of the rough row and column offsets is as follows: Calculate the gray projection values of each row and column of the image stabilization template; An (m + 2p)×(n + 2q) sized image area is extracted with the origin of coordinates as the center point as the specified area, and the rough offset is obtained through the following relevant functions for calculation: Among them, d x and d y respectively represent the rough offsets of rows and columns; X k (x), Y k (y) respectively represent the gray projection values of the x-th column and the y-th row of the image stabilization template of the k-th frame; X k (x + p), Y k (y + q) respectively represent the gray projection values of the (x + p)-th row and the (y + q)-th column of the specified area of the k-th frame image.

3. The fast and robust electronic image stabilization method according to claim 2, characterized in that Calculate the gray projection values of each row and column of the image stabilization template according to the following formula: where f k (x, y) is the gray value of the pixel at the x-th column and y-th row of the image stabilization template.

4. The fast and robust electronic image stabilization method according to claim 2, characterized in that Calculate the gray projection values of each row and column of the image stabilization template according to the following formula: Among them, X kAvg , Y kAvg respectively represent the column gray projection mean value and the row gray projection mean value of the k-th frame image stabilization template; f k (x, y) is the gray value of the pixel at the x-th column and the y-th row of the image stabilization template.

Images