Image stabilization method and system based on gray projection

Abstract

The invention relates to an image stabilization method and system based on gray projection. The method comprises the following steps: dividing adjacent images into a plurality of sub-image areas; determining a gray projection curve of each sub-image area in the adjacent images; taking the adjacent images as a reference frame image and a current frame image respectively, performing cross-correlation operation on the sub-image regional degree projection curve corresponding to each group, and determining the global motion offset of the current frame image according to an operation result; and performing motion compensation on the current frame image according to the global motion offset, and outputting the compensated current frame image. According to the invention, the current frame image can be compensated based on the global motion offset and the bilinear interpolation calculation to generate the compensation image with higher resolution, so that the resolution quality of the compensated current frame image is higher, the visual effect of the image is obviously improved, scene details are effectively enhanced, and the image quality is improved. Therefore, human eye observation and the target recognition capability and feature extraction capability of a subsequent algorithm are facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a method and system for image stabilization based on gray projection. BACKGROUND

[0002] In actual applications, when acquiring video image information, the imaging device can randomly shake, resulting in unstable video. The unstable image sequence can present a blurred and shaking real effect, which is not conducive to the human eye to observe and can easily cause visual fatigue, thereby affecting the accurate acquisition of effective video information and being not conducive to the post-processing of the image. Therefore, it is of great significance to reduce the image shaking caused by the random shaking of the imaging device itself in order to obtain a stable image sequence.

[0003] The current image stabilization technology includes mechanical image stabilization, optical image stabilization and electronic image stabilization. The mechanical image stabilization mainly obtains the motion translation of the imaging device through a sensor and uses a servo system to correct the posture of the imaging device, so as to stabilize the output image. The optical image stabilization mainly relies on moving or rotating a light wedge to change the angle and direction of the outgoing light, thereby achieving the purpose of image stabilization. However, the cost is the increase of the volume, weight, cost and complexity of the device, and there are inherent technical limitations in durability and response speed. The electronic image stabilization estimates the motion offset of the image directly from the video sequence through digital image processing technology, and then compensates the image.

[0004] However, in the above-mentioned image stabilization technology, the mechanical image stabilization is easily affected by the error of the sensor and requires high precision of the sensor. The optical image stabilization can only be applied to the application scene with small shaking. The traditional gray projection electronic image stabilization algorithm solves the bias displacement of the whole image and then compensates the image motion. However, the displacement size of each region on the image in many scenes is not consistent. SUMMARY

[0005] The present application aims to provide an image stabilization method and system based on gray projection, which can compensate the current frame image based on the global motion offset and bilinear interpolation calculation to generate a higher resolution compensation image, so that the resolution quality of the compensated current frame image is higher, the visual effect of the image is obviously improved, and the scene details are effectively enhanced, thereby facilitating the observation of the human eye and the recognition ability and feature extraction ability of the subsequent algorithm to the target.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0007] On the one hand, an image stabilization method based on gray projection is provided, which includes the following steps:

[0008] The kth frame and the k+1th frame of adjacent images are each divided into a plurality of sub-image regions, and each sub-image region of the kth frame has a sub-image region of the k+1th frame corresponding thereto and equal in size;

[0009] The gray projection curves of each sub-image region in the kth frame and the k+1th frame of adjacent images are determined;

[0010] The kth frame and the k+1th frame are respectively taken as a reference frame image and a current frame image, and the cross-correlation operation is performed on the corresponding sub-image region and the gray projection curve of each group, and the global motion offset of the current frame image relative to the reference frame image is determined according to the cross-correlation operation result;

[0011] The current frame image is motion compensated according to the global motion offset, and the compensated current frame image is output.

[0012] Preferably, the kth frame and the k+1th frame are filtered before the sub-image regions are divided.

[0013] Preferably, the following formula is used for cross-correlation calculation to obtain the row offset and the column offset of the sub-image region of the current frame image relative to the sub-image region of the reference frame image in each group of corresponding sub-image regions:

[0014] ; (1)

[0015] ; (2)

[0016] Wherein, W u (Δt i ), W u (Δt j ) are respectively the row offset and the column offset of the sub-image region of the current frame image relative to the sub-image region of the reference frame image in each group of corresponding sub-image regions; i, j represent respectively the row and the column where the pixel point is located; P u (i), P u (j) represent respectively the row gray projection value and the column gray projection value of the sub-image region of the reference frame image in each group of corresponding sub-image regions; P c (j+Δt i ), P c (i+Δt j ) represent respectively the row gray projection value and the column gray projection value of the sub-image region of the current frame image in each group of corresponding sub-image regions; M, N are respectively the width and the height of the current frame image and the width and the height of the reference frame image, and the width and the height of the current frame image and the reference frame image correspond to each other and are equal; Δt i , Δt jThe row offset and the column offset of the sub-image region of the current frame image relative to the sub-image region of the reference frame image at a pixel point (i, j) in each corresponding sub-image region of each group.

[0017] Preferably, the global motion offset of the current frame image relative to the reference frame image is determined according to the cross-correlation operation result, and specifically includes the following steps:

[0018] The maximum value in all the row offsets and the maximum value in all the column offsets are determined as the global motion offset of the current frame image relative to the reference frame image.

[0019] Or, the average value of all the row offsets and the average value of all the column offsets are taken, and the average value of the row offsets and the average value of the column offsets are taken as the global motion offset of the current frame image relative to the reference frame image.

[0020] Preferably, after the motion compensation of the current frame image according to the global motion offset, the current frame image obtained after the motion compensation is compensated again through bilinear difference calculation.

[0021] Preferably, the motion compensation of the current frame image according to the global motion offset includes moving the current frame image along the X direction / Y direction, and the moving amount is equal to the row offset of the global motion offset.

[0022] Preferably, the compensation of the current frame image obtained after the motion compensation again through bilinear difference calculation includes the following steps:

[0023] The compensation again is completed by adopting the mode of twice linear interpolation calculation in the x direction and once interpolation calculation in the y direction.

[0024] Preferably, the twice linear interpolation calculation in the x direction is performed according to the following formula:

[0025] ;

[0026] ;

[0027] And, the once interpolation calculation in the y direction is performed according to the following formula:

[0028] ;

[0029] Wherein, (x0, y0), (x1, y1), (x0, y1), (x1, y0) are four known points; f(x, y) is the pixel value of any point in the region surrounded by the four points (x0, y0), (x1, y1), (x0, y1), (x1, y0).

[0030] Preferably, the current frame image which is motion compensated according to the global motion offset is taken as the final compensated current frame image, or the current frame image which is motion compensated according to the global motion offset and then re-compensated by bilinear interpolation calculation is taken as the final compensated current frame image.

[0031] In another aspect, the application also provides a gray scale projection-based image stabilization system, which comprises:

[0032] an image sub-region division unit for dividing the adjacent kth frame and k+1th frame images into a plurality of sub-image regions according to the same division scheme, and each sub-image region of the former image has a sub-image region of the latter image corresponding thereto and having the same size;

[0033] a gray scale projection calculation unit for determining the gray scale projection curve of each sub-image region in the adjacent kth frame and k+1th frame images;

[0034] a motion offset calculation unit for performing cross-correlation operation on each group of corresponding sub-image region gray scale projection curves in the adjacent kth frame and k+1th frame images, and determining the global motion offset of the current frame image relative to the reference frame image according to the cross-correlation operation result;

[0035] an image compensation unit for performing motion compensation on the current frame image according to the global motion offset, and outputting the compensated current frame image.

[0036] Compared with the prior art, the application has the following beneficial effects:

[0037] The application compensates the current frame image by the global motion offset and bilinear interpolation calculation to generate a compensated image with higher resolution, so that the resolution quality of the compensated current frame image is higher, the PSNR value is higher and more stable, the visual effect of the image is obviously improved, and the scene details are effectively enhanced, thereby being beneficial to the observation of the human eye and the recognition ability and feature extraction ability of the subsequent algorithm to the target. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 a flow chart of the steps of the gray scale projection-based image stabilization method in the application;

[0039] Figure 2 the adjacent kth frame image and k+1th frame image, and the row gray scale projection curve and column gray scale projection curve of the kth frame image and k+1th frame image in the application;

[0040] Figure 3 a sub-image region division diagram corresponding to the adjacent kth frame image and k+1th frame image in the application;

[0041] Figure 4 This is a schematic diagram illustrating the compensation calculated using bilinear interpolation in this invention.

[0042] Figure 5 In this invention, the reference frame image, the current frame image, and the compensated current frame image are used.

[0043] Figure 6 The PSNR value curves are obtained by image stabilization processing based on grayscale projection in the prior art and by image stabilization processing using the method in this embodiment.

[0044] Figure 7 This is a schematic diagram of the image stabilization system based on grayscale projection in this invention. Detailed Implementation

[0045] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0046] Example 1

[0047] like Figure 1 As shown, this embodiment provides an image stabilization method based on grayscale projection, which includes the following steps:

[0048] S1. Select the k-th frame and the (k+1)-th frame from a series of consecutive frames. Figure 2 As shown in the figure, the k-th frame and the (k+1)-th frame are two adjacent frames with equal pixels. After selecting these two frames, the k-th frame and the (k+1)-th frame are preprocessed, for example, by Kalman filtering, to reduce the influence of noise.

[0049] S2. Divide the selected two adjacent frames into several sub-image regions according to the same division scheme, and each sub-image region of the k-th frame image has a unique corresponding sub-image region of the (k+1)-th frame image of the same size; and determine the grayscale projection curve of each sub-image region in the two adjacent frames.

[0050] like Figure 3As shown, for example, in this embodiment, the kth frame and the k+1th frame are first divided into left, middle and right regions along the x direction, and then the middle region is divided into two regions along the y direction, thereby forming four groups of sub-image regions corresponding to each other uniquely: sub-image region 1 and sub-image region 1', sub-image region 2 and sub-image region 2', sub-image region 3 and sub-image region 3', and sub-image region 4 and sub-image region 4', and the size of sub-image region 1 and sub-image region 1' is equal, the size of sub-image region 2 and sub-image region 2' is equal, the size of sub-image region 3 and sub-image region 3' is equal, and the size of sub-image region 4 and sub-image region 4' is equal;

[0051] Further, the row and column gray projection curves of each sub-image region (including all sub-image regions of the kth frame and all sub-image regions of the k+1th frame) can be calculated based on the Halcon gray projection algorithm or the like;

[0052] For example, as shown in Figure 2 (a) and (b) are the kth frame and the k+1th frame respectively, (c) and (d) are the row and column gray projection curves of the kth frame respectively, and (e) and (f) are the row and column gray projection curves of the k+1th frame respectively;

[0053] S3, the kth frame and the k+1th frame are respectively taken as a reference frame and a current frame, and the cross-correlation operation is performed on the gray projection curves of each group of corresponding sub-image regions, and the global motion offset of the current frame relative to the reference frame is determined according to the cross-correlation operation result;

[0054] Suppose P c (i) and P c (j) represent the row and column gray projection values of a sub-image region K0 (such as sub-image region 1) in the current frame respectively, P u (i) and P u (j) and P u (j) represent the row and column gray projection values of a sub-image region K0' (such as sub-image region 1') corresponding to the sub-image region K0 of the current frame in the reference frame, then the row and column gray projections of the corresponding sub-image regions in the current frame and the reference frame can be represented as:

[0055] ;

[0056] ;

[0057] Specifically, the following formula is used for cross-correlation calculation in this embodiment to obtain the row and column offsets of the sub-image region K0 of the current frame relative to the sub-image region K0' of the reference frame in each group of corresponding sub-image regions:

[0058] (1)

[0059] (2)

[0060] wherein, W u (Δt i ), W u (Δt j ) are respectively the row offset, the column offset of the sub-image region of the current frame image relative to the sub-image region of the reference frame image in each corresponding sub-image region; i, j represent respectively the row, the column where the pixel point is located; P u (i), P u (j) represent respectively the row gray projection value, the column gray projection value of the sub-image region of the reference frame image in each corresponding sub-image region; P c (j+Δt i ), P c (i+Δt j ) represent respectively the row gray projection value, the column gray projection value of the sub-image region of the current frame image in each corresponding sub-image region; M, N are respectively the width, the height of the current frame image and the width, the height of the reference frame image, and the width and the height of the current frame image and the reference frame image correspond to equal; Δt i , Δt j are respectively the row offset, the column offset of the sub-image region K0 of the current frame image relative to the sub-image region K0' of the reference frame image at the pixel point (i, j) in each corresponding sub-image region, which can be directly read according to the row gray projection curve and the column gray projection curve;

[0061] Further, the global motion offset of the current frame image relative to the reference frame image is determined according to the cross-correlation operation result, and specifically includes the following steps:

[0062] The maximum value in all row offsets and the maximum value in all column offsets are determined as the global motion offset of the current frame image relative to the reference frame image;

[0063] Or, the average value of all row offsets and the average value of all column offsets are taken, and the average value of the row offsets and the average value of the column offsets are taken as the global motion offset of the current frame image relative to the reference frame image;

[0064] It should be noted that the determination manner of the global motion offset can be determined according to a platform where the imaging device is located. For example, if the imaging device is mounted on a movable platform such as a vehicle or a ship, the shaking amplitudes of different parts are quite different. In this case, it is more appropriate to select the average value of the row offset and the average value of the column offset as the global motion offset. On the contrary, if the imaging device is mounted on an immovable fixed platform, the shaking amplitudes of different parts are close when the imaging device shakes due to external force. In this case, the maximum value of the offset can be directly determined as the global motion offset.

[0065] S4, motion compensation is performed on the current frame image according to the global motion offset, and a compensated current frame image is output.

[0066] For example, in the embodiment, if the current frame image is offset in the positive X direction relative to the reference frame image, the current frame image is moved in the negative X direction for compensation, and the movement amount is equal to the row offset of the global motion offset. Similarly, the current frame image can be moved in the Y direction, and the movement amount is equal to the row offset of the global motion offset. In this way, motion compensation is performed on the current frame image.

[0067] Further, after motion compensation is performed on the current frame image, some image pixels can be lost. Therefore, the current frame image obtained after motion compensation needs to be compensated again by bilinear interpolation to fill in the lost pixels.

[0068] Specifically, the current frame image obtained after motion compensation is compensated again by bilinear interpolation, including the following steps:

[0069] The compensation again is completed by twice linear interpolation calculation in the x direction and once linear interpolation calculation in the y direction. Specifically, as shown in FIG. 3, assuming that f(x, y) is a binary function, and it is assumed that the values of four points f(x0, y0), f(x1, y1), f(x0, y1), and f(x1, y0) are known. First, twice linear interpolation calculation is performed in the x direction to obtain the following calculation results: Figure 4

[0070] ; (3-1)

[0071] ; (3-2)

[0072] Then, once linear interpolation is performed in the y direction to obtain the following calculation results:

[0073] ; (3-3)

[0074] By combining formulas (3-1)-(3-3), the following bilinear interpolation result is obtained:

[0075] ​ ; wherein f(x,y) is the pixel value of any point in the region surrounded by the four points (x0,y0), (x1,y1), (x0,y1), (x1,y0).

[0076] The current frame image compensated according to the global motion offset is output as the final compensated current frame image, or the current frame image compensated according to the global motion offset and then compensated again through bilinear difference calculation is output as the final compensated current frame image.

[0077] Thus, the embodiment compensates the current frame image based on the global motion offset and bilinear interpolation calculation to generate a higher-resolution compensated image based on the original current frame image, as shown in Figure 5 The compensated current frame image has higher image resolution quality and the image visual effect is obviously improved, and the scene details are effectively enhanced, thus being beneficial to the observation of human eyes and the recognition ability and feature extraction ability of subsequent algorithms.

[0078] Further, the compensation steps of the current frame image are repeated, thus outputting a complete sequence of video images, reducing the spatial redundancy in the sequence of video images, improving the compression ratio of the video, and thus obtaining a steady visual effect.

[0079] The peak signal-to-noise ratio (PSNR) is an evaluation index for objectively evaluating the quality of the steady image algorithm, which reflects the peak signal-to-noise ratio γ between the reference frame image and the current frame image, and is defined as:

[0080] ;

[0081] wherein the mean square error (MSE) ΔεM is the deviation value of the pixel gray scale corresponding to the two images, and objectively reflects the similarity between the two images, and is defined as:

[0082] ;

[0083] wherein M and N respectively represent the width and height of the image; P1(i,j) represents the gray scale value of the pixel at the pixel point (i,j) in the compensated current frame image, and P0(i,j) represents the gray scale value of the pixel at the pixel point (i,j) in the current frame image without compensation.

[0084] The greater the PSNR value between the frames, the smaller the gray scale difference of the image, and a higher PSNR value reflects a better steady image quality. Figure 6The PSNR value curves obtained after the image stabilization processing based on the gray scale projection in the prior art (i.e., "before improvement") and the image stabilization processing by the image stabilization method in the embodiment (i.e., "after improvement") can be seen, and it can be seen that, compared with the traditional gray scale projection compensation method, the PSNR value after the image stabilization method in the embodiment is higher and more stable.

[0085] Embodiment 2:

[0086] The embodiment provides a gray scale projection-based image stabilization system for implementing the image stabilization method in Embodiment 1, as shown in Figure 7 The image stabilization system comprises:

[0087] An image processing unit 1 configured to pre-process adjacent kth and (k+1)th images;

[0088] An image sub-region division unit 2 configured to divide the adjacent kth and (k+1)th images into a plurality of sub-image regions according to the same division scheme, and each sub-image region of the former image has a sub-image region of the latter image corresponding thereto and having the same size;

[0089] A gray scale projection calculation unit 3 configured to determine the gray scale projection curve of each sub-image region in the adjacent kth and (k+1)th images, and the process is the same as that in step S2;

[0090] A motion offset calculation unit 4 configured to perform cross-correlation operation on each group of corresponding sub-image region projection curves in the adjacent kth and (k+1)th images, and determine the global motion offset of the current image relative to the reference image according to the cross-correlation operation result, and the process is the same as that in step S3;

[0091] An image compensation unit 5 configured to perform motion compensation on the current image according to the global motion offset, and output the compensated current image, and the process is the same as that in step S4.

[0092] In summary, the current image can be compensated based on the global motion offset and the bilinear interpolation calculation to generate a higher-resolution compensated image based on the original current image, so that the resolution quality of the compensated current image is higher, the PSNR value is higher and more stable, the visual effect of the image is obviously improved, and the scene details are effectively enhanced, thereby facilitating the observation of the human eye and the recognition ability and feature extraction ability of the subsequent algorithm to the target.

[0093] The above only describes the preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for image stabilization based on gray scale projection, characterized in that, The method comprises the following steps: Divide the adjacent kth frame and k+1th frame images into a plurality of sub-image regions, and each sub-image region of the kth frame image has a sub-image region of the k+1th frame image corresponding thereto and having the same size; Determine the gray projection curve of each sub-image region in the adjacent kth frame and k+1th frame images; Take the kth frame image and the k+1th frame image as a reference frame image and a current frame image respectively, perform cross-correlation operation on each group of corresponding sub-image region gray projection curves, and determine the global motion offset of the current frame image relative to the reference frame image according to the cross-correlation operation result; Perform motion compensation on the current frame image according to the global motion offset, and output the compensated current frame image.

2. The image stabilizing method according to claim 1, wherein Before the sub-image region division, filter the kth frame and k+1th frame images.

3. The image stabilizing method according to claim 1, wherein The cross-correlation calculation is performed by using the following formula to obtain the row offset and column offset of the sub-image region of the current frame image relative to the sub-image region of the reference frame image in each group of corresponding sub-image regions: ； (1) ； (2) wherein W u (Δt i ), W u (Δt j ) are respectively the row offset and the column offset of the sub-image region of the current frame image relative to the sub-image region of the reference frame image in each corresponding sub-image region; i, j represent respectively the row and the column where the pixel is located; P u (i), P u (j) represent respectively the row and the column gray scale projection values of the sub-image region of the reference frame image in each corresponding sub-image region; P c (j+Δt i ), P c (i+Δt j ) represent respectively the row and the column gray scale projection values of the sub-image region of the current frame image in each corresponding sub-image region; M, N are respectively the width and the height of the current frame image and the width and the height of the reference frame image, and the width and the height of the current frame image and the reference frame image correspond to be equal; Δt i , Δt j are respectively the row offset and the column offset of the sub-image region of the current frame image relative to the sub-image region of the reference frame image at the pixel (i, j) in each corresponding sub-image region; and the method comprises the steps of: The column offset.

4. The image stabilizing method according to claim 3, wherein The global motion offset of the current frame image relative to the reference frame image is determined according to the cross-correlation operation result, and specifically comprises the following steps: Determine the maximum value in all row offsets and the maximum value in all column offsets as the global motion offset of the current frame image relative to the reference frame image; Or, take the average value of all row offsets and the average value of all column offsets, and take the row offset average value and the column offset average value as the global motion offset of the current frame image relative to the reference frame image.

5. The image stabilizing method according to claim 1, wherein After the motion compensation on the current frame image according to the global motion offset, the current frame image obtained after the motion compensation is further compensated through bilinear difference calculation.

6. The image stabilizing method according to claim 5, wherein The motion compensation on the current frame image according to the global motion offset comprises: moving the current frame image along the X direction / Y direction, and the moving amount is equal to the row offset of the global motion offset.

7. The image stabilizing method according to Claim 5, wherein The further compensation on the current frame image obtained after the motion compensation is performed through bilinear difference calculation, comprising the following steps: The further compensation is completed by using the mode of twice linear interpolation calculation in the x direction and once interpolation calculation in the y direction.

8. The image stabilizing method according to Claim 7, wherein Twice linear interpolation calculation in the x direction is performed according to the following formula: ； ； And once interpolation calculation in the y direction is performed according to the following formula: ； Where (x0, y0), (x1, y1), (x0, y1), (x1, y0) are four known points; f(x, y) is the pixel value of any point in the region surrounded by the four points (x0, y0), (x1, y1), (x0, y1), (x1, y0).

9. The image stabilizing method according to claim 5, wherein The current frame image after the motion compensation according to the global motion offset is taken as the final compensated current frame image, or the current frame image after the motion compensation according to the global motion offset and the further compensation through bilinear difference calculation is taken as the final compensated current frame image.

10. A gray scale projection based image stabilization system, characterized by, It comprises: an image sub-region division unit, configured to divide each of the kth frame and the k+1th frame of images into a plurality of sub-image regions according to a same division scheme, and each sub-image region of the kth frame has a corresponding sub-image region of the k+1th frame which is unique to the sub-image region of the kth frame and has an equal size; a gray projection calculation unit, configured to determine a gray projection curve of each sub-image region in the kth frame and the k+1th frame of images; a motion offset calculation unit, configured to perform cross-correlation operation on each group of corresponding sub-image region gray projection curves in the kth frame and the k+1th frame of images, and determine a global motion offset of the current frame of images relative to the reference frame of images according to a result of the cross-correlation operation; an image compensation unit, configured to perform motion compensation on the current frame of images according to the global motion offset, and output the compensated current frame of images.

Images