A method for removing motion blur based on optical flow

Abstract

The application discloses a method for removing motion blur based on an optical flow method and relates to the field of video image processing. First, the collected video data is discretized into continuous images, each image is subjected to gray scale processing, and feature points are extracted according to the gray scale values; then, an image data buffer is constructed in the computer memory to store the image data; then, the optical flow method is used to estimate the continuous motion change of the image in the data buffer, the optical flow vector is calculated, and a motion error matrix is constructed; finally, the predicted coordinate information and the observed coordinate information of each pixel point on the next image are obtained, the Kalman gain coefficient is calculated, the motion blur is removed, and the image restoration is realized. The application can effectively remove the image motion blur, improves the image restoration efficiency under the premise of meeting the image restoration quality.

Description

Technical Field

[0001] This invention relates to the field of video image processing, and in particular to a method for eliminating motion blur based on optical flow. Background Technology

[0002] Image motion blur removal is a crucial aspect of video image processing. Image motion blur is caused by the relative motion between the camera and the scene during exposure. Images with motion blur offer a poor viewing experience, easily causing dizziness; they also hinder subsequent target recognition and tracking. Therefore, research on image motion blur removal is of great significance.

[0003] Methods for removing motion blur from images can be divided into model-based de-motion blur techniques and blind de-motion blur techniques, depending on whether prior knowledge is sufficient.

[0004] Model-based motion blur removal technology removes motion blur by establishing an image degradation model and then solving the inverse process using optimization equations. The difficulty of this method lies in the fact that the relative motion between the camera and the shooting scene is random during the process of establishing the degradation model, and the established degradation model often cannot meet the needs of image restoration.

[0005] Blind deblurring methods aim to recover blurred images and improve image processing in the absence of a model of the relative motion between the camera and the scene. These methods assume that neighboring pixels in the image are highly correlated and employ minimum mean square error estimation to reduce noise interference. Essentially, these methods optimize the image through iterative algorithms, which typically involve very high computational costs.

[0006] Existing methods have shortcomings in degradation model establishment and image restoration efficiency. For complex motion conditions, existing image restoration methods have significant room for improvement in both restoration effect and restoration quality. Summary of the Invention

[0007] In view of this, the present invention proposes a method for eliminating motion blur based on optical flow. The present invention can effectively remove motion blur from images and improve the efficiency of image restoration while meeting the requirements of image restoration quality.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for eliminating motion blur based on optical flow includes the following steps:

[0010] (1) Obtain video stream data from the camera and convert the video stream data into continuous image data through discretization processing according to the camera's refresh rate;

[0011] (2) Grayscale processing is performed on each image. For each image, the Fast corner extraction method is used to extract multiple feature points.

[0012] (3) Store the images sequentially in the image cache circular queue. Between two adjacent frames, divide all feature points into multiple groups of data according to their different gray values. The gray value interval between feature points in each group is greater than the gray value interval between each feature point in each group. Each group of data includes the coordinates and gray values ​​of all feature points between two adjacent frames within the current gray value range. In each group of data, select the two feature points with the smallest Mahalanobis distance between two adjacent frames as a set of matching feature points.

[0013] (4) Substitute multiple sets of matching feature point data on two adjacent frames into the optical flow equation, solve the optical flow vector according to the least squares algorithm, calculate the variance of all optical flow vectors in the current image buffer circular queue, and construct the real-time motion error matrix based on the obtained variance;

[0014] (5) Calculate the predicted coordinate information of each pixel in the next frame image based on the coordinate information of each pixel in the current image and the optical flow vector between the current image and the next frame image; observe and obtain the observation coordinate information of each pixel in the next frame image, and obtain the observation gain matrix and observation error matrix based on the predicted coordinate information and the observation coordinate information; calculate the Kalman gain coefficient from the observation gain matrix, the observation error matrix and the real-time motion error matrix.

[0015] (6) Based on the Kalman gain coefficient, obtain the weighting coefficients of the coordinate information of each pixel in the current image and the weighting coefficients of the predicted coordinate information of each pixel in the next frame image. Multiply the coordinate information of each pixel in the current image and the predicted coordinate information of each pixel in the next frame image by their respective weighting coefficients and then add them together to obtain the coordinates of each pixel in the repaired image of the current image.

[0016] Complete motion blur removal based on optical flow method.

[0017] Furthermore, in step (2), for each image, the Fast corner extraction method is used to extract multiple feature points. The specific method is as follows:

[0018] For the image at time t, the FAST corner extraction method is used. The neighborhood range and grayscale threshold are set. Each pixel in the image at time t is traversed. The grayscale value of the pixel at coordinates (x,y) in the image at time t is I(x,y,t). If the difference between the grayscale value of the pixel and the grayscale values ​​of all pixels in its neighborhood range is greater than the grayscale threshold, then the current pixel is selected as one of the feature points in the image at time t.

[0019] Furthermore, in step (1), the camera refresh rate is f, and in step (3), the length of the image buffer circular queue is n+1, where n is equal to f, and the time interval between two adjacent frames is... The specific method for step (3) is as follows:

[0020] The images are stored sequentially in an image buffer circular queue, and matching feature points on two adjacent frames are selected according to the following formula:

[0021]

[0022] Where I(x+Δx,y+Δy,t+Δt) is the gray value of a certain feature point on the image at time t+Δt; the first formula means selecting feature points with similar gray values ​​between two adjacent frames, thereby dividing all feature points between two adjacent frames into multiple groups of data, and the gray value interval between feature points in each group is greater than the gray value interval between each feature point in each group of data.

[0023] Z t Let Z be the coordinates of a feature point on the image at time t. t+Δt D represents the coordinates of a feature point in the same data set as the given feature point in the next frame image. t,t+Δt The second formula is used to calculate the Mahalanobis distance between two feature points. In each set of data selected by the first formula, the Mahalanobis distance between two adjacent feature points in the image is calculated. The two feature points with the smallest Mahalanobis distance are selected as a set of matching feature points.

[0024] Furthermore, in step (4), multiple sets of matching feature point data from two adjacent frames are substituted into the optical flow equation, and the optical flow vector is solved using the least squares algorithm as follows:

[0025] I x u+I y v+I t =0

[0026] in, I x with I y I represents the rate at which the gray values ​​of two matched feature points change along the x and y directions, respectively. t The grayscale value between two matched feature points changes at a rate between two adjacent frames; u and v are the velocities of the two matched feature points moving along the x and y directions, respectively, i.e., the optical flow vectors being solved, as follows:

[0027]

[0028] in, The covariance matrix represents the variation of gray values ​​of matching feature points along the x and y directions between two adjacent image frames. The variance vector along the x and y directions represents the gray values ​​of the matching feature points between two adjacent frames. m indicates that there are m pairs of matching feature points between two adjacent frames.

[0029] In the current image buffer circular queue, the latest n-1 optical flow vectors are obtained in real time. In step (4), the variance of all optical flow vectors in the current image buffer circular queue is calculated. The real-time motion error matrix P constructed based on the obtained variance is as follows:

[0030]

[0031] in,

[0032]

[0033]

[0034] The variance of the velocity change along the x-direction in the current n frames;

[0035]

[0036]

[0037] Let V be the variance of the velocity change along the y-direction in the current n frames of the image.

[0038] Furthermore, the method for calculating the predicted coordinate information of each pixel in the next frame image based on the coordinate information of each pixel in the current image and the optical flow vector between the current image and the next frame image in step (5) is as follows:

[0039] X t+Δt =AX t +D

[0040] Among them, X t Let X be the coordinates of each pixel in the image at time t. t+Δt To predict the coordinates of each pixel in the next frame of the image, A is the transition matrix. D is the control matrix.

[0041] In step (5), the observation coordinate information of each pixel in the next frame image is obtained. The observation gain matrix and observation error matrix are obtained based on the predicted coordinate information and the observed coordinate information, as follows:

[0042] X' t+Δt =HX t+Δt +R

[0043] Where X' t+Δt Let H be the coordinates of each pixel in the image at time t+Δt, H be the observation gain matrix, and R be the observation error matrix. Here, we assume that the observed values ​​and the predicted results are approximately the same, so H is the identity matrix and R is the zero matrix.

[0044] In step (5), the Kalman gain coefficient K is calculated from the observation gain matrix H, the observation error matrix R, and the real-time motion error matrix P as follows:

[0045]

[0046] Furthermore, the method for step (6) is as follows:

[0047]

[0048] in, Let E be the coordinates of each pixel in the restored image at time t, and E be the identity matrix.

[0049] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0050] 1. This invention constructs an image cache circular queue and estimates the relative motion between the camera and the scene using the data in the image cache circular queue, resulting in a more accurate motion model.

[0051] 2. This invention uses optical flow equations for solving, which has high algorithm efficiency, low computer configuration requirements, and is easy to port to embedded devices.

[0052] 3. This invention uses a Kalman filter for image inpainting and simultaneously solves for the motion error matrix of the data in the image buffer circular queue, resulting in a more accurate model.

[0053] 4. This invention uses a Kalman filter for image restoration, which is faster than existing deconvolution algorithms. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the system connection in an embodiment of the present invention.

[0055] Figure 2 This is a flowchart of the steps in an embodiment of the present invention.

[0056] Figure 3 This is a flowchart of the image restoration process in an embodiment of the present invention. Detailed Implementation

[0057] The following is in conjunction with the appendix Figure 1-3 The present invention will be further described in detail with reference to specific embodiments.

[0058] like Figure 1 As shown, the system in this example consists of two parts: a camera and a PC processor. The camera performs image capture, while the PC processor performs image processing and display. The PC processor comprises a data acquisition unit, an image buffer, and an image processor. The data acquisition unit performs data acquisition, image preprocessing, and feature point extraction. The image buffer performs image circular queue storage, optical flow equation establishment and solution, and motion error matrix solution. The image processor performs feature point matching, Kalman filter construction, image inpainting, and image display.

[0059] like Figure 2 The flowchart shown is a flowchart of this embodiment. Referring to the flowchart, the implementation process of this embodiment is as follows:

[0060] A method for eliminating motion blur based on optical flow includes the following steps:

[0061] (1) Obtain video stream data from the camera and convert the video stream data into continuous image data through discretization processing according to the camera's refresh rate;

[0062] (2) Grayscale processing is performed on each image. For each image, the Fast corner extraction method is used to extract multiple feature points.

[0063] (3) Store the images sequentially in the image cache circular queue. Between two adjacent frames, divide all feature points into multiple groups of data according to their different gray values. The gray value interval between feature points in each group is greater than the gray value interval between each feature point in each group. Each group of data includes the coordinates and gray values ​​of all feature points between two adjacent frames within the current gray value range. In each group of data, select the two feature points with the smallest Mahalanobis distance between two adjacent frames as a set of matching feature points.

[0064] (4) Substitute multiple sets of matching feature point data on two adjacent frames into the optical flow equation, solve the optical flow vector according to the least squares algorithm, calculate the variance of all optical flow vectors in the current image buffer circular queue, and construct the real-time motion error matrix based on the obtained variance;

[0065] (5) Calculate the predicted coordinate information of each pixel in the next frame image based on the coordinate information of each pixel in the current image and the optical flow vector between the current image and the next frame image; observe and obtain the observation coordinate information of each pixel in the next frame image, and obtain the observation gain matrix and observation error matrix based on the predicted coordinate information and the observation coordinate information; calculate the Kalman gain coefficient from the observation gain matrix, the observation error matrix and the real-time motion error matrix.

[0066] (6) Figure 3As shown, the weighting coefficients of the coordinate information of each pixel in the current image and the weighting coefficients of the predicted coordinate information of each pixel in the next frame image are obtained based on the Kalman gain coefficient. The coordinate information of each pixel in the current image and the predicted coordinate information of each pixel in the next frame image are multiplied by their respective weighting coefficients and then added together to obtain the coordinates of each pixel in the restored image of the current image.

[0067] Complete motion blur removal based on optical flow method.

[0068] Furthermore, in step (2), for each image, the Fast corner extraction method is used to extract multiple feature points. The specific method is as follows:

[0069] For the image at time t, the FAST corner extraction method is used. The neighborhood range and grayscale threshold are set. Each pixel in the image at time t is traversed. The grayscale value of the pixel at coordinates (x,y) in the image at time t is I(x,y,t). If the difference between the grayscale value of the pixel and the grayscale values ​​of all pixels in its neighborhood range is greater than the grayscale threshold, then the current pixel is selected as one of the feature points in the image at time t.

[0070] Furthermore, in step (1), the camera refresh rate is f, and in step (3), the length of the image buffer circular queue is n+1, where n is equal to f, and the time interval between two adjacent frames is... The specific method for step (3) is as follows:

[0071] The images are stored sequentially in an image buffer circular queue, and matching feature points on two adjacent frames are selected according to the following formula:

[0072]

[0073] Where I(x+Δx,y+Δy,t+Δt) is the gray value of a certain feature point on the image at time t+Δt; the first formula means selecting feature points with similar gray values ​​between two adjacent frames, thereby dividing all feature points between two adjacent frames into multiple groups of data, and the gray value interval between feature points in each group is greater than the gray value interval between each feature point in each group of data.

[0074] Z t Let Z be the coordinates of a feature point on the image at time t. t+Δt D represents the coordinates of a feature point in the same data set as the given feature point in the next frame image. t,t+Δt The second formula is used to calculate the Mahalanobis distance between two feature points. In each set of data selected by the first formula, the Mahalanobis distance between two adjacent feature points in the image is calculated. The two feature points with the smallest Mahalanobis distance are selected as a set of matching feature points.

[0075] Furthermore, in step (4), multiple sets of matching feature point data from two adjacent frames are substituted into the optical flow equation, and the optical flow vector is solved using the least squares algorithm as follows:

[0076] I x u+I y v+I t =0

[0077] in, I x with I y I represents the rate at which the gray values ​​of two matched feature points change along the x and y directions, respectively. t The grayscale value between two matched feature points changes at a rate between two adjacent frames; u and v are the velocities of the two matched feature points moving along the x and y directions, respectively, i.e., the optical flow vectors being solved, as follows:

[0078]

[0079] in, The covariance matrix represents the variation of gray values ​​of matching feature points along the x and y directions between two adjacent image frames. The variance vector along the x and y directions represents the gray values ​​of the matching feature points between two adjacent frames. m indicates that there are m pairs of matching feature points between two adjacent frames.

[0080] In the current image buffer circular queue, the latest n-1 optical flow vectors are obtained in real time. In step (4), the variance of all optical flow vectors in the current image buffer circular queue is calculated. The real-time motion error matrix P constructed based on the obtained variance is as follows:

[0081]

[0082] in,

[0083]

[0084]

[0085] The variance of the velocity change along the x-direction in the current n frames;

[0086]

[0087]

[0088] Let V be the variance of the velocity change along the y-direction in the current n frames of the image.

[0089] Furthermore, the method for calculating the predicted coordinate information of each pixel in the next frame image based on the coordinate information of each pixel in the current image and the optical flow vector between the current image and the next frame image in step (5) is as follows:

[0090] X t+Δt =AX t +D

[0091] Among them, X t Let X be the coordinates of each pixel in the image at time t. t+Δt To predict the coordinates of each pixel in the next frame of the image, A is the transition matrix. D is the control matrix.

[0092] In step (5), the observation coordinate information of each pixel in the next frame image is obtained. The observation gain matrix and observation error matrix are obtained based on the predicted coordinate information and the observed coordinate information, as follows:

[0093] X' t+Δt =HX t+Δt +R

[0094] Where X' t+Δt Let H be the coordinates of each pixel in the image at time t+Δt, H be the observation gain matrix, and R be the observation error matrix. Here, we assume that the observed values ​​and the predicted results are approximately the same, so H is the identity matrix and R is the zero matrix.

[0095] In step (5), the Kalman gain coefficient K is calculated from the observation gain matrix H, the observation error matrix R, and the real-time motion error matrix P as follows:

[0096]

[0097] Furthermore, the method for step (6) is as follows:

[0098]

[0099] in, Let E be the coordinates of each pixel in the restored image at time t, and E be the identity matrix.

[0100] In summary, this invention, by selecting matching feature points and setting a Kalman filter for image restoration, can effectively remove motion blur from images and improve image restoration efficiency while meeting image restoration quality requirements.

[0101] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the present invention as defined in the appended claims should be included within the protection scope of the present invention.

Claims

1. A method for eliminating motion blur based on optical flow, characterized in that, Includes the following steps: (1) Obtain video stream data from the camera and convert the video stream data into continuous image data through discretization processing according to the camera's refresh rate; (2) Grayscale processing is performed on each image. For each image, the Fast corner extraction method is used to extract multiple feature points. (3) Store the images sequentially in the image cache circular queue. Between two adjacent frames, divide all feature points into multiple groups of data according to their different gray values. The gray value interval between feature points in each group is greater than the gray value interval between each feature point in each group. Each group of data includes the coordinates and gray values ​​of all feature points between two adjacent frames within the current gray value range. In each group of data, select the two feature points with the smallest Mahalanobis distance between two adjacent frames as a set of matching feature points. (4) Substitute multiple sets of matching feature point data on two adjacent frames into the optical flow equation, solve the optical flow vector according to the least squares algorithm, calculate the variance of all optical flow vectors in the current image buffer circular queue, and construct the real-time motion error matrix based on the obtained variance; (5) Calculate the predicted coordinate information of each pixel in the next frame image based on the coordinate information of each pixel in the current image and the optical flow vector between the current image and the next frame image; The observation coordinate information of each pixel in the next frame image is obtained by observation, and the observation gain matrix and observation error matrix are obtained based on the predicted coordinate information and the observed coordinate information. The Kalman gain coefficient is calculated from the observation gain matrix, the observation error matrix, and the real-time motion error matrix. (6) Based on the Kalman gain coefficient, obtain the weighting coefficients of the coordinate information of each pixel in the current image and the weighting coefficients of the predicted coordinate information of each pixel in the next frame image. Multiply the coordinate information of each pixel in the current image and the predicted coordinate information of each pixel in the next frame image by their respective weighting coefficients and then add them together to obtain the coordinates of each pixel in the repaired image of the current image. Complete motion blur removal based on optical flow method.

2. The method for eliminating motion blur based on optical flow according to claim 1, characterized in that, In step (2), for each image, the Fast corner extraction method is used to extract multiple feature points. The specific method is as follows: For the image at time t, the FAST corner extraction method is used. The neighborhood range and grayscale threshold are set. Each pixel in the image at time t is traversed. The grayscale value of the pixel at coordinates (x,y) in the image at time t is I(x,y,t). If the difference between the grayscale value of the pixel and the grayscale values ​​of all pixels in its neighborhood range is greater than the grayscale threshold, then the current pixel is selected as one of the feature points in the image at time t.

3. The method for eliminating motion blur based on optical flow according to claim 2, characterized in that, In step (1), the camera's refresh rate is f. In step (3), the length of the image buffer circular queue is n+1, where n is equal to f. The time interval between two adjacent frames is... The specific method for step (3) is as follows: The images are stored sequentially in an image buffer circular queue, and matching feature points on two adjacent frames are selected according to the following formula: Where I(x+Δx,y+Δy,t+Δt) is the gray value of a certain feature point on the image at time t+Δt; the first formula means selecting feature points with similar gray values ​​between two adjacent frames, thereby dividing all feature points between two adjacent frames into multiple groups of data, and the gray value interval between feature points in each group is greater than the gray value interval between each feature point in each group of data. Z t Let Z be the coordinates of a feature point on the image at time t. t+Δt D represents the coordinates of a feature point in the same data set as the given feature point in the next frame image. t,t+Δt The second formula is used to calculate the Mahalanobis distance between two feature points. In each set of data selected by the first formula, the Mahalanobis distance between two adjacent feature points in the image is calculated. The two feature points with the smallest Mahalanobis distance are selected as a set of matching feature points.

4. The method for eliminating motion blur based on optical flow according to claim 3, characterized in that, In step (4), multiple sets of matching feature point data from two adjacent frames are substituted into the optical flow equation, and the optical flow vector is solved using the least squares algorithm as follows: I x u+I y v+I t =0 in, I x with I y I represents the rate at which the gray values ​​of two matched feature points change along the x and y directions, respectively. t The grayscale value between two matched feature points changes at a rate between two adjacent frames; u and v are the velocities of the two matched feature points moving along the x and y directions, respectively, i.e., the optical flow vectors being solved, as follows: in, The covariance matrix represents the variation of gray values ​​of matching feature points along the x and y directions between two adjacent image frames. The variance vector along the x and y directions represents the gray values ​​of the matching feature points between two adjacent frames. m indicates that there are m pairs of matching feature points between two adjacent frames. In the current image buffer circular queue, the latest n-1 optical flow vectors are obtained in real time. In step (4), the variance of all optical flow vectors in the current image buffer circular queue is calculated. The real-time motion error matrix P constructed based on the obtained variance is as follows: in, The variance of the velocity change along the x-direction in the current n frames; Let V be the variance of the velocity change along the y-direction in the current n frames of the image.

5. The method for eliminating motion blur based on optical flow according to claim 4, characterized in that, The method for calculating the predicted coordinate information of each pixel in the next frame image based on the coordinate information of each pixel in the current image and the optical flow vector between the current image and the next frame image in step (5) is as follows: X t+Δt =AX t +D Among them, X t Let X be the coordinates of each pixel in the image at time t. t+Δt To predict the coordinates of each pixel in the next frame of the image, A is the transition matrix. D is the control matrix. In step (5), the observation coordinate information of each pixel in the next frame image is obtained. The observation gain matrix and observation error matrix are obtained based on the predicted coordinate information and the observed coordinate information, as follows: X' t+Δt =HX t+Δt +R Where X' t+Δt Let H be the coordinates of each pixel in the image at time t+Δt, H be the observation gain matrix, and R be the observation error matrix. Here, we assume that the observed values ​​and the predicted results are approximately the same, so H is the identity matrix and R is the zero matrix. In step (5), the Kalman gain coefficient K is calculated from the observation gain matrix H, the observation error matrix R, and the real-time motion error matrix P as follows:

6. The method for eliminating motion blur based on optical flow according to claim 5, characterized in that, The specific method for step (6) is as follows: in, Let E be the coordinates of each pixel in the restored image at time t, and E be the identity matrix.

Images