Method and apparatus for deblurring a motion-blurred image

Abstract

This invention proposes a method and apparatus for deblurring motion-blurred images, belonging to the field of machine vision technology. The method includes: acquiring inertial sensor data recording camera motion information within the exposure time of the motion-blurred image; estimating an initial blur kernel using the inertial sensor data; and iteratively solving the motion-blurred image using the initial blur kernel by establishing an image blur model with blur kernel error to obtain a deblurred, clear image. This invention transforms inertial sensor data error into blur kernel error, and utilizes a neural network to learn kernel error priors in the iterative optimization solution, greatly improving the robustness of the deblurring method to sensor errors and significantly enhancing the deblurring effect of motion-blurred images.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically relating to a method and apparatus for deblurring motion-blurred images. Background Technology

[0002] When a camera shakes during exposure, the captured image becomes motion-blurred, which not only degrades image quality but also negatively impacts subsequent image applications. In recent years, deblurring methods for motion-blurred images have been an active research area. Typically, these methods require prior knowledge or additional information to obtain a sharp image. By recording data from the camera's inertial sensors (such as gyroscopes and accelerometers) during exposure, motion information of the camera within the exposure time can be obtained, thus aiding the deblurring process for motion-blurred images.

[0003] Currently, inertial sensor-assisted deblurring methods typically assume, or slightly relax this assumption, that the data recorded by the sensor is reliable. However, due to time synchronization errors and noise, there are often errors between inertial sensor data and actual motion information. Therefore, existing deblurring methods are not robust enough to sensor errors, especially when the inertial sensor data error is large, the deblurring performance will significantly decrease. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a method and apparatus for deblurring motion-blurred images. This invention transforms inertial sensor data errors into blur kernel errors. In the iterative optimization process, a neural network is used to learn prior knowledge related to the kernel error, greatly improving the robustness of the deblurring method to sensor errors and significantly enhancing the deblurring effect on motion-blurred images.

[0005] A first aspect of this invention provides a method for deblurring motion-blurred images, comprising:

[0006] Acquire inertial sensor data that records camera motion information during the exposure time of a motion-blurred image;

[0007] The initial ambiguity kernel is estimated using the inertial sensor data;

[0008] Using the initial blur kernel, an image blur model with blur kernel error is established, and the motion-blurred image is iteratively solved to obtain the clear image after deblurring.

[0009] In one specific embodiment of the present invention, the inertial sensor data includes: the camera's three-axis acceleration and three-axis angular velocity.

[0010] In one specific embodiment of the present invention, estimating the initial ambiguity kernel using the inertial sensor data includes:

[0011] 1) Establish the geometric blur model for the motion-blurred image, with the following expression:

[0012]

[0013] In the formula, f is the sharp image, g is the motion-blurred image, x represents the pixel position in homogeneous coordinates, and H... t Let g(x) represent the homography matrix at time t after the start of exposure, and g(x) represent the corresponding pixel value at coordinate x in the motion-blurred image. Let f(H) represent the homography matrix at time t after the start of exposure. t x) represents the coordinates H in the clear image. t The corresponding pixel value at x, n represents the noise term, and Np represents the number of camera poses during the exposure time;

[0014] Mapping the initial projection at the initial exposure time (t=0) to any other time t, the homography matrix H t Represented as:

[0015]

[0016] In the formula, R t It is the rotation transformation matrix at time t after the start of exposure, T t N is the translation vector at time t after the exposure begins. T is a unit vector orthogonal to the imaging plane, d is the distance between the object being photographed and the camera; M is the camera eigenvalue matrix, expressed as follows:

[0017]

[0018] In the formula, f represents the focal length, (O x O y ) represents the camera optical center coordinates, O x and O y These are the x-axis and y-axis coordinates of the optical center, respectively.

[0019] 2) Calculate the rotation transformation matrix and translation vector using inertial sensor data;

[0020] The expression for calculating the rotation transformation matrix is ​​as follows:

[0021]

[0022]

[0023] In the formula, ω t =[ω tx ω ty ωtz ] T It is the triaxial angular velocity value at time t after the start of exposure, ω tx ω ty ω tz ωt represents the angular velocity values ​​of the x, y, and z axes at time t; Δt is the sampling time interval.

[0024] The expression for calculating the translation vector is:

[0025]

[0026] T t =T t-1 +(v t-1 +v t )*Δt / 2 (7)

[0027] In the formula, a t =[a tx a ty a tz ] T It is the triaxial acceleration value at time t after the start of exposure, a tx a ty a tz These are the acceleration values ​​along the x, y, and z axes at time t, respectively.

[0028] 3) Calculate the initial fuzzy kernel based on the geometric fuzzy model;

[0029] Based on the results of step 2), the homography matrix H for all sampling times is calculated using equation (2). t Then, the initial blur kernel is obtained based on the projection trajectory of the center pixel.

[0030] In a specific embodiment of the present invention, the image fuzzing model expression with fuzzy kernel error is as follows:

[0031]

[0032] In the formula, Let K be the initial fuzzy kernel and ΔK be the fuzzy kernel error.

[0033] In one specific embodiment of the present invention, the method further includes:

[0034] Let the auxiliary variable u represent the error term ΔKf. An optimization problem based on total least squares for image deblurring is established, expressed as follows:

[0035]

[0036] In the formula, Φ(·) represents the regularization term with respect to the image prior; λ is the Lagrange coefficient, which takes a value between 0 and 1;

[0037] Let z be the high-frequency component of the motion-blurred image. Applying semi-quadratic splitting, the optimization problem shown in equation (9) is updated as follows:

[0038]

[0039] In the formula, Let Γ represent the regularization term of the prior related to the fuzzy kernel error, Γ represent the high-pass filter, and ρ represent the regularization term of the prior related to the high-frequency component z.

[0040] In one specific embodiment of the present invention, the method further includes:

[0041] Solving equation (10) using the alternating iterative method yields the following iterative expression:

[0042]

[0043]

[0044]

[0045] Where λ is a preset coefficient, with a value between 0 and 1; during iteration, the motion-blurred image g and the initial blur kernel are used. As input, the error term solution u of the 0th iteration. (0) and the high-frequency component solution z of the 0th iteration (0) All values ​​are set to 0, and the initial sharp image solution f is obtained by using the inverse solution of the discrete Fourier transform (13). (0) ;

[0046] In the nth iteration, where n≥1, the clear image solution f is obtained from the (n-1)th iteration. (n-1) As input, the high-frequency component correlation prior of the image is learned using the denoising convolutional neural network preset in the nth iteration, and the high-frequency component solution z in the nth iteration of equation (11) is obtained. (n) ; with f (n-1) As input, the fuzzy kernel error related prior is learned using the dual-channel U-shaped network preset during the nth iteration, and the error term solution u of the nth iteration in equation (12) is obtained. n ; using the motion-blurred image g and the initial blur kernel z(n) and u (n) Using the discrete Fourier transform as input, the clear image solution f of the nth iteration is obtained by solving equation (13). (n) ;

[0047] When the number of iterations n reaches the preset total number of iterations N, the clear image solution f is obtained. (n) This is the final, clear image.

[0048] In a specific embodiment of the present invention, before solving equation (10) using the alternating iterative method, the method further includes:

[0049] The deblurring neural network is constructed based on the denoising convolutional neural network and the dual-channel U-shaped network;

[0050] The number of layers in the deblurring neural network is the same as the total number of iterations. Each layer of the deblurring neural network contains one denoising convolutional neural network and one dual-channel U-shaped network to correspond to the iterative solution of each step. The output of the previous layer's denoising neural network and dual-channel U-shaped network is used as the input of the next layer's denoising neural network and dual-channel U-shaped network to perform the next iteration by solving equation (13) through discrete Fourier transform.

[0051] In one specific embodiment of the present invention, the method further includes:

[0052] Training the deblurring neural network includes:

[0053] 1) Construct a training set; each training sample in the training set contains a pair of sharp images and motion-blurred images, and a corresponding blur kernel with error;

[0054] 2) Construct the deblurring neural network;

[0055] 3) Train the deblurring neural network using the training set to obtain the trained deblurring neural network; wherein, the overall loss function for training the deblurring neural network is...

[0056]

[0057] Among them, f j The j-th training sample is a clear image, where J is the number of training samples and N is the number of iterations; μ i The coefficient for the preset i-th iteration is between 0 and 1. This is the sharp image solution corresponding to the j-th training sample image output in the i-th iteration.

[0058] In one specific embodiment of the present invention, the method further includes:

[0059] The motion-blurred image in the training samples is obtained by convolving the clear image with the corresponding blur kernel, and the blur kernel is calculated using simulated inertial sensor data;

[0060] The fuzzy kernel with errors in the training samples is obtained by adding a set synchronization delay and preset white noise to the simulated inertial sensor data.

[0061] A second aspect of the present invention provides a deblurring device for motion-blurred images, comprising:

[0062] An inertial sensor data acquisition module is used to acquire inertial sensor data that records camera motion information during the exposure time of a motion-blurred image.

[0063] An initial fuzzy kernel estimation module is used to estimate the initial fuzzy kernel using the inertial sensor data;

[0064] The deblurring module is used to iteratively solve the motion-blurred image by establishing an image blurring model with blur kernel error using the initial blur kernel, and obtain the clear image after deblurring.

[0065] A third aspect of the present invention provides an electronic device comprising:

[0066] At least one processor; and a memory communicatively connected to said at least one processor;

[0067] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to perform the aforementioned method for deblurring a motion-blurred image.

[0068] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to perform the above-described method for deblurring motion-blurred images.

[0069] Features and beneficial effects of the present invention:

[0070] This invention first establishes a geometric blur model for motion-blurred images and calculates an initial blur kernel using inertial sensor data, transforming inertial sensor data errors into blur kernel errors. Then, an image blur model with blur kernel errors is established, and an optimization scheme based on Total-Least-Squares (TLS) estimation is used for iterative solution. Part of the iterative process is implemented using a neural network to learn the relevant priors of the image and blur kernel errors. This invention overcomes the shortcomings of existing deblurring methods in terms of insufficient robustness to sensor errors, ensuring deblurring performance even with errors in inertial sensor data, and providing support for the wider application of deblurring methods in industrial fields. Attached Figure Description

[0071] Figure 1 This is an overall flowchart of a motion-blurred image deblurring method according to an embodiment of the present invention. Detailed Implementation

[0072] This invention proposes a method and apparatus for deblurring motion-blurred images, which are further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] A first aspect of this invention provides a method for deblurring motion-blurred images, comprising:

[0074] Acquire inertial sensor data that records camera motion information during the exposure time of a motion-blurred image;

[0075] The initial ambiguity kernel is estimated using the inertial sensor data;

[0076] Using the initial blur kernel, an image blur model with blur kernel error is established, and the motion-blurred image is iteratively solved to obtain the clear image after deblurring.

[0077] In a specific embodiment of the present invention, the overall process of the method for deblurring motion-blurred images is as follows: Figure 1 As shown, it includes the following steps:

[0078] 1) Acquire inertial sensor data that records camera motion information during the exposure time of a motion-blurred image.

[0079] When the camera shakes during exposure, a motion-blurred image will ultimately be obtained. In this embodiment, to obtain camera motion information during the exposure time, an inertial sensor containing a three-axis accelerometer and a three-axis gyroscope is rigidly connected to the camera. Simultaneously, the camera's exposure trigger signal is connected to the inertial sensor's control terminal, ensuring that the inertial sensor outputs data only during the camera's exposure time. For each motion-blurred image generated, a corresponding set of inertial sensor data is obtained, including the three-axis acceleration and three-axis angular velocity of the inertial sensor at each sampling moment during the motion time (i.e., from the start to the end of the exposure). This embodiment of the invention does not have special requirements regarding the sampling frequency.

[0080] 2) Estimate the initial ambiguity kernel using the inertial sensor data obtained in step 1). The specific steps are as follows:

[0081] 2-1) Establish a geometric blur model for motion-blurred images.

[0082] In this embodiment, in order to incorporate inertial sensor data that records camera motion information, the motion-blurred image is represented as the sum of multiple sharp images under a series of projected motions within the exposure time, that is:

[0083]

[0084] In the formula, f is the sharp image, g is the motion-blurred image, x represents the pixel position in homogeneous coordinates, and H... tLet g(x) represent the homography matrix at time t after the start of exposure, and g(x) represent the corresponding pixel value at coordinate x in the motion-blurred image. Let f(H) represent the homography matrix at time t after the start of exposure. t x) represents the coordinates H in the clear image. t The corresponding pixel value at x, n represents the noise term, and Np represents the number of camera poses during the exposure time (obtained by exposing time × sensor sampling frequency). Consider mapping the initial projection at t=0 (i.e., the initial exposure time) to any other time t, the homography matrix H t It can be represented as:

[0085]

[0086] In the formula, R t It is the rotation transformation matrix at time t after the start of exposure, T t N is the translation vector at time t after the exposure begins. T It is a unit vector orthogonal to the imaging plane, d is the distance between the subject and the camera, and M is the camera eigenvalue matrix, which can be expressed using the focal length f and the camera optical center coordinates (O). x O y ) is represented as:

[0087]

[0088] In the formula, O x and O y These are the x-axis and y-axis coordinates of the optical center, respectively.

[0089] 2-2) Calculate the rotation transformation matrix and translation vector using inertial sensor data.

[0090] In this embodiment, the parameters related to camera motion include the rotation transformation matrix R. t Translation vector T t The value can be calculated using measurement data from the gyroscope and accelerometer, respectively. Assuming the rotation center is located at the camera's optical center, the expression for calculating the rotation transformation matrix is:

[0091]

[0092]

[0093] In the formula, ω t =[ω tx ω ty ω tz ] T It is the triaxial angular velocity value measured by the gyroscope at time t after the start of exposure, ω tx ω ty ω tzΔt represents the angular velocity values ​​of the x, y, and z axes at time t; Δt is the sampling time interval. In this embodiment, the exposure time / sampling time interval must be no less than 20.

[0094] After obtaining the rotation transformation matrix, the expression for calculating the translation vector is:

[0095]

[0096] T t =T t-1 +(v t-1 +v t )*Δt / 2 (7)

[0097] In the formula, a t =[a tx a ty a tz ] T It is the triaxial acceleration value measured by the accelerometer at time t after the start of exposure, a tx a ty a tz These are the acceleration values ​​along the x, y, and z axes at time t, respectively.

[0098] 2-3) Calculate the initial fuzzy kernel based on the geometric fuzzy model.

[0099] In this embodiment, based on the result of step 2-2-), the homography matrix H for all sampling times is calculated using equation (2). t Then, the initial blur kernel can be calculated using the projection trajectory of the center pixel. The calculation process is as follows: set the value of the center pixel of the image to 255, and set the values ​​of all other pixels to 0 to obtain the image f. c (x); then the noise term n is set to 0, and f c (x) as f(H) t x) Input equation (1) to calculate the corresponding motion-blurred image g. c (x) is the initial fuzzy kernel.

[0100] 3) Using the initial blur kernel obtained in step 2), an image blur model with blur kernel error is established, and the motion-blurred image is iteratively solved to obtain the deblurred, clear image. The specific steps are as follows:

[0101] 3-1) Establish an image fuzzy model with fuzzy kernel error.

[0102] In this embodiment, after calculating the initial blur kernel, the sensor error is transformed into the blur kernel error ΔK, and the image motion model can be described as follows:

[0103]

[0104] In the formula, f represents the sharp image, g represents the motion-blurred image, and n represents the noise term. The initial fuzzy kernel is obtained from step 1).

[0105] 3-2) Establish a defuzzification optimization problem based on total least squares TLS.

[0106] In this embodiment, an auxiliary variable u is introduced to represent the error term ΔKf. The optimization problem for image deblurring based on TLS can be expressed as:

[0107]

[0108] In the formula, Φ(·) represents the regularization term with respect to the prior of a specific image; λ is the Lagrange coefficient, which takes a value between 0 and 1.

[0109] Introducing the high-frequency component z of the motion-blurred image and applying semi-quadratic splitting, the optimization problem shown in equation (9) is updated as follows:

[0110]

[0111] In the formula, Let Γ represent the regularization term of the prior related to the fuzzy kernel error, Γ represent the high-pass filter, and ρ represent the regularization term of the prior related to the high-frequency component z.

[0112] 3-3) Iteratively solve the optimization problem established in step 2) to obtain the clear image after deblurring.

[0113] In this embodiment, based on the result of step 3-2), the alternating iteration method is used to solve equation (10), and the iterative expression is obtained as follows:

[0114]

[0115]

[0116]

[0117] In the 0th iteration, the motion-blurred image g and the initial blur kernel are used. As input, the error term solution u of the 0th iteration. (0) and the high-frequency component solution z of the 0th iteration (0) All values ​​are set to 0, and the initial sharp image solution f is obtained by using the inverse solution of the discrete Fourier transform (13). (0) In one specific embodiment of the present invention, the coefficient λ is 0.005 in the 0th iteration and 0.5 in the nth (n≥1) iteration.

[0118] In the nth (n≥1) iteration, the clear image solution f is obtained from the (n-1)th iteration. (n-1) As input, the high-frequency component prior of the image is learned using the pre-defined denoising convolutional neural network (Dn-CNN) in the nth iteration, and the high-frequency component solution z in the nth iteration is obtained in equation (11). (n) ; with f (n-1) As input, the prior knowledge related to the blur kernel error is learned using the dual-path U network (DP-Unet) preset during the nth iteration, and the error term solution u(n) of the nth iteration in equation (12) is obtained; with the motion-blurred image g and the initial blur kernel as input, the error term solution u(n) of the nth iteration is obtained. z (n) Using u(n) as input, the clear image solution f of the nth iteration is obtained by inversely solving equation (13) using the discrete Fourier transform. (n) .

[0119] When the number of iterations n reaches the preset total number of iterations N (N≥1), the clear image solution f is obtained. (n) This is the final clear image. In one specific embodiment of the present invention, N is 3, so the final deblurred image is the clear image f obtained in the third iteration. 3 .

[0120] Furthermore, in a specific embodiment of the present invention, a deblurring neural network is designed according to the setting of the total number of iteration steps. The number of layers of the deblurring neural network is the same as the total number of iteration steps. Each layer of the deblurring neural network includes a denoising convolutional neural network and a dual-channel U-shaped network to correspond to the iterative solution of each step.

[0121] In one specific embodiment of the present invention, the deblurring neural network has three layers, comprising three denoising convolutional neural networks and three dual-channel U-shaped networks.

[0122] The training method for the deblurring neural network is as follows:

[0123] 1) Construct a training set, wherein each training sample in the training set contains a pair of sharp images and motion-blurred images, and a corresponding blur kernel with error; the specific steps are as follows:

[0124] 1-1) Acquiring clear images;

[0125] In one embodiment of the present invention, a dedicated dataset was constructed based on real images from the BSDS500 dataset. For each image in the BSDS500 training set, it was cropped into 256×256 blocks. In the direction with a pixel value of 321, the first cropping incremented by 1 pixel, followed by increments of 32 pixels each time, resulting in a total of 4 blocks. In the direction with a pixel value of 481, the first two cropping increments were 1 pixel, the third cropping incremented by 2 pixels, and then increments of 4 pixels each time, resulting in a total of 60 blocks. Therefore, a total of 200×4×60=48000 clear images were obtained.

[0126] 1-2) Inertial sensor data generation;

[0127] In one embodiment of the invention, the angular velocity and acceleration of each axis are modeled as a Gaussian distribution with a mean of 0. The standard deviation of the angular velocity along the x and y axes is σ. ωx =σ ωy =1×10 -6 rad / s, the standard deviation of the z-axis angular velocity is σ ωz =0.1 rad / s; the standard deviation of acceleration along the x and y axes is σ ax =σ ay =1×10 -3 m 2 / s, the standard deviation of z-axis acceleration is σ az =1×10 -5 m 2 / s. After randomly determining the exposure time within the range of (0.02, 0.2) seconds, inertial sensor data is sampled within the exposure time at a sampling frequency of f_s = 200Hz. To simulate continuous motion, each inertial sensor data sample is interpolated from the previous data point, where angular velocity data is linearly interpolated and acceleration data is approximately interpolated.

[0128] 1-3) Generation of motion-blurred images;

[0129] In one embodiment of the present invention, after constructing a set of inertial sensor data within the exposure time for each clear image, a blur kernel is calculated based on the generated inertial sensor data through step 2) of the method described in the embodiment of the present invention. Then, each clear image and the corresponding blur kernel are convolved to obtain the corresponding motion-blurred image.

[0130] 1-4) Generation of fuzzy kernels with errors;

[0131] In one embodiment of the present invention, the Gaussian distribution N(0.03, 0.01) is used. 2 Randomly select synchronization delay t in ) dAfter adding a synchronization delay to the exposure time of each set of inertial sensor data, the inertial sensor data is sampled within the exposure time at a sampling frequency of f_s = 200Hz. After sampling, additive white noise with a standard deviation of 1 / 10 of the corresponding data standard deviation is added to the inertial sensor data, finally obtaining inertial sensor data with error. Using the inertial sensor data with error, an error-injected blur kernel can be generated through step 2) of the method described in this embodiment of the invention.

[0132] Through the above steps, the training set for training the deblurring neural network is finally obtained, which includes 48,000 pairs of clear images and motion-blurred images, as well as corresponding blur kernels with errors.

[0133] 2) Construct a deblurring neural network;

[0134] In this embodiment, the total number of network layers is determined by the number of iteration steps. Each layer contains a denoising neural network and a dual-channel U-shaped network. The outputs of the previous denoising neural network and the dual-channel U-shaped network are used to obtain a clear image solution by inverse discrete Fourier transform (13), and are used as the input of the next layer network for the next iteration.

[0135] 3) Use the training set obtained in step 1) to train the deblurring neural network constructed in step 2).

[0136] In this embodiment, the overall loss function for training the deblurring neural network is:

[0137]

[0138] Among them, f j The j-th training sample is the clear image, J is the number of training samples, N is the number of iteration steps (3 in a specific embodiment of the present invention), and μ i The coefficient for the preset i-th iteration is between 0 and 1 (in a specific embodiment of the present invention, it is always 0.8). This is the sharp image solution corresponding to the j-th training sample image output in the i-th iteration. In this embodiment, the learning rate, batch size, and number of training rounds are set to 1×102. -3 , 4 and 100.

[0139] To implement the above embodiments, a second aspect of the present invention provides a deblurring device for motion-blurred images, comprising:

[0140] An inertial sensor data acquisition module is used to acquire inertial sensor data that records camera motion information during the exposure time of a motion-blurred image.

[0141] An initial fuzzy kernel estimation module is used to estimate the initial fuzzy kernel using the inertial sensor data;

[0142] The deblurring module is used to iteratively solve the motion-blurred image by establishing an image blurring model with blur kernel error using the initial blur kernel, and obtain the clear image after deblurring.

[0143] It should be noted that the foregoing explanation of an embodiment of a motion-blurred image deblurring method also applies to a motion-blurred image deblurring device of this embodiment, and will not be repeated here. According to an embodiment of the present invention, a motion-blurred image deblurring device acquires inertial sensor data recording camera motion information during the exposure time of the motion-blurred image; estimates an initial blur kernel using the inertial sensor data; and iteratively solves the motion-blurred image using the initial blur kernel by establishing an image blur model with blur kernel error, thereby obtaining a deblurred clear image. This allows the inertial sensor data error to be converted into a blur kernel error. In the iterative optimization solution, a neural network is used to learn the kernel error-related priors, greatly improving the robustness of the deblurring method to sensor errors and significantly improving the deblurring effect of motion-blurred images.

[0144] To implement the above embodiments, a third aspect of the present invention provides an electronic device, comprising:

[0145] At least one processor; and a memory communicatively connected to said at least one processor;

[0146] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to perform the aforementioned method for deblurring a motion-blurred image.

[0147] To implement the above embodiments, a fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described method for deblurring motion-blurred images.

[0148] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0149] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform a motion-blurred image deblurring method according to the above embodiments.

[0150] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0152] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0153] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0154] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0155] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0156] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0158] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for deblurring motion-blurred images, characterized in that, include: Acquire inertial sensor data that records camera motion information during the exposure time of a motion-blurred image; The initial ambiguity kernel is estimated using the inertial sensor data; Using the initial blur kernel, an image blur model with blur kernel error is established, and the motion-blurred image is iteratively solved to obtain the clear image after deblurring. The inertial sensor data includes: the camera's three-axis acceleration and three-axis angular velocity; The step of estimating the initial ambiguity kernel using the inertial sensor data includes: 1) Establish the geometric blur model for the motion-blurred image, with the following expression: （1） In the formula, For a clear image, For motion-blurred images, This indicates the position of a pixel in homogeneous coordinates. This represents the homography matrix at time t after the start of exposure. The coordinates in the motion-blurred image are The corresponding pixel value at that location, The coordinates in the clear image are The corresponding pixel value at that location, Indicates the noise term. This indicates the number of camera poses during the exposure time; Mapping the initial projection at the initial exposure time (t=0) to any other time t, the homography matrix... Represented as: （2） In the formula, It is the rotation transformation matrix at time t after the start of exposure. It is the translation vector at time t after the start of exposure. It is a unit vector orthogonal to the imaging plane, and d is the distance between the object being photographed and the camera; The camera's eigenvalue matrix is ​​expressed as follows: （3） In the formula, F represents the focal length, ( , () represents the coordinates of the camera's optical center. and These are the x-axis and y-axis coordinates of the optical center, respectively. 2) Calculate the rotation transformation matrix and translation vector using inertial sensor data; The expression for calculating the rotation transformation matrix is ​​as follows: （4） （5） In the formula, It is the triaxial angular velocity value at time t after the start of the exposure. These are the angular velocities of the x, y, and z axes at time t, respectively. It is the sampling time interval; The expression for calculating the translation vector is: （6） （7） In the formula, These are the triaxial acceleration values ​​at time t after the start of the exposure. These are the acceleration values ​​along the x, y, and z axes at time t, respectively. 3) Calculate the initial fuzzy kernel based on the geometric fuzzy model; Based on the results of step 2), the homography matrix for all sampling times is calculated using equation (2). Then, the initial blur kernel is obtained based on the projection trajectory of the center pixel.

2. The method according to claim 1, characterized in that, The image fuzzing model expression with fuzzy kernel error is as follows: （8） In the formula, As the initial fuzzy kernel, This is the fuzzy kernel error.

3. The method according to claim 2, characterized in that, The method further includes: Let the auxiliary variable u represent the error term. An optimization problem for image deblurring based on total least squares is established, expressed as follows: （9） In the formula, Represents the regularization term regarding image priors; These are the Lagrange coefficients, with values ​​between 0 and 1; Let the high-frequency components of a motion-blurred image be . Applying semi-quadratic splitting, the optimization problem shown in equation (9) is updated as follows: （10） In the formula, The regularization term represents the prior related to the fuzzy kernel error. Indicates a high-pass filter. Represents high-frequency components The regularization term of the relevant priors.

4. The method according to claim 3, characterized in that, The method further includes: Solving equation (10) using the alternating iterative method yields the following iterative expression: (11) (12) (13) During iteration, the motion-blurred image is used. and initial fuzzy kernel For input, the solution of the error term in step 0. and the high-frequency component solution of the 0th iteration All values ​​are set to 0, and the initial sharp image solution is obtained by using the inverse solution of the discrete Fourier transform (13). ; In the nth iteration, where n≥1, the clear image solution is obtained from the (n-1)th iteration. As input, the high-frequency component correlation prior of the image is learned using the denoising convolutional neural network preset in the nth iteration, and the high-frequency component solution of the nth iteration in equation (11) is obtained. ;by As input, the fuzzy kernel error related prior is learned using the dual-channel U-shaped network preset during the nth iteration, and the error term solution of the nth iteration in equation (12) is obtained. ; with motion-blurred images Initial fuzzy kernel and Using the discrete Fourier transform as input, the clear image solution of the nth iteration is obtained by solving equation (13). ; When the number of iterations n reaches the preset total number of iterations N, a clear image solution is obtained. This is the final, clear image.

5. The method according to claim 4, characterized in that, Before employing the alternating iterative method to solve equation (10), the method further includes: A deblurring neural network is constructed based on the denoising convolutional neural network and the dual-channel U-shaped network; The number of layers in the deblurring neural network is the same as the total number of iterations. Each layer of the deblurring neural network contains one denoising convolutional neural network and one dual-channel U-shaped network to correspond to the iterative solution of each step. The output of the previous layer's denoising neural network and dual-channel U-shaped network is used as the input of the next layer's denoising neural network and dual-channel U-shaped network to perform the next iteration by solving equation (13) through discrete Fourier transform.

6. The method according to claim 5, characterized in that, The method further includes: Training the deblurring neural network includes: 1) Construct a training set; each training sample in the training set contains a pair of sharp images and motion-blurred images, and a corresponding blur kernel with error; 2) Construct the deblurring neural network; 3) Train the deblurring neural network using the training set to obtain the trained deblurring neural network; The overall loss function for training the deblurring neural network is: in, For a clear image of the j-th training sample, N is the number of training samples, and N is the number of iterations. The coefficient for the preset i-th iteration is between 0 and 1. This is the sharp image solution corresponding to the j-th training sample image output in the i-th iteration.

7. The method according to claim 6, characterized in that, The method further includes: The motion-blurred image in the training samples is obtained by convolving the clear image with the corresponding blur kernel, and the blur kernel is calculated using simulated inertial sensor data; The fuzzy kernel with errors in the training samples is obtained by adding a set synchronization delay and preset white noise to the simulated inertial sensor data.

8. A deblurring device for motion-blurred images, characterized in that, include: An inertial sensor data acquisition module is used to acquire inertial sensor data that records camera motion information during the exposure time of a motion-blurred image. An initial fuzzy kernel estimation module is used to estimate the initial fuzzy kernel using the inertial sensor data; The deblurring module is used to iteratively solve the motion-blurred image by establishing an image blurring model with blurring kernel error using the initial blur kernel, so as to obtain a clear image after deblurring. The inertial sensor data includes: the camera's three-axis acceleration and three-axis angular velocity; The step of estimating the initial ambiguity kernel using the inertial sensor data includes: 1) Establish the geometric blur model for the motion-blurred image, with the following expression: （1） In the formula, For a clear image, For motion-blurred images, This indicates the position of a pixel in homogeneous coordinates. This represents the homography matrix at time t after the start of exposure. The coordinates in the motion-blurred image are The corresponding pixel value at that location, The coordinates in the clear image are The corresponding pixel value at that location, Indicates the noise term. This indicates the number of camera poses during the exposure time; Mapping the initial projection at the initial exposure time (t=0) to any other time t, the homography matrix... Represented as: （2） In the formula, It is the rotation transformation matrix at time t after the start of exposure. It is the translation vector at time t after the start of exposure. It is a unit vector orthogonal to the imaging plane, and d is the distance between the object being photographed and the camera; The camera's eigenvalue matrix is ​​expressed as follows: （3） In the formula, F represents the focal length, ( , () represents the coordinates of the camera's optical center. and These are the x-axis and y-axis coordinates of the optical center, respectively. 2) Calculate the rotation transformation matrix and translation vector using inertial sensor data; The expression for calculating the rotation transformation matrix is ​​as follows: （4） （5） In the formula, It is the triaxial angular velocity value at time t after the start of the exposure. These are the angular velocities of the x, y, and z axes at time t, respectively. It is the sampling time interval; The expression for calculating the translation vector is: （6） （7） In the formula, These are the triaxial acceleration values ​​at time t after the start of the exposure. These are the acceleration values ​​along the x, y, and z axes at time t, respectively. 3) Calculate the initial fuzzy kernel based on the geometric fuzzy model; Based on the results of step 2), the homography matrix for all sampling times is calculated using equation (2). Then, the initial blur kernel is obtained based on the projection trajectory of the center pixel.

Images