Method for removing image rotation motion blur, electronic equipment and storage medium
By training an image deblurring model and using a fast Fourier transform and denoising fill module to process moving images, the problem of low image deblurring efficiency in existing technologies is solved, and more efficient and accurate image deblurring processing is achieved.
Patent Information
- Application Number
- CN202411044223.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-29
AI Technical Summary
Existing image deblurring schemes have high computational complexity, limited generalization ability, and are unable to extract local optima, resulting in low efficiency and insufficient accuracy in image deblurring processing.
By acquiring image samples, an image deblurring model is trained. Fast Fourier Transform, denoising, and filling modules are used to process moving images. End-to-end constraint training is performed using the focus space loss function to improve the efficiency and accuracy of the image deblurring model.
It reduces the complexity of image deblurring in terms of environment, improves processing flexibility and speed, avoids wasting time, and enhances the accuracy and efficiency of image deblurring.
Smart Images

Figure CN119048398A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to a method, electronic device and storage medium for removing image rotational motion blur. Background Art
[0002] With the development of modern science and technology, object detection has made a major breakthrough and has been applied to the field of intelligent electronic referees in various events. Electronic referees use cameras to capture images of athletes and sports equipment, analyze the images, and then make judgments on whether the athletes have violated the rules. Because athletes move too fast, the images taken will have rotational motion blur. In order to remove the image blur caused by rotational motion in moving images, removing image rotational motion blur has become a promising direction.
[0003] In the prior art, there are two main methods for image deblurring. One is to completely solve the problem of rotational motion blur through non-blind deblurring methods, such as coordinate transformation recovery (CTR) and blur path method to eliminate rotational motion blur; the other is to introduce two-dimensional information in the image to remove image rotational motion blur, such as end-to-end rotational motion deblurring based on generative adversarial networks and a progressive framework for rotational motion deburring.
[0004] In the prior art, the computational complexity of the image deblurring scheme is high. At the same time, it is restricted by the application environment, has limited generalization ability, and is difficult to extract features. It is impossible to extract local optimal solutions at each stage of the image deblurring scheme, and thus it is impossible to obtain the global optimal solution of the overall scheme, which reduces the accuracy of the image deblurring processing and there is a technical problem of low efficiency of the image deblurring processing. Summary of the invention
[0005] The present application provides a method, an electronic device and a storage medium for removing rotational motion blur of an image, so as to achieve the technical effect of improving the efficiency of image deblurring processing.
[0006] In a first aspect, the present application provides a method for removing rotational motion blur of an image, comprising:
[0007] Acquire an image sample, the image sample comprising a plurality of motion image pairs, each motion image pair comprising a rotational motion blurred image and a corresponding clear motion image;
[0008] According to each motion image pair, a plurality of basic model parameters are obtained, wherein the basic model parameters include a blur kernel of the rotational motion blurred image and a corresponding blur kernel radius;
[0009] The image deblurring model is trained according to the basic parameters of the model to obtain the trained image deblurring model, wherein the image deblurring model is used to remove image blur caused by rotational motion in the rotational motion blurred image;
[0010] A target motion image is obtained, and the target motion image is input into an image deblurring model to obtain a clear image output by the image deblurring model.
[0011] Optionally, an image deblurring model, comprising:
[0012] A fast Fourier transform module is used to perform image frequency domain conversion on the moving image pair;
[0013] A fast Fourier transform inversion module is used to perform image spatial domain conversion on the motion image pairs that have undergone frequency domain conversion;
[0014] A denoising module for removing image noise from a moving image pair;
[0015] A filling module is used to fill missing pixels in moving image pairs.
[0016] Optionally, the fast Fourier transform module is obtained by:
[0017] Convolution modeling is performed based on the rotational motion blurred image, blur kernel, clear motion image and blur kernel radius to obtain a one-dimensional blur sequence and an ideal sequence. The process expression of convolution modeling is as follows:
[0018]
[0019] in, Subscript i is the i-th element of the fuzzy sequence, g r 、h r 、f r represent one-dimensional fuzzy sequence, fuzzy kernel sequence and ideal sequence respectively, N r Yes r Length;
[0020] According to the one-dimensional fuzzy sequence and the ideal sequence, the original image sequence of the moving image pair is determined, where the expression of the original image sequence is as follows:
[0021]
[0022] in, is the original image sequence, λ is the regularization parameter, Φ(f r ) represents image prior;
[0023] Iterate the original image sequence to obtain the corresponding clear sequence. The expression of the clear sequence is as follows:
[0024]
[0025] in, is a clear sequence, T is the series, μ represents the auxiliary parameter, and z represents the auxiliary variable;
[0026] According to the clear sequence, a fast Fourier transform module is obtained, wherein the expression of the fast Fourier transform module is as follows:
[0027]
[0028] Among them, Hr* is the conjugate operator, Represents the element-wise multiplication operator, F r =o(f r ), H r =o(h r ), o() is the fast Fourier transform operator.
[0029] Optionally, the fast Fourier transform inversion module is obtained by:
[0030] The fast Fourier transform module is inverted to obtain the fast Fourier transform inversion module. The inversion process of the fast Fourier transform module is expressed as follows:
[0031]
[0032] Among them, F=FTF(f), H=FTF(h), G=FTF(g) and Z=FTF(z).
[0033] Optionally, the denoising module is obtained by:
[0034] According to the original image sequence, a plurality of target scales including denoising residual blocks are obtained, wherein each target scale includes a plurality of denoising residual blocks;
[0035] According to the target scale, the denoising module is obtained. The expression of the denoising module in the spatial domain is as follows:
[0036]
[0037] Among them, zT+1 is the expression of the denoising module in the spatial domain, z is the two-dimensional spatial domain representation of the auxiliary variable zr, is the denoising level, f = IFTF(F);
[0038] The expression of the denoising module in the frequency domain is as follows:
[0039]
[0040] Among them, Z is the auxiliary variable Z r Two-dimensional frequency domain representation of .
[0041] Optionally, the fill module is obtained by:
[0042] The denoising residual block in the denoising module is replaced with a padding residual block, and the denoising module including the padding residual block is used as a padding module.
[0043] Optionally, training the image deblurring model according to the basic model parameters to obtain the trained image deblurring model includes:
[0044] According to the basic parameters of the model, the motion image pair is input into the image deblurring model to obtain a restored image of the motion image pair output by the image deblurring model;
[0045] The matrix coordinates of the clear motion image and the restored image are obtained respectively to determine the corresponding focal space loss function. The expression of the focal space loss function is as follows:
[0046]
[0047] Where s is the restored image, f is the clear motion image, M and N are the width and height of f, s(x, y) is the matrix coordinate of the restored image, f(x, y) is the matrix coordinate of the clear motion image, and the weight matrix w(x, y) is defined as:
[0048] ω(x,y)=|s(x,y)-f(x,y)|
[0049] According to the focal space loss function, the image deblurring model is trained end-to-end with constraints to obtain a trained image deblurring model.
[0050] In a second aspect, the present application provides a device for removing image rotational motion blur, comprising:
[0051] A first acquisition module, used for acquiring image samples, the image samples comprising a plurality of motion image pairs, each motion image pair comprising a rotation motion blurred image and a corresponding clear motion image;
[0052] A second acquisition module is used to acquire a plurality of basic model parameters according to each motion image pair, wherein the basic model parameters include a rotation motion blurred image, a blur kernel, a clear motion image and a blur kernel radius;
[0053] A processing module, used for training an image deblurring model according to basic model parameters, and obtaining a trained image deblurring model, wherein the image deblurring model is used for removing image blur caused by rotational motion in a rotational motion blurred image;
[0054] The second processing module is used to obtain a target motion image, and input the target motion image into an image deblurring model to obtain a clear image output by the image deblurring model.
[0055] Optionally, the first processing module is further used for:
[0056] A fast Fourier transform module is used to perform image frequency domain conversion on the moving image pair;
[0057] A fast Fourier transform inversion module is used to perform image spatial domain conversion on the motion image pairs that have undergone frequency domain conversion;
[0058] A denoising module for removing image noise from a moving image pair;
[0059] A filling module is used to fill missing pixels in moving image pairs.
[0060] Optionally, the first processing module is further used for:
[0061] Convolution modeling is performed based on the rotational motion blurred image, blur kernel, clear motion image and blur kernel radius to obtain a one-dimensional blur sequence and an ideal sequence. The process expression of convolution modeling is as follows:
[0062]
[0063] in, Subscript i is the i-th element of the fuzzy sequence, g r 、h r 、f r represent one-dimensional fuzzy sequence, fuzzy kernel sequence and ideal sequence respectively, N r Yes r Length;
[0064] According to the one-dimensional fuzzy sequence and the ideal sequence, the original image sequence of the moving image pair is determined, where the expression of the original image sequence is as follows:
[0065]
[0066] in, is the original image sequence, λ is the regularization parameter, Φ(f r ) represents image prior;
[0067] Iterate the original image sequence to obtain the corresponding clear sequence. The expression of the clear sequence is as follows:
[0068]
[0069] in, is a clear sequence, T is the series, μ represents the auxiliary parameter, and z represents the auxiliary variable;
[0070] According to the clear sequence, a fast Fourier transform module is obtained, wherein the expression of the fast Fourier transform module is as follows:
[0071]
[0072] Among them, Hr* is the conjugate operator, Represents the element-wise multiplication operator, F r =o(f r ), H r =o(h r ), o() is the fast Fourier transform operator.
[0073] Optionally, the first processing module is further used for:
[0074] The fast Fourier transform module is inverted to obtain the fast Fourier transform inversion module. The inversion process of the fast Fourier transform module is expressed as follows:
[0075]
[0076] Among them, F=FTF(f), H=FTF(h), G=FTF(g) and Z=FTF(z).
[0077] Optionally, the first processing module is further used for:
[0078] According to the original image sequence, a plurality of target scales including denoising residual blocks are obtained, wherein each target scale includes a plurality of denoising residual blocks;
[0079] According to the target scale, the denoising module is obtained. The expression of the denoising module in the spatial domain is as follows:
[0080]
[0081] Among them, z T+1 is the expression of the denoising module in the spatial domain, and z is the auxiliary variable z r The two-dimensional space domain representation is: is the denoising level, f = IFTF(F);
[0082] The expression of the denoising module in the frequency domain is as follows:
[0083]
[0084] Among them, Z is the auxiliary variable Z r Two-dimensional frequency domain representation of .
[0085] Optionally, the first processing module is further used for:
[0086] The denoising residual block in the denoising module is replaced with a padding residual block, and the denoising module including the padding residual block is used as a padding module.
[0087] Optionally, the first processing module is further used for:
[0088] According to the basic parameters of the model, the motion image pair is input into the image deblurring model to obtain a restored image of the motion image pair output by the image deblurring model;
[0089] The matrix coordinates of the clear motion image and the restored image are obtained respectively to determine the corresponding focal space loss function. The expression of the focal space loss function is as follows:
[0090]
[0091] Where s is the restored image, f is the clear motion image, M and N are the width and height of f, s(x, y) is the matrix coordinate of the restored image, f(x, y) is the matrix coordinate of the clear motion image, and the weight matrix w(x, y) is defined as:
[0092] ω(x,y)=|s(x,y)-f(x,y)|
[0093] According to the focal space loss function, the image deblurring model is trained end-to-end with constraints to obtain a trained image deblurring model.
[0094] In a third aspect, the present application provides a device for removing image rotational motion blur, comprising:
[0095] Processor and memory;
[0096] Memory stores computer-executable instructions;
[0097] The processor executes the computer-executable instructions stored in the memory, so that the processor executes various possible implementations of the first aspect as described above.
[0098] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement various possible implementation methods as in the first aspect.
[0099] In a fifth aspect, the present application provides a computer program product, which, when executed by a processor, is used to implement various possible implementations as in the first aspect.
[0100] The present application provides a method, electronic device and storage medium for removing rotational motion blur of an image. The method comprises obtaining an image sample, wherein the image sample comprises a plurality of motion image pairs, each motion image pair comprises a rotational motion blurred image and a corresponding clear motion image; obtaining a plurality of basic model parameters according to each motion image pair, wherein the basic model parameters comprise a blur kernel of the rotational motion blurred image and a corresponding blur kernel radius; training an image deblurring model according to the basic model parameters to obtain a trained image deblurring model; obtaining a target motion image, and inputting the target motion image into the image deblurring model to obtain a clear image output by the image deblurring model, thereby performing image deblurring processing on the overall motion image pair and the basic model parameters, reducing the restriction of the environment on the image, and improving the accuracy of the image deblurring processing. At the same time, through the image deblurring model, the complexity of the overall process calculation is reduced, the flexibility, rapidity and convenience of the overall processing process are improved, and the waste of time cost is avoided, thereby solving the technical problem of low efficiency in removing rotational motion blur of images and achieving the technical effect of improving the efficiency in removing rotational motion blur of images. BRIEF DESCRIPTION OF THE DRAWINGS
[0101] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0102] Figure 1 The process of the method for removing image rotational motion blur provided in the embodiment of the present application Figure 1 ;
[0103] Figure 2 The process of the method for removing image rotational motion blur provided in the embodiment of the present application Figure 2 ;
[0104] Figure 3 A schematic diagram of the structure of the CV channel attention provided in an embodiment of the present application;
[0105] Figure 4 A schematic diagram of the structure of a denoising module in the method of the present invention provided in an embodiment of the present application;
[0106] Figure 5 A schematic diagram of the structure of a filling module in the method of the present invention provided in an embodiment of the present application;
[0107] Figure 6 A schematic diagram of the structure of a device for removing image rotational motion blur provided in an embodiment of the present application;
[0108] Figure 7 A hardware structure diagram of a device for removing image rotational motion blur provided in an embodiment of the present application.
[0109] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0110] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0111] In the prior art, the computational complexity of the image deblurring scheme is high. At the same time, it is restricted by the application environment, has limited generalization ability, and is difficult to extract features. It is impossible to extract local optimal solutions at each stage of the image deblurring scheme, and thus it is impossible to obtain the global optimal solution of the overall scheme, which reduces the accuracy of the image deblurring processing and there is a technical problem of low efficiency of the image deblurring processing.
[0112] The present application provides a method, electronic device and storage medium for removing image rotational motion blur. By performing image deblurring processing on the overall motion image pair and the basic parameters of the model, the restrictions of the environment on the image are reduced and the accuracy of the image deblurring processing is improved. At the same time, through the image deblurring model, the complexity of the overall process calculation is reduced, the flexibility, speed and convenience of the overall processing process are improved, the waste of time cost is avoided, the technical problem of low efficiency in removing image rotational motion blur is solved, and the technical effect of improving the efficiency of removing image rotational motion blur is achieved.
[0113] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0114] Figure 1 The process of the method for removing image rotational motion blur provided in the embodiment of the present application Figure 1 .like Figure 1 As shown, the method for removing image rotational motion blur provided by the embodiment of the present application includes:
[0115] S101, acquiring an image sample, where the image sample includes a plurality of motion image pairs, each motion image pair including a rotational motion blurred image and a corresponding clear motion image;
[0116] In this embodiment, each motion image pair includes at least one rotational motion blurred image and a corresponding clear motion image.
[0117] An image sample that needs to be deblurred is obtained, wherein the obtained image sample includes a plurality of motion image pairs consisting of a rotational motion blurred image and a corresponding clear motion image.
[0118] S102, obtaining a plurality of basic model parameters according to each pair of motion images;
[0119] In this embodiment, the basic parameters of the model include the blur kernel of the rotational motion blurred image and the corresponding blur kernel radius, wherein the blur kernel of the rotational motion blurred image refers to the mathematical representation of the motion blurred image, which mathematical representation can be a two-dimensional array or matrix, and the blur kernel radius refers to the distance radius from the rotation center to the edge of the blurred area in the motion blurred image.
[0120] According to each motion image pair, a blur kernel parameter and a corresponding blur kernel radius parameter in the motion blurred image are obtained, and the blur kernel parameter and the corresponding blur kernel radius parameter are used as basic parameters of the model.
[0121] S103, training the image deblurring model according to the basic parameters of the model to obtain the trained image deblurring model;
[0122] In this embodiment, the image deblurring model is used to remove image blur caused by rotational motion in a moving image pair.
[0123] An image deblurring model is established according to the obtained basic model parameters. The model is composed of four modules and the model is trained to obtain the trained image deblurring model.
[0124] S104: Acquire a target motion image, and input the target motion image into an image deblurring model to acquire a clear image output by the image deblurring model.
[0125] In this embodiment, the target motion image is used to indicate a motion image from which motion rotation blur needs to be removed.
[0126] A target motion image that needs to remove motion rotation blur is obtained, and the target motion image is input into a trained image deblurring model so that the image deblurring model deblurs the target motion image, thereby obtaining a clear image output by the image deblurring model.
[0127] The present application provides a method for removing rotational motion blur of an image, which comprises obtaining an image sample, wherein the image sample comprises a plurality of motion image pairs, each motion image pair comprises a rotational motion blurred image and a corresponding clear motion image; according to each motion image pair, a plurality of basic model parameters are obtained, wherein the basic model parameters comprise a blur kernel of the rotational motion blurred image and a corresponding blur kernel radius; an image deblurring model is trained according to the basic model parameters to obtain a trained image deblurring model; a target motion image is obtained, and the target motion image is input into the image deblurring model to obtain a clear image output by the image deblurring model, thereby performing image deblurring processing on the overall motion image pair and the basic model parameters, thereby reducing the restriction of the environment on the image and improving the accuracy of the image deblurring processing; at the same time, through the image deblurring model, the complexity of the overall process calculation is reduced, the flexibility, rapidity and convenience of the overall processing process are improved, the waste of time cost is avoided, the technical problem of low efficiency in removing rotational motion blur of images is solved, and the technical effect of improving the efficiency in removing rotational motion blur of images is achieved.
[0128] Figure 2 The process of the method for removing image rotational motion blur provided in the embodiment of the present application Figure 2 .like Figure 2 As shown, based on the above embodiment, this embodiment provides a supplementary description of the image rotation motion blur removal and training process, including:
[0129] S201, performing convolution modeling according to the rotational motion blurred image, blur kernel, clear motion image and blur kernel radius to obtain a one-dimensional blur sequence and an ideal sequence; determining the original image sequence of the motion image pair according to the one-dimensional blur sequence and the ideal sequence;
[0130] The rotational motion blurred image is represented as g, the blur kernel is represented as h, the clear motion image is represented as f, and the blur kernel radius is represented as r. The rotational motion blur of the image sample along the blur kernel radius r is modeled through the convolution process, and its expression is as follows:
[0131]
[0132] in, Subscript i is the i-th element of the fuzzy sequence, g r 、h r 、f r represent one-dimensional fuzzy sequence, fuzzy kernel sequence and ideal sequence respectively, N r Yes r Length.
[0133] Convert the above expression into matrix form:
[0134] g r =Cr f r +n
[0135] Among them, g r and f r Is a N r A column vector of elements. r Represents an N r ×N r Square circular matrix. n represents additive white Gaussian noise.
[0136] According to the one-dimensional fuzzy sequence and the ideal sequence, the original image sequence of the motion image pair is obtained. The expression of the original image sequence is as follows:
[0137]
[0138] in, is the original image sequence, λ is the regularization parameter, Φ(f r ) represents the image prior.
[0139] S202, iterating the original image sequence to obtain a corresponding clear sequence; obtaining a fast Fourier transform module according to the clear sequence, the fast Fourier transform module being used to perform image frequency domain conversion on the motion image pair;
[0140] The semi-quadratic splitting algorithm is used to solve the original image sequence. It can be solved by the following iterative scheme to obtain a clear sequence, which is expressed as follows:
[0141]
[0142] in, is a clear sequence, T is the series, μ represents the auxiliary parameter, and z represents the auxiliary variable.
[0143] According to the clear sequence, a fast Fourier transform module is obtained, wherein the expression of the fast Fourier transform module is as follows:
[0144]
[0145] Among them, Hr* is the conjugate operator, Represents the element-wise multiplication operator, F r =o(f r ), H r =o(h r ), o() is the fast Fourier transform operator.
[0146] By establishing a fast Fourier transform module, the images in the moving image pair are converted from the spatial domain to the frequency domain, making it easier to identify and process the blur caused by rotational motion in the frequency domain.
[0147] S203, inverting the fast Fourier transform module to obtain a fast Fourier transform inversion module, where the fast Fourier transform inversion module is used to perform image spatial domain conversion on the motion image pair that has undergone frequency domain conversion;
[0148] The fast Fourier transform module is inverted to obtain the fast Fourier transform inversion module. The inversion process of the fast Fourier transform module is expressed as follows:
[0149]
[0150] Among them, F=FTF(f), H=FTF(h), G=FTF(g) and Z=FTF(z).
[0151] It should be noted that in the above expression, a CV channel attention module is needed to learn μ and choose the appropriate μ T ;
[0152] F T+1 The denoising level of Z depends on μ, and the noise is used to predict μ in turn; T-1 -F T-1 represents the noise removed in stage T-1.
[0153] Application of Figure 3 The CV channel attention module shown in Figure 2 is used to obtain the attention weight of the next stage, i.e., μ T , with Z T-1 -F T-1 is the input, and then the input is passed to the complex-valued fully connected layer (C-FC), complex-valued ReLu (C-ReLu) and C-FC in turn. In order to obtain the real-valued μ T , calculate the modulus of the feature, and then pass it to the Sigmoid layer to finally get μ T .
[0154] For CV input X and CV weight W, C-FC and C-ReLu are respectively expressed as follows:
[0155]
[0156] in, and S are the real and imaginary parts of the complex value, respectively. j represents the imaginary unit.
[0157] By building a fast Fourier transform inversion module, after the motion image pair is deblurred in the frequency domain, the processed image is converted from the frequency domain back to the spatial domain.
[0158] S204, obtaining a plurality of target scales including denoising residual blocks according to the original image sequence, and obtaining a denoising module according to the target scales, the denoising module being used to remove image noise of the motion image pair;
[0159] The original image sequence expression obtained in S201 above can be regarded as the denoising process of the image sample, that is, the expression of the denoising module is as follows:
[0160]
[0161] The expression of the denoising module in the spatial domain is as follows:
[0162]
[0163] Among them, z T+1 is the expression of the denoising module in the spatial domain, and z is the auxiliary variable z r The two-dimensional space domain representation is: is the denoising level, f = IFTF(F);
[0164] The expression of the denoising module in the frequency domain is as follows:
[0165]
[0166] Among them, Z is the auxiliary variable Z r Two-dimensional frequency domain representation of .
[0167] Specifically, in stage T, the denoising module is transformed from [F (0) ,...F (T) ] estimates Z, and the denoising module is as follows Figure 4 As shown, it consists of target scales, each of which includes multiple denoising residual blocks (C-ResBlock).
[0168] It should be noted that at small scales, there are skip connections between the strided convolution and the upsampling convolution. Due to the small batch size, the batch normalization layer of the original residual block in the denoising residual block is removed.
[0169] Let K represent the CV kernel, and C-Conv is defined as follows:
[0170]
[0171] Among them, SConv and UConv are strided convolution and upsampling convolution respectively.
[0172] By establishing a denoising module, the delay estimation in the frequency domain is denoised, the noise in the frequency domain is suppressed, and the interference of image noise on the deblurring process is avoided.
[0173] S205, replacing the denoising residual block in the denoising module with a filling residual block, and using the denoising module including the filling residual block as a filling module, the filling module being used to fill the missing pixels in the motion image pair;
[0174] All denoising residual block layers in the denoising module in S204 are replaced with padding residual block (R-ResBlock) layers, and the replaced module is determined as follows Figure 5 Filling module shown.
[0175] Specifically, in stage T, the filling module is from [F (0) ,...F (T) ]To estimate Z, the padding module consists of multiple target scales, each of which includes multiple padding residual blocks.
[0176] It should be noted that at small scales, there are skip connections between the strided convolution and the upsampling convolution.
[0177] By establishing a filling module, the missing pixels after the image frequency domain conversion by the fast Fourier transform module and the inverse fast Fourier transform module are supplemented.
[0178] S206, establishing a corresponding image deblurring model according to the fast Fourier transform module, the fast Fourier transform inversion module, the denoising module and the filling module;
[0179] The fast Fourier transform module, the fast Fourier transform inversion module, the denoising module and the filling module are combined to obtain a combined image deblurring model.
[0180] S207, inputting the motion image pair into the image deblurring model according to the basic parameters of the model, obtaining a restored image of the motion image pair output by the image deblurring model; respectively obtaining the matrix coordinates of the clear motion image and the restored image to determine the corresponding focal space loss function;
[0181] The motion image pair that needs to be deblurred is input into the image deblurring model. The image deblurred image outputs the corresponding restored image. A spatial matrix is established based on the clear motion image and the restored image in the motion image pair, and the corresponding matrix coordinates are obtained. The focal space loss function is determined based on the matrix coordinates of the clear motion image and the restored image. Its expression is as follows:
[0182]
[0183] Among them, s is the restored image, f is the clear motion image, M and N are the width and height of f, s(x, y) is the matrix coordinate of the restored image, f(x, y) is the matrix coordinate of the clear motion image, and the weight matrix w(x,
[0184] y) is defined as:
[0185] ω(x,y)=|s(x,y)-f(x,y)|
[0186] S208. Perform end-to-end constraint training on the image deblurring model according to the focal space loss function to obtain a trained image deblurring model.
[0187] The image deblurring model is trained in an end-to-end manner so that all modules can reach the optimal solution constrained by a global loss function.
[0188] It should be noted that in order to avoid the influence of noise and outliers, it is necessary to max ], w(x,y) is clipped within the range of ];
[0189] Among them, α and ω(x,y) max is the scaling factor and maximum value of ω(x,y).
[0190] The present application provides a method for removing image rotational motion blur. The method transforms the motion blurred image and blur kernel in a motion image pair into a two-dimensional frequency domain matrix by establishing a fast Fourier transform module and a fast Fourier transform inversion module, and then denoises the delay estimate in the frequency domain through a noise module to suppress the noise in the frequency domain. At the same time, a filling module is used to supplement the missing pixels after the fast Fourier transform module and the fast Fourier transform inversion module convert the image into frequency domain. Finally, an intersection space loss function is designed to train the image deblurring model end-to-end to obtain a global optimal solution. The flexibility, speed and convenience of the overall deblurring process are improved, the influence of the environment on the overall deblurring process is reduced, and the overall calculation speed is improved while reducing the complexity of the overall process calculation, avoiding the waste of time cost in the overall deblurring process, improving the accuracy of image deblurring, solving the technical problem of low efficiency of image deblurring, and achieving the technical effect of improving the efficiency of image deblurring.
[0191] Figure 6 The schematic diagram of the structure of the device for removing image rotational motion blur provided in the embodiment of the present application. The device of the present embodiment may be in the form of software and / or hardware. Figure 6 As shown, the device 600 for removing image rotational motion blur provided in the embodiment of the present application includes: a first acquisition module 601, a second acquisition module 602, a first processing module 603, and a second processing module 604:
[0192] A first acquisition module 601 is used to acquire image samples, where the image samples include a plurality of motion image pairs, each motion image pair includes a rotation motion blurred image and a corresponding clear motion image;
[0193] A second acquisition module 602 is used to acquire a plurality of basic model parameters according to each motion image pair, wherein the basic model parameters include a rotation motion blurred image, a blur kernel, a clear motion image and a blur kernel radius;
[0194] The first processing module 603 is used to train the image deblurring model according to the basic parameters of the model, and obtain the trained image deblurring model, wherein the image deblurring model is used to remove image blur caused by rotational motion in a moving image pair;
[0195] The second processing module 604 is used to obtain a target motion image, and input the target motion image into an image deblurring model to obtain a clear image output by the image deblurring model.
[0196] In a possible implementation, the first processing module 603 is further configured to:
[0197] A fast Fourier transform module is used to perform image frequency domain conversion on the moving image pair;
[0198] A fast Fourier transform inversion module is used to perform image spatial domain conversion on the motion image pairs that have undergone frequency domain conversion;
[0199] A denoising module for removing image noise from a moving image pair;
[0200] A filling module is used to fill missing pixels in moving image pairs.
[0201] In a possible implementation, the first processing module 603 is further configured to:
[0202] Convolution modeling is performed based on the rotational motion blurred image, blur kernel, clear motion image and blur kernel radius to obtain a one-dimensional blur sequence and an ideal sequence. The process expression of convolution modeling is as follows:
[0203]
[0204] in, Subscript i is the i-th element of the fuzzy sequence, g r 、h r 、f r represent one-dimensional fuzzy sequence, fuzzy kernel sequence and ideal sequence respectively, N r Yes r Length;
[0205] According to the one-dimensional fuzzy sequence and the ideal sequence, the original image sequence of the moving image pair is determined, where the expression of the original image sequence is as follows:
[0206]
[0207] in, is the original image sequence, λ is the regularization parameter, Φ(f r ) represents image prior;
[0208] Iterate the original image sequence to obtain the corresponding clear sequence. The expression of the clear sequence is as follows:
[0209]
[0210] in, is a clear sequence, T is the series, μ represents the auxiliary parameter, and z represents the auxiliary variable;
[0211] According to the clear sequence, a fast Fourier transform module is obtained, wherein the expression of the fast Fourier transform module is as follows:
[0212]
[0213] Among them, Hr* is the conjugate operator, Represents the element-wise multiplication operator, F r =o(f r ), H r =o(h r ), o() is the fast Fourier transform operator.
[0214] In a possible implementation, the first processing module 603 is further configured to:
[0215] The fast Fourier transform module is inverted to obtain the fast Fourier transform inversion module. The inversion process of the fast Fourier transform module is expressed as follows:
[0216]
[0217] Among them, F=FTF(f), H=FTF(h), G=FTF(g) and Z=FTF(z).
[0218] In a possible implementation, the first processing module 603 is further configured to:
[0219] According to the original image sequence, a plurality of target scales including denoising residual blocks are obtained, wherein each target scale includes a plurality of denoising residual blocks;
[0220] According to the target scale, the denoising module is obtained. The expression of the denoising module in the spatial domain is as follows:
[0221]
[0222] Among them, z T+1 is the expression of the denoising module in the spatial domain, and z is the auxiliary variable z r The two-dimensional space domain representation is: is the denoising level, f = IFTF(F);
[0223] The expression of the denoising module in the frequency domain is as follows:
[0224]
[0225] Among them, Z is the auxiliary variable Z r Two-dimensional frequency domain representation of .
[0226] In a possible implementation, the first processing module 603 is further configured to:
[0227] The denoising residual block in the denoising module is replaced with a padding residual block, and the denoising module including the padding residual block is used as a padding module.
[0228] In a possible implementation, the first processing module 603 is further configured to:
[0229] According to the basic parameters of the model, the motion image pair is input into the image deblurring model to obtain a restored image of the motion image pair output by the image deblurring model;
[0230] The matrix coordinates of the clear motion image and the restored image are obtained respectively to determine the corresponding focal space loss function. The expression of the focal space loss function is as follows:
[0231]
[0232] Among them, s is the restored image, f is the clear motion image, M and N are the width and height of f, s(x, y) is the matrix coordinate of the restored image, f(x, y) is the matrix coordinate of the clear motion image, and the weight matrix w(x,
[0233] y) is defined as:
[0234] ω(x,y)=|s(x,y)-f(x,y)|
[0235] According to the focal space loss function, the image deblurring model is trained end-to-end with constraints to obtain a trained image deblurring model.
[0236] The present application provides a device for removing rotational motion blur of an image, which can implement the above method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.
[0237] Figure 7 The hardware structure diagram of the device for removing image rotational motion blur provided in the embodiment of the present application. Figure 7 As shown, the device 700 for removing image rotational motion blur comprises:
[0238] Processor 701 and memory 702;
[0239] Memory stores computer-executable instructions;
[0240] The processor executes the computer-executable instructions stored in the memory 702 , so that the device for removing image rotational motion blur performs the method for removing image rotational motion blur as described above.
[0241] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.
[0242] The memory 702 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.
[0243] An embodiment of the present application accordingly provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method for removing image rotational motion blur as described above.
[0244] The embodiment of the present application also provides a computer program product accordingly. When the computer program is executed by a processor, it is used to implement the method for removing image rotational motion blur as described above.
[0245] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0246] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0247] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0248] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0249] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.
[0250] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0251] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0252] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0253] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for removing image rotational motion blur, characterized in that: include: Acquire an image sample, wherein the image sample includes a plurality of motion image pairs, each motion image pair including a rotational motion blurred image and a corresponding clear motion image; According to each of the motion image pairs, a plurality of basic model parameters are acquired, wherein the basic model parameters include a blur kernel of the rotational motion blurred image and a corresponding blur kernel radius; The image deblurring model is trained according to the basic parameters of the model to obtain a trained image deblurring model, wherein the image deblurring model is used to remove image blur caused by rotational motion in the rotational motion blurred image.
2. The method according to claim 1, characterized in that The image deblurring model includes: A fast Fourier transform module, used for performing image frequency domain conversion on the motion image pair; A fast Fourier transform inversion module is used to perform image spatial domain conversion on the motion image pairs that have undergone frequency domain conversion; A denoising module, used to remove image noise from the moving image pair; A filling module is used to fill missing pixels in the moving image pair.
3. The method according to claim 2, characterized in that The fast Fourier transform module is obtained by: Convolution modeling is performed according to the rotational motion blurred image, the blur kernel, the clear motion image and the blur kernel radius to obtain a one-dimensional blur sequence and an ideal sequence. The process expression of the convolution modeling is as follows: in, Subscript i is the i-th element of the fuzzy sequence, g r 、h r 、f r denote the one-dimensional fuzzy sequence, fuzzy kernel sequence and the ideal sequence respectively, N r Yes r Length; According to the one-dimensional fuzzy sequence and the ideal sequence, the original image sequence of the motion image pair is determined, wherein the expression of the original image sequence is as follows: in, is the original image sequence, λ is the regularization parameter, Φ(f r ) represents image prior; The original image sequence is iterated to obtain the corresponding clear sequence, and the expression of the clear sequence is as follows: in, is the clear sequence, T is the series, μ represents the auxiliary parameter, and z represents the auxiliary variable; According to the clear sequence, the fast Fourier transform module is obtained, wherein the expression of the fast Fourier transform module is as follows: Among them, Hr* is the conjugate operator, Represents the element-wise multiplication operator, F r =o(f r ), H r =o(h r ), o() is the fast Fourier transform operator.
4. The method according to claim 3, characterized in that The fast Fourier transform inversion module is obtained by: The fast Fourier transform module is inverted to obtain the fast Fourier transform inversion module. The inversion process of the fast Fourier transform module is expressed as follows: Among them, F=FTF(f), H=FTF(h), G=FTF(g) and Z=FTF(z).
5. The method according to claim 4, characterized in that The denoising module is obtained in the following way: According to the original image sequence, a plurality of target scales including denoised residual blocks are acquired, wherein each of the target scales includes a plurality of the denoised residual blocks; According to the target scale, the denoising module is obtained, and the expression of the denoising module in the spatial domain is as follows: Among them, z T+1 is the expression of the denoising module in the spatial domain, and z is the auxiliary variable z r The two-dimensional space domain representation is: is the denoising level, f = IFTF(F); The expression of the denoising module in the frequency domain is as follows: Among them, Z is the auxiliary variable Z r Two-dimensional frequency domain representation of .
6. The method according to claim 5, characterized in that The filling module is obtained in the following manner: The denoising residual block in the denoising module is replaced with a filling residual block, and the denoising module including the filling residual block is used as the filling module.
7. The method according to claim 1, characterized in that The step of training the image deblurring model according to the basic parameters of the model to obtain the trained image deblurring model includes: According to the basic parameters of the model, the moving image pair is input into the image deblurring model to obtain a restored image of the moving image pair output by the image deblurring model; The matrix coordinates of the clear motion image and the restored image are obtained respectively to determine the corresponding focal space loss function, and the expression of the focal space loss function is as follows: Where s is the restored image, f is the clear motion image, M and N are the width and height of f, s(x, y) is the matrix coordinate of the restored image, f(x, y) is the matrix coordinate of the clear motion image, and the weight matrix w(x, y) is defined as: ω(x,y)=|s(x,y)-f(x,y) According to the focal space loss function, end-to-end constraint training is performed on the image deblurring model to obtain the trained image deblurring model.
8. A method for removing rotational motion blur of an image, characterized in that: include: A target motion image is acquired, and the target motion image is input into an image deblurring model to acquire a clear image output by the image deblurring model, wherein the image deblurring model is trained by the method according to any one of claims 1 to 7.
9. A device for removing image rotational motion blur, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.
