A method and device for super-resolution image restoration of non-uniform motion blur

Abstract

The present invention discloses a method for super-resolution image restoration of non-uniform motion blur, comprising the following steps: S1, constructing a dataset; S2, inputting the preprocessed dataset into a generator to obtain a preliminary restored image; S3, using a discriminator to discriminate between the preliminary restored image and the true image to obtain a discrimination result, wherein the discriminator is a Markov discriminant network; S4, optimizing a generative adversarial network using a loss function to obtain a network result with optimal performance, thereby obtaining an optimal restored image; the generative adversarial network includes a generator and a discriminator; S5, outputting the restoration result. This method solves the problem of irreversible restoration and unclear restoration results caused by image information loss due to a single source of image information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for restoring a non-uniform motion blurred super-resolution image. Background Art

[0002] With the rapid development of imaging technology, images have become another major information carrier, alongside language and text. Image resolution is crucial for capturing information. During the imaging process, especially when imaging moving objects, non-uniform motion blur can easily occur due to the object's relative motion to the lens and the camera's exposure time. This reduces image resolution and significantly hinders subsequent image processing. Super-resolution technology restores low-resolution images to high-resolution. The resulting pleasing high-resolution images can significantly improve the performance of other machine vision tasks. In recent years, super-resolution technology has also garnered widespread attention.

[0003] The deblurring problem based on a single image is highly underdetermined, especially for non-uniform motion blurred images. During the blurring process, the sharp image is equivalent to being convolved with different convolution kernels at different locations in the image, making the deblurring significantly more difficult than for uniform motion blurred images. Furthermore, image information at high-frequency zero points in non-uniform motion blurred images is lost during the imaging process, and the degree of loss varies. Therefore, deblurring non-uniform motion blurred images requires segmenting different regions and applying different convolution kernels to achieve deblurring, making non-uniform motion blur a complex, difficult, and pathological problem.

[0004] For example, a method for adaptive restoration of non-uniform motion blurred images based on an attention model, published in Chinese patent literature, has the publication number CN111275637A. It designs a conditional generative adversarial network combined with an attention mechanism. The generative network is a codec structure. In the encoding stage, a densely connected network is used to extract features, improve feature utilization, and enhance feature propagation. A visual attention mechanism is added to enable the network to adaptively adjust network parameters for different input images and dynamically remove image blur. The method incorporates an attention mechanism to assign corresponding weights to areas with different blur levels, which improves the deblurring effect compared to previous restoration methods. However, its input is a single image, and the image information is obtained from a single source. During the deblurring process, information is easily lost, resulting in irreversible effects. The clarity of the restored image needs to be improved. Summary of the Invention

[0005] The present invention aims to solve the problems of irreversible restoration process and unclear restoration effect caused by loss of image information due to a single image information source. It provides a non-uniform motion blurred super-resolution image restoration method and device, which can solve the problems of single information amount in the restoration process and unsatisfactory restoration effect of non-uniform motion blurred images.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A non-uniform motion blurred super-resolution image restoration method comprises the following steps:

[0008] S1. Build a dataset

[0009] S1-1, calculating the rotation angle of the image with the maximum complementarity;

[0010] S1-2, obtaining a high-resolution blurred image and a high-resolution clear image with high complementarity;

[0011] S1-3, image preprocessing to build dataset;

[0012] S2, input the preprocessed data set into the generator to obtain a preliminary restored image;

[0013] S3, using a discriminator to discriminate the preliminary restored image and the real image to obtain a discrimination result, where the discriminator is a Markov discriminant network;

[0014] S4. Optimizing a generative adversarial network using a loss function to obtain a network result with optimal performance and an optimal restored image, wherein the generative adversarial network includes a generator and a discriminator;

[0015] S5. Output the restoration result.

[0016] Preferably, the step S1-1 includes the following sub-steps:

[0017] S1-1-1. Set up a non-uniform motion blurred image acquisition device, and set a rotating prism directly between the target object and the camera, wherein the target object, the rotating prism and the camera are on the same optical axis;

[0018] S1-1-2. The target object light beam enters the camera lens after passing through the rotating prism. An image is obtained every time the prism is rotated one degree, and a total of 360 original images are obtained;

[0019] S1-1-3. Preprocess the 360 ​​original images and calculate the image combination with the greatest complementarity and its rotation angle.

[0020] Preferably, the step S1-1-3 includes:

[0021] 1) Use MATLAB to perform Gaussian filtering on 360 original images;

[0022] 2) Subtract the Gaussian filtered original image from the blurred image to obtain the high-frequency component h of the blurred image. i ,

[0023] hi =y i -G0*y i

[0024] where y i is a blurred image, G0 is a two-dimensional Gaussian filter convolution operator;

[0025] 3) Obtain the high-frequency component h of the blurred image i The gradient image in the horizontal and vertical directions is calculated as follows:

[0026] hd ix =h i *d x hd iy =h i *d y i=1, 2

[0027] 4) Get the global gradient image hd of the blurred image i , the calculation formula is:

[0028]

[0029] 5) Global gradient image hd of the blurred image i Perform binarization operation to obtain the effective feature T of the complementary blurred image i =(x,y):

[0030]

[0031] Among them, k i is the binarization threshold;

[0032] When performing the binarization operation, the Ostu algorithm is first used to calculate the optimal threshold of the original image. The Ostu algorithm assumes that the image is composed of two parts: the foreground area and the background area. By traversing the grayscale histogram of the foreground area and the background area in the segmentation result in the interval range of [0, 255], and then comparing the variance between the two, the grayscale threshold that maximizes the variance is the binarization threshold k. i ;

[0033] According to the optimal threshold of the original image, the gradient image of the blurred image is normalized, and the threshold selection range is set to k i ∈[0.2,0.5];

[0034] 6) Combine the binary images of the 360 ​​blurred images obtained in pairs, for a total of 360 2 Combination methods are used to obtain their binary images, and the complementarity of each combination method is calculated respectively. The complementarity formula is:

[0035]

[0036] Where T(A) and T(B) are binary images of the blurred image, M and N are the length and width of the image, and MATLAB is used to finally obtain the image combination with the greatest complementarity and its rotation angle.

[0037] Preferably, in step S1-2, the method for obtaining a high-resolution blurred image and a high-resolution clear image with high complementarity is:

[0038] S1-2-1. Monitor the motion state of the target object;

[0039] S1-2-2. When the target object is detected to be moving, the camera captures multiple frames of clear images, and then controls the rotating prism to rotate to the most complementary rotation angle. After the rotating prism rotates, the camera captures a non-uniform motion blurred image.

[0040] S1-2-3. Finally, complementary high-resolution blurred images and multiple frames of high-resolution clear images can be captured, and all complementary blurred images and the high-resolution clear image of the last frame are retained.

[0041] Preferably, in step S1-3, the obtained high-resolution blurred image pair with high complementarity is downsampled twice to obtain a low-resolution blurred image with high complementarity, and the high-resolution clear image and the low-resolution blurred image pair are combined into a data set.

[0042] Preferably, the data set includes 1000 acquired high-resolution clear images and 1000 pairs of corresponding low-resolution blurred images, the resolution of the clear images is 1280×1024, and the resolution of the low-resolution blurred images is 640×512.

[0043] Preferably, the generator is obtained by jointly optimizing the perceptual loss function, the adversarial loss function, the edge loss function and the MSE loss function, and the jointly optimized loss function is:

[0044] L=L adv +L p +L edge +L MSE .

[0045] Preferably, the optimization method in step S4 is as follows:

[0046] The difference in edge features between the initial restored image and the clear image is measured by the edge loss function, which is:

[0047]

[0048] Where S and G(b) are the real clear image and the preliminary restored image respectively, W and H are the length and width of the image, and They are gradient operations along the horizontal and vertical directions respectively;

[0049] Optimization is performed through the MSE loss function, which is:

[0050]

[0051] Where L and S are the secondary restored image generated by the multi-scale feature extraction module and the real clear image, respectively, and N is the number of elements in S and L;

[0052] Through the optimization of perceptual loss function and adversarial loss function, the perceptual loss function measures the overall difference between the generated secondary restored image and the corresponding real clear image features, and the adversarial loss function makes the generated high-quality image difficult to distinguish from the real clear image.

[0053] The adversarial loss function formula is:

[0054]

[0055] The perceptual loss function formula is:

[0056]

[0057] Among them, I B Represents the input blurred image. Since the network input is two blurred images with high complementarity, the two images are fused as I B , I S is a real clear image, G represents the generator, D represents the discriminator, and N represents the number of training images in a batch.

[0058] The present invention also discloses a non-uniform motion blurred super-resolution image restoration device, which includes an image acquisition mechanism, a memory, a processor, and a computer program stored in the memory and capable of executing the above-mentioned non-uniform motion blurred super-resolution image restoration method on the processor.

[0059] Preferably, the image acquisition mechanism includes a spectrometer, a motion sensing sensor, a rotating prism, a high-speed camera, and a first controller and a second controller. The high-speed camera includes a lens and a shutter. The rotating prism is respectively placed on the transmission and reflection light paths of the spectrometer and is coaxially arranged. The first controller and the second controller are STM32 single-chip microcomputers. The first controller and the second controller respectively receive signals transmitted by the sensor. The first controller is connected to the shutter of the high-speed camera to control the shutter action. The second controller is connected to the two rotating prisms to control the rotation of the prisms. The first controller and the second controller work together.

[0060] The present invention has the following characteristics and beneficial effects:

[0061] 1. This invention addresses the problem of non-uniform motion deblurring by constructing a new dataset. This dataset consists of complementary image pairs and clear images. To effectively acquire these complementary and clear images, a new image acquisition device was constructed. By using the image complementarity formula to calculate the rotation angle that maximizes complementarity, a motion sensing sensor and a second controller precisely and efficiently control signal transmission and reception, as well as prism rotation. This produces a clear image of the target scene and a complementary blurred image pair. After processing, this new dataset is ultimately constructed.

[0062] 2. Deformable convolutions are incorporated into the network, replacing conventional convolutions. The regular sampling grid in conventional convolutions makes it difficult for the network to adapt to geometric deformations. Deformable convolutions, however, adapt the convolution kernel to the sampling position, allowing the network to adapt to geometric deformations in the image. This reduces the inaccurate feature extraction caused by the fixed convolution kernel in traditional convolutions and avoids information redundancy. Furthermore, using deformable convolutions in the upsampling layer allows the network to more accurately reconstruct and restore the image.

[0063] 3. The image restoration method uses the Attention Residual Block (RAM) to combine channel attention and spatial attention, using a sequential connection. The feature maps input to the channel attention are divided into two groups. Using skip connections, one group enters the channel attention to obtain weights, while the other group, through a skip connection, is fused with the weighted feature maps of the other group and enters the spatial attention. This allows the spatial attention module to receive more information, mitigates spatial information loss, and reduces the network's computational workload. Unlike previous spatial attention designs, this approach uses GN instead of BN and deformable convolution instead of standard convolution, generating more accurate attention maps while avoiding batch size limitations.

[0064] 4. The image restoration method implements a multi-scale residual block. This block primarily consists of dilated convolutions, which are sequentially connected with dilation rates set to 1, 2, and 3. This block exploits deep image information, avoids information omission during the convolution process, and reduces network computational overhead. Skip connections are added and a fusion module is introduced to fuse information from different levels to obtain multi-scale information, accelerating network convergence.

[0065] 5. Unlike previous generative adversarial networks that only use a blurred image as network input, resulting in a single source of image information and, due to various reasons, loss of image information during the restoration process, the resulting restoration effect is unsatisfactory. Using image pairs with maximum complementarity as network input enriches the source of image information, retains more scene information, improves the clarity of the restored image, and reduces the possibility of irreversibility in the restoration process. Furthermore, to address the issue of complementary image information fusion, an adaptive residual block is proposed. This module combines image information from the edge feature extraction module, the multi-scale feature extraction module, and the upper-layer sub-pixel convolution through deformable convolution and spatial attention mechanisms, fully leveraging information complementarity to effectively reconstruct a high-resolution, clear image. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 , a flow chart of a method according to an embodiment of the present invention.

[0068] Figure 2 , schematic diagram of the image acquisition unit in an embodiment of the present invention.

[0069] Figure 3 , a principle block diagram of an embodiment of the present invention.

[0070] Figure 4 , a schematic structural diagram of the generator in an embodiment of the present invention.

[0071] Figure 5 , a schematic structural diagram of the discriminator in an embodiment of the present invention.

[0072] Figure 6 , structural diagram of the edge feature extraction module and its submodules in an embodiment of the present invention.

[0073] Figure 7 , a structural diagram of a multi-scale feature extraction module and its submodules in an embodiment of the present invention;

[0074] Figure 8 , multi-stream feature fusion and super-resolution reconstruction module diagram in an embodiment of the present invention; DETAILED DESCRIPTION

[0075] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0076] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0077] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0078] The present invention provides a method for super-resolution image restoration of non-uniform motion blur, such as Figure 1 As shown, the following steps are included:

[0079] S1. Build a dataset

[0080] S1-1, calculating the rotation angle of the image with the maximum complementarity;

[0081] S1-2, obtaining a high-resolution blurred image and a high-resolution clear image with high complementarity;

[0082] S1-3, image preprocessing to build dataset;

[0083] S2, input the preprocessed data set into the generator to obtain a preliminary restored image;

[0084] Specifically, such as Figure 3 and Figure 4As shown in the figure, the generator includes an edge feature extraction module, a multi-scale feature extraction module, and a multi-stream feature fusion and super-resolution reconstruction module. The multi-scale feature extraction module is connected in parallel with the edge feature extraction module and to the multi-stream feature fusion and super-resolution reconstruction module. The generator extracts image features in two branches, then fuses the image feature information from different sources and performs super-resolution on the blurred image, ultimately completing the image deblurring task and obtaining a pleasingly clear image.

[0085] Further, such as Figure 6-Figure 8 As shown in Figure 1, the network randomly crops the blurred image with a resolution of 640×360 to 256×256 pixels for image preprocessing. The low-resolution blurred image pair (blurred image 1 and blurred image 2) is then input into the edge feature extraction module and multi-scale feature extraction module of the generator respectively.

[0086] Among them, the edge feature extraction module adopts an asymmetric U-net network structure. The specific structure of the module is as follows: Figure 6 As shown in (a) of the figure, the blurred image 1 is input and features are extracted through a 7×7 convolutional layer with a stride of 1, a padding of 3, and 0 padding to obtain a 64-feature map. The features are then subjected to GN regularization and activation operations.

[0087] The present invention proposes an attention residual block (RAM), the specific structure of which is shown in the figure Figure 6As shown in (b) of the figure, this module consists of channel attention, spatial attention, and deformable convolution. It can focus on severely blurred areas of the image, generate corresponding weight maps, and guide the network to extract edge information from the blurred image. RAM combines channel attention and spatial attention in a sequential manner. Since image motion blur is the result of convolution between spatial pixels and convolution kernels, from a spatial perspective, the channel attention mechanism is applied globally and focuses on which feature maps are more important, while the spatial attention mechanism focuses more on the local areas of the image, that is, where the feature maps are more important. Therefore, placing spatial attention after channel attention is more conducive to focusing on severely blurred areas. In addition, unlike the CBAM block, the CBAM block reduces the number of channels and then uses convolution to obtain spatial image information. However, this can cause spatial information to be lost due to global pooling. To address this problem, a skip connection is added between the channel attention and spatial attention. At the same time, the feature maps input to the channel attention are divided into two groups. One group enters the channel attention to obtain weights, and the other group is fused with the weighted feature maps through a skip connection and enters the spatial attention. This allows the spatial attention module to receive more information, alleviates spatial information loss, and reduces the network's computational workload. At the same time, different from the previous spatial attention design, GN is used instead of BN and deformable convolution is used instead of standard convolution to generate more accurate attention maps while avoiding batch size limitations. A total of three RAM blocks are set up in sequence to form a residual structure in the residual, which helps to extract clean image information, such as Figure 6 As shown in (c) in

[15] , the feature map is fed into the attention residual block and downsampled after RAM, which helps to reduce the loss of image spatial information. A residual block is added to the downsampling layer, and a jump connection is added to it, as shown in

[15] : Figure 6 As shown in (d) of the figure, the upsampling layer adds pixel shuffling to avoid image information loss and redundant information addition. Furthermore, skip connections are added between downsampling and upsampling layers of the same scale to accelerate network convergence. Feature maps and preliminary restored images of different scales are obtained.

[0088] The blurred image 2 is input into the multi-scale feature extraction module. The multi-scale feature extraction module includes a multi-scale residual block and a deformable convolution. The network structure is as follows: Figure 7As shown in (a) in the figure. The multi-scale feature extraction module adopts a symmetrical U-net network structure, including skip connections between the downsampling layer and the upsampling layer that retain the mirror image. The downsampling layer consists of three multi-scale residual blocks and a convolution layer with a stride of 2, and the upsampling layer consists of pixel shuffling and deformable convolution. The multi-scale residual block consists of three dilated convolutions with a convolution kernel of 3×3 and an output layer with a convolution kernel size of 1×1 for feature fusion. The dilation rates are 1, 2, and 3 respectively. By setting the dilation rates of the continuously arranged dilated convolutions to a sawtooth shape, information omission during the convolution process is avoided. The multi-scale residual block is different from the previous parallel form of convolution. It connects the dilated convolutions in series to mine deep information of the image while reducing the amount of network calculation. In order to make better use of information at different levels, skip connections are added and a fusion module is introduced to fuse the features extracted from different receptive fields to obtain multi-scale information and accelerate network convergence. The structure of the multi-scale residual block is shown as follows: Figure 7 As shown in (b) in the figure, deformable convolution and pixel shuffling are added to the decoder network. Different from the upsampling method used in the previous encoder, deformable convolution is added. The deformable convolution structure is as follows: Figure 7 As shown in (c) of the figure, the convolution kernel is adaptively changed according to the sampling position to adapt the network to geometric deformations. Pixel shuffling is added to avoid image information loss and redundant information during upsampling. Feature maps and preliminary restored images of different scales are obtained.

[0089] Finally, the feature maps obtained from the edge feature extraction module and the multi-scale feature extraction module are input into the multi-stream feature fusion and super-resolution reconstruction module to achieve feature fusion and super-resolution reconstruction. Among them, the sizes of the preliminary restored images and feature maps of each scale obtained by the two modules are 256×256, 128×128, and 64×64 respectively. The multi-stream feature fusion and super-resolution reconstruction module includes four adaptive residual blocks, three sub-pixel convolution layers, one convolution layer with LReLU and one convolution layer with tanh, and the structure is as follows: Figure 8 The proposed adaptive residual block consists of deformable convolution and spatial attention mechanism, which can better utilize the spatial information of the image and adaptively fuse features from different streams. Then, the sub-pixel convolution layer, also known as pixel shuffling, is used to achieve image super-resolution to avoid the checkerboard effect. The adaptive residual block structure is shown in Figure 8 As shown in (b) in .

[0090] S3, using a discriminator to discriminate the preliminary restored image and the real image to obtain a discrimination result, where the discriminator is a Markov discriminant network;

[0091] Specifically, such as Figure 5As shown in the figure, the discriminator uses a Markov discriminant network. The input is the real clear image and the generator restored image. The discriminator structure includes 5 convolutional layers to extract image features. Except for the last layer, each layer includes a convolutional layer with a size of 4*4 and a stride of 2, an instance normalization layer and a Leaky ReLU layer.

[0092] S4. Optimizing a generative adversarial network using a loss function to obtain a network result with optimal performance and an optimal restored image, wherein the generative adversarial network includes a generator and a discriminator;

[0093] S5. Output the restoration result.

[0094] Specifically, step S1-1 includes the following sub-steps:

[0095] S1-1-1. Set up a non-uniform motion blurred image acquisition device, and set a rotating prism directly between the target object and the camera, wherein the target object, the rotating prism and the camera are on the same optical axis;

[0096] S1-1-2, the target object light beam enters the camera lens after passing through the rotating prism. An image is obtained every time the prism is rotated one degree, and a total of 360 original images are obtained;

[0097] S1-1-3. Preprocess the 360 ​​original images and calculate the image combination with the greatest complementarity and its rotation angle.

[0098] Furthermore, the step S1-1-3 includes:

[0099] 1) Use MATLAB to perform Gaussian filtering on 360 original images;

[0100] 2) Subtract the Gaussian filtered original image from the blurred image to obtain the high-frequency component h of the blurred image. i ,

[0101] h i =y i -G0*y i

[0102] where y i is a blurred image, G0 is a two-dimensional Gaussian filter convolution operator;

[0103] 3) Obtain the high-frequency component h of the blurred image i The gradient image in the horizontal and vertical directions is calculated as follows:

[0104] hd ix =h i *d x hd iy =h i *dy i=1, 2

[0105] 4) Get the global gradient image hd of the blurred image i , the calculation formula is:

[0106]

[0107] 5) Global gradient image hd of the blurred image i Perform binarization operation to obtain the effective feature T of the complementary blurred image i =(x,y):

[0108]

[0109] Among them, k i is the binarization threshold;

[0110] When performing the binarization operation, the Ostu algorithm (maximum inter-class variance method) is first used to calculate the optimal threshold of the original image. The Ostu algorithm assumes that the image is composed of two parts: the foreground area and the background area. By traversing the grayscale histogram of the foreground area and the background area in the segmentation result in the interval range of [0, 255], and then comparing the variance between the two, the grayscale threshold that maximizes the variance is the binarization threshold k. i ;

[0111] According to the optimal threshold of the original image, the gradient image of the blurred image is normalized. The threshold selection range is generally set to k i ∈[0.2,0.5];

[0112] 6) Combine the binary images of the 360 ​​blurred images obtained in pairs, for a total of 360 2 Combination methods are used to obtain their binary images, and the complementarity of each combination method is calculated respectively. The complementarity formula is:

[0113]

[0114] Where T(A) and T(B) are binary images of the blurred image, M and N are the length and width of the image, and MATLAB is used to finally obtain the image combination with the greatest complementarity and its rotation angle.

[0115] Specifically, in step S1-2, the method for obtaining a high-resolution blurred image and a high-resolution clear image with high complementarity is:

[0116] S1-2-1. Monitor the motion state of the target object;

[0117] S1-2-2. When the target object is detected to be moving, the camera captures multiple frames of clear images, and then controls the rotating prism to rotate to the most complementary rotation angle. After the rotating prism rotates, the camera captures a non-uniform motion blurred image.

[0118] S1-2-3. Finally, complementary high-resolution blurred images and multiple frames of high-resolution clear images can be captured, and all complementary blurred images and the high-resolution clear image of the last frame are retained.

[0119] As will be appreciated, the motion of the target object is monitored by motion sensor 4 and signals are sent to a first controller and a second controller. The first controller is used to control the movement of shutter 102 of high-speed camera 1, and the second controller is used to control the rotation of rotating prism 2. There is a certain time difference between the signals sent by the first and second controllers. When motion sensor 4 detects the movement of the target object, it sends signals to the first and second controllers, with the first and second controllers sending signals sequentially at intervals. The first controller is activated first, quickly controlling shutter 102 to capture multiple frames of clear images. At this time, the angle of rotating prism 2 is 0 degrees, meaning that the second controller has not yet activated to control prism rotation, and no rotation occurs. After the interval expires, the second controller quickly sends signals to control the rotation of the two prisms to the most complementary angle. Simultaneously, the first controller controls the camera shutter to maintain a certain exposure time, capturing a non-uniform motion-blurred image. Ultimately, two blurred images and multiple clear images are captured, retaining all blurred images and the final clear image.

[0120] Furthermore, in step S1-3, the obtained high-resolution blurred image pair with high complementarity is downsampled by a factor of two to obtain a low-resolution blurred image with high complementarity, and the high-resolution clear image and the low-resolution blurred image pair are combined into a data set.

[0121] The dataset includes 1000 high-resolution clear images and 1000 pairs of corresponding low-resolution blurred images. The resolution of the clear images is 1280×1024, and the resolution of the low-resolution blurred images is 640×512.

[0122] In a further arrangement of the present invention, the generator is jointly optimized by a perceptual loss function, an adversarial loss function, an edge loss function, and an MSE loss function, and the joint loss function of the generative adversarial network is:

[0123] L=L adv +L p +L edge +L MSE .

[0124] Specifically, the optimization method of step S4 is as follows:

[0125] The difference in edge features between the initial restored image and the clear image is measured by the edge loss function, which is:

[0126]

[0127] Where S and G(b) are the real clear image and the preliminary restored image generated by the edge feature extraction module, respectively. W and H are the length and width of the image. and They are gradient operations along the horizontal and vertical directions respectively;

[0128] In the aforementioned technical solution, the edge feature extraction module is optimized by using the Sobel operator to establish an edge loss function, which measures the difference in edge features between a low-resolution blurred image and a clear image. The edge loss function takes the real image and the low-resolution blurred image as input. By calculating the image gradients and the loss function, the edge feature extraction module is optimized, helping to extract more edge details and improving its performance.

[0129] Optimization is performed through the MSE loss function, which is:

[0130]

[0131] Where L and S are the preliminary restored image generated by the multi-scale feature extraction module and the real clear image, respectively, and N is the number of elements in S and L;

[0132] In the above technical solution, for the multi-scale feature extraction module, the MSE loss function is selected for optimization to help the network extract more available features and texture details in the blurred image.

[0133] Through the optimization of perceptual loss function and adversarial loss function, the perceptual loss function measures the overall difference between the generated preliminary restored image and the corresponding real clear image features, and the adversarial loss function makes the generated high-quality image difficult to distinguish from the real clear image.

[0134] The adversarial loss function formula is:

[0135]

[0136] The perceptual loss function formula is:

[0137]

[0138] Among them, I B Represents the input blurred image. Since the network input is two blurred images with high complementarity, the two images are fused as I B , IS is a real clear image, G represents the generator, D represents the discriminator, and N represents the number of training images in a batch.

[0139] In the above technical solution, perceptual loss and adversarial loss are used to optimize the generative adversarial network, both of which impose constraints on the generator to generate restored images. Perceptual loss function L p It can measure the overall difference between the features of the generated image and the corresponding real clear image. The input is the real clear image and the restored image. The perceptual loss uses the VGG19 feature layer to calculate the difference between the generated data and the original real data.

[0140] WGAN-GP is used as the adversarial loss function, L adv This results in high-quality images that are indistinguishable from real images, and it has been shown to be robust to the choice of generator.

[0141] It is understandable that a self-constructed data set is selected for network training and testing, and each clear image and blurred image in the data set is flipped, cut, etc. to expand the database, prevent network overfitting, and enhance network adaptability.

[0142] This paper follows the training procedure proposed in WGAN. For optimization, the discriminator is trained five times, followed by the generator. Momentum-based optimizers, such as Adam and momentum, should not be used as optimizers, as they can cause instability. RMSProp or SGD can be used instead. Mini-batch stochastic gradient descent is used with the RMSProp solver. The initial learning rates for the generator and discriminator are set to 1-4, the batch size is set to 1, and the epoch is set to 300. The learning rate is linearly decayed to 0 after the last 150 epochs. Experiments show that this method achieves better performance on the test set without significant differences in experimental effectiveness.

[0143] SSIM (structural similarity) and PSNR (peak signal-to-noise ratio) are used to calculate the difference between the restored image and the real clear image, and these two values ​​are used as evaluation indicators to verify the effectiveness of the model.

[0144] The present invention also discloses a non-uniform motion blurred super-resolution image restoration device, such as Figure 2 As shown, it includes an image acquisition mechanism, a memory, a processor, and a computer program stored in the memory and capable of executing the above-mentioned non-uniform motion blurred super-resolution image restoration method on the processor.

[0145] Furthermore, the image acquisition mechanism includes a spectrometer, a motion sensing sensor, a rotating prism, a high-speed camera, and a first controller and a second controller. The high-speed camera includes a lens and a shutter. The rotating prism is respectively placed on the transmission and reflection light paths of the spectrometer and is coaxially arranged. The first controller and the second controller are STM32 single-chip microcomputers. The first controller and the second controller respectively receive signals transmitted by the sensor. The first controller is connected to the shutter of the high-speed camera to control the shutter action. The second controller is connected to the two rotating prisms to control the rotation of the prisms. The first controller and the second controller work in coordination.

[0146] In the above technical solution, a motion sensing sensor is used to monitor the movement of the target object. When the movement of the object is detected, the sensor sends a signal to the first controller and the second controller, wherein the first controller and the second controller send signals successively at a certain interval. The first controller acts first, quickly controlling the shutter to shoot multiple frames of clear pictures. At this time, the angle of the rotating prism is 0 degrees, that is, the second controller has not yet taken action to control the rotation of the prism, and no rotation action occurs. After the interval time is reached, the second controller quickly sends a signal to control the two prisms to rotate to the most complementary rotation angle. At the same time, the first controller controls the camera shutter to maintain a certain exposure time to shoot non-uniform motion blurred images. In the end, two blurred images and multiple frames of clear images can be captured, and all blurred images and the last frame of clear image are retained. Finally, the blurred image is downsampled twice to obtain a low-resolution image, which together with the high-resolution clear image constitutes the data set of the present invention.

[0147] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A non-uniform motion blurred super-resolution image restoration method, characterized in that: The steps include: S1. Build a dataset S1-1, calculating the rotation angle of the image with the maximum complementarity; S1-1-1. Set up a non-uniform motion blurred image acquisition device, and set a rotating prism directly between the target object and the camera, wherein the target object, the rotating prism and the camera are on the same optical axis; S1-1-2. The target object light beam enters the camera lens after passing through the rotating prism. An image is obtained every time the prism is rotated one degree, and a total of 360 original images are obtained; S1-1-3, pre-processing the 360 ​​original images, and calculating the image combination with the greatest complementarity and its rotation angle; S1-2, obtaining a high-resolution blurred image and a high-resolution clear image with high complementarity; S1-3, image preprocessing to build dataset; S2, input the preprocessed data set into the generator to obtain a preliminary restored image; S3, using a discriminator to discriminate the preliminary restored image and the real image to obtain a discrimination result, where the discriminator is a Markov discriminant network; S4. Optimizing a generative adversarial network using a loss function to obtain a network result with optimal performance and an optimal restored image, wherein the generative adversarial network includes a generator and a discriminator; The generator is obtained by jointly optimizing the perceptual loss function, the adversarial loss function, the edge loss function and the MSE loss function. The loss function of the joint optimization is: L=L adv +L p +L edge +L MSE ; The joint optimization method is: The difference in edge features between the initial restored image and the clear image is measured by the edge loss function, which is: Where S and G(b) are the real clear image and the preliminary restored image respectively, W and H are the length and width of the image, and They are gradient operations along the horizontal and vertical directions respectively; Optimization is performed through the MSE loss function, which is: Where L and S are the preliminary restored image and the real clear image respectively, and N is the number of elements in S and L; Through the optimization of perceptual loss function and adversarial loss function, the perceptual loss function measures the overall difference between the generated preliminary restored image and the corresponding real clear image features, and the adversarial loss function makes the generated high-quality image difficult to distinguish from the real clear image. The adversarial loss function formula is: The perceptual loss function formula is: Among them, I B Represents the input blurred image. Since the network input is two blurred images with high complementarity, the two images are fused as I B , I S is a real clear image, G represents the generator, D represents the discriminator, and N represents the number of training images in a batch; S5. Output the restoration result.

2. The method for super-resolution image restoration of non-uniform motion blur according to claim 1, characterized in that: The step S1-1-3 includes: 1) Use MATLAB to perform Gaussian filtering on 360 original images; 2) Subtract the Gaussian filtered original image from the blurred image to obtain the high-frequency component h of the blurred image. i , h i =y i -G0*y i where y i is a blurred image, G0 is a two-dimensional Gaussian filter convolution operator; 3) Obtain the high-frequency component h of the blurred image i The gradient image in the horizontal and vertical directions is calculated as follows: hd ix =h i *d x hd iy =h i *d y i=1、2 4) Get the global gradient image hd of the blurred image i , the calculation formula is: 5) Global gradient image hd of the blurred image i Perform binarization operation to obtain the effective feature T of the complementary blurred image i =(x,y): Among them, k i is the binarization threshold; When performing the binarization operation, the Ostu algorithm is first used to calculate the optimal threshold of the original image. The Ostu algorithm assumes that the image is composed of two parts: the foreground area and the background area. By traversing the grayscale histogram of the foreground area and the background area in the segmentation result in the interval range of [0, 255], and then comparing the variance between the two, the grayscale threshold that maximizes the variance is the binarization threshold k. i ; According to the optimal threshold of the original image, the gradient image of the blurred image is normalized, and the threshold selection range is set to k i ∈[0.2,0.5]; 6) Combine the binary images of the 360 ​​blurred images obtained in pairs, for a total of 360 2 Combination methods are used to obtain their binary images, and the complementarity of each combination method is calculated respectively. The complementarity formula is: Where T(A) and T(B) are binary images of the blurred image, M and N are the length and width of the image, and MATLAB is used to finally obtain the image combination with the greatest complementarity and its rotation angle.

3. The method for super-resolution image restoration of non-uniform motion blur according to claim 2, characterized in that: In step S1-2, the method for obtaining a high-resolution blurred image and a high-resolution clear image with high complementarity is: S1-2-1, monitoring the motion state of the target object; S1-2-2. When the target object is detected to be moving, the camera captures multiple frames of clear images, and then controls the rotating prism to rotate to the most complementary rotation angle. After the rotating prism rotates, the camera captures a non-uniform motion blurred image. S1-2-3. Finally, complementary high-resolution blurred images and multiple frames of high-resolution clear images can be captured, and all complementary blurred images and the high-resolution clear image of the last frame are retained.

4. The method for super-resolution image restoration of non-uniform motion blur according to claim 1, characterized in that: In step S1-3, the obtained high-resolution blurred image pair with high complementarity is downsampled by a factor of two to obtain a low-resolution blurred image with high complementarity, and the high-resolution clear image and the low-resolution blurred image pair are combined into a data set.

5. The method for super-resolution image restoration of non-uniform motion blur according to claim 4, characterized in that: The dataset includes 1000 high-resolution clear images and 1000 pairs of corresponding low-resolution blurred images. The resolution of the clear images is 1280×1024, and the resolution of the low-resolution blurred images is 640×512.

6. A non-uniform motion blurred super-resolution image restoration device, characterized in that: The invention comprises an image acquisition mechanism, a memory, a processor and a computer program stored in the memory and capable of executing the non-uniform motion blurred super-resolution image restoration method according to any one of claims 1 to 5 on the processor.

7. The non-uniform motion blurred super-resolution image restoration device according to claim 6, characterized in that: The image acquisition mechanism includes a spectroscope, a motion sensing sensor, a rotating prism, a high-speed camera, and a first controller and a second controller. The high-speed camera includes a lens and a shutter. The rotating prism is respectively placed on the transmission and reflection light paths of the spectroscope and is coaxially arranged. The first controller and the second controller are STM32 single-chip microcomputers. The first controller and the second controller respectively receive signals transmitted by the sensors. The first controller is connected to the shutter of the high-speed camera to control the shutter action. The second controller is connected to the two rotating prisms to control the rotation of the prisms. The first controller and the second controller work in coordination.

Images