A block scan based image super-resolution method and system

Abstract

The application provides a kind of image super-resolution method and system based on block scanning, utilize low-resolution detector to combine high-resolution spatial modulator by block scanning to obtain multiple low-resolution images containing target different pixel position information, simultaneously propose a kind of reconstruction method based on total variation to multiple low-resolution images are fused and reconstructed, finally obtain target high-resolution reconstructed image.The mechanism of block scanning is different from point scanning, line scanning and compressed sensing method, by the way of overlapping moving block scanning, grid and stripe noise that can appear in scanning process is effectively avoided, and image super-resolution can be realized under lower sampling number.Through adjusting sampling number, effective reconstruction result can be obtained under different sampling time, and with the increase of sampling number, the reconstruction quality reaches excellent effect.The method provided by the application has remarkable resolution enhancement effect, and the method is competitive.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image super-resolution, and relates to an image super-resolution method and system based on block scanning, which is suitable for various resource-limited image super-resolution application scenarios. BACKGROUND

[0002] Image super-resolution (SR) is an image reconstruction technology that obtains a single or multiple low-resolution images containing different spatial information of the same scene and restores a high-resolution (HR) image of the scene by using a corresponding algorithm. This technology can obtain a high-resolution image by using a low-resolution detector, improve the visual clarity of an image, and has been widely applied in medical diagnosis, remote sensing detection, computer vision, military and other fields [1-4] .

[0003] Traditional image super-resolution methods include interpolation, reconstruction and learning methods. The earliest interpolation method [5] can only use the pixel information of a low-resolution image to reconstruct an image, without considering the image degradation model, and it is difficult to obtain satisfactory results. In order to alleviate this problem, the reconstruction-based method [6-7] mines the mapping relationship between the LR and HR image blocks as much as possible, and establishes an artificial priori to help solve the underdetermined problem of image super-resolution. However, in the case of complex actual mapping relationship, the artificial priori is difficult to accurately describe, thereby limiting the improvement of the reconstruction quality. In recent years, the method based on deep learning [8-11] has become a research hotspot, and some excellent results have been achieved by relying on the learning of data features of a convolutional neural network. However, due to the limited real information of the target contained in the input image, the reconstruction method based on software algorithm alone is still difficult to realize the reconstruction of a complex image.

[0004] In recent years, the light path multiplexing technology for images or scenes based on spatial light modulation devices

[12] brings new ideas to the super-resolution imaging problem. The full high-pixel information of the target can be obtained most directly by the pixel-by-pixel scanning method, but the time resolution is sacrificed. In order to speed up the imaging, the compressive sensing technology is applied to this subject [13-14] , but due to the influence of the detector fill factor, the reconstruction result often appears grid noise. A recent image super-resolution method based on line scanning

[15] has effectively balanced the sampling time and the reconstruction quality, but the stripe noise caused by the line scanning mechanism is difficult to completely remove, which affects the visual effect to some extent. SUMMARY

[0005] The present application aims at the deficiencies of the prior art, adopts a spatial block scanning mode, uses different position space blocks to scan a target to obtain multiple low-resolution images containing different sub-pixel information of the target, and then performs joint reconstruction to obtain a high-resolution target image. Due to the overlap and offset of different spatial position information in the block scanning process, the grid and stripe noise in the compression sensing and line scanning processes will not appear after reconstruction, effectively improving the image reconstruction quality and visual effect.

[0006] The block scanning based image super-resolution method is designed by using a block based light multiplexing technology to design a new imaging framework, collects multiple low-resolution images containing different spatial block information of the target, and further proposes a reconstruction algorithm based on total variation to reconstruct the multiple low-resolution images. How to realize higher quality image super-resolution at a lower sampling rate is the key problem of the block scanning based image super-resolution method.

[0007] The technical scheme adopted by the present application is: a block scanning based image super-resolution method, first, a block scanning modulation matrix is designed, and is used as a mask matrix to act on an original high-resolution image to simulate the modulation process of the spatial light modulator on the target light field. Secondly, a downsampling operation is performed on each modulated high-resolution image to simulate the imaging process of the low-resolution detector, so as to obtain multiple block scanning low-resolution images containing different spatial position information of the target. Finally, a reconstruction algorithm based on total variation is proposed to reconstruct the observation data to obtain the final high-resolution image. The imaging system principle is shown in Figure 1 The target is first imaged on the working surface of the high-resolution spatial light modulator through the objective lens, and then imaged on the low-resolution detector through the imaging lens after being modulated by different modulation matrices, so that multiple low-resolution images containing different spatial information of the target are collected. In this imaging process, the spatial light modulator modulation frequency and the detector detection frequency are kept consistent by computer control. The method includes the following steps:

[0008] Step 1: design a block scanning sampling matrix, so that each scanning matrix contains different spatial positions.

[0009] Step 2: make each scanning matrix act on the target high-resolution image to simulate the block scanning process of the spatial light modulator on the target light field.

[0010] Step 3: perform a downsampling operation on the scanned and modulated result to simulate the imaging process of the low-resolution detector.

[0011] Step 4: construct a reasonable mathematical model according to the image processing process and image characteristics.

[0012] Step 5: Solve the constructed mathematical model to obtain the final high-resolution image reconstruction result.

[0013] Further, the specific implementation of step 1 includes the following sub-steps:

[0014] Step 1.1, when the target high-resolution image is M and N represent the size of the target high-resolution image, the detector resolution size is m x n, and the resolution enhancement factor S is the ratio of the high-resolution image to the detector resolution, that is:

[0015]

[0016] The resolution size of each modulation matrix is set to

[0017] Step 1.2, in order to obtain different spatial position pixel information of the target in different modulations, each block simultaneously performs a moving scan with an interval of w, and k scanning modulation matrices are obtained. In the k modulation matrices, the i-th scanning matrix Φ i is expressed as:

[0018]

[0019] where the modulation unit The specific content is:

[0020]

[0021] where

[0022] k is flexibly adjusted with the change of the scanning moving interval w, where As k increases, the target information collected is more abundant and the reconstruction quality is higher, but the acquisition time is increased to some extent.

[0023] Further, in step 2, in order to simulate the modulation process of the spatial light modulator on the target light field information, we respectively multiply the different scanning matrices with the target high-resolution image to obtain multiple modulated high-resolution images, that is

[0024] M i = Φ i I (i = {1, k}) (4)

[0025] where k is the modulation number, and as k increases, the modulation information obtained is more abundant and the reconstruction quality is high, but the sampling time is increased to some extent.

[0026] Further, in step 3, in order to simulate the imaging process of the low-resolution detector, we perform a low-pass filter on the modulated image M iFurther down-sampling operation is performed with down-sampling operator Ψ, and a plurality of low-resolution images with resolution of m x n containing different spatial modulation information of the target are obtained.

[0027] Further, the specific implementation of step 4 includes the following sub-steps:

[0028] Step 4.1, constructing a mathematical model intrinsic term. According to the above steps, the mathematical relationship between the observed image and the high-resolution true value to be reconstructed image is:

[0029] F i = ΨM i I (i = {1, k}) (5)

[0030] Step 4.2, the above equation is an underdetermined problem, in order to solve this problem, we set an artificial priori term, considering the piecewise smooth characteristics of the solution target I, we add a total variation constraint to I.

[0031] According to step 4, we describe the block scanning image reconstruction problem as the following mathematical model:

[0032]

[0033] Wherein, F refers to F norm, defined as the arithmetic square root of the sum of squares of all elements of the matrix, that is TV is the total variation constraint, defined as ||I|| TV = ∑ i,j |I i+1,j -I i,j |+|I i,j+1 -I i,j |。

[0034] Further, in step 5, we can directly use the augmented Lagrange method

[16] to solve the mathematical model in step 4 to obtain the final reconstructed high-resolution result I.

[0035] The application also provides a block scanning based image super-resolution system, comprising the following modules:

[0036] A sampling matrix design module is used to design a block scanning sampling matrix, so that each scanning matrix contains a block with a value of 1 in different spatial positions.

[0037] A modulation module is used to make each scanning matrix act on the target high-resolution image respectively, simulating the block scanning process of the spatial light modulator on the target light field.

[0038] An imaging module is used to perform down-sampling operation on the scanning and modulation results, simulating the imaging process of the low-resolution detector.

[0039] A model construction module is configured to construct a reasonable mathematical model according to the image processing process and the image features;

[0040] A reconstruction module is configured to solve the constructed mathematical model to obtain a final high-resolution image reconstruction result.

[0041] Further, the specific implementation of the sampling matrix design module includes the following sub-steps:

[0042] Step 1.1, when the target high-resolution image is M and N represent the size of the target high-resolution image, the detector resolution size is mxn, and the resolution enhancement factor S is the ratio of the high-resolution image to the detector resolution, that is:

[0043]

[0044] The resolution size of each modulation matrix is set to

[0045] Step 1.2, in order to obtain the pixel information of different spatial positions of the target in different modulations, each block simultaneously performs a moving scan with an interval of w, and k scanning modulation matrices are obtained, in the k modulation matrices, the i-th scanning matrix Φ i is expressed as:

[0046]

[0047] Where the modulation unit The specific content is:

[0048]

[0049] Where

[0050] Further, the specific implementation of the model construction module includes the following sub-steps:

[0051] Step 4.1, construct the mathematical model intrinsic term, the mathematical relationship between the observed image and the high-resolution true value to be reconstructed image is:

[0052] F i = ΨM i I (i={1,k}) (5)

[0053] Step 4.2, the above equation (5) is an underdetermined problem, in order to solve this problem, an artificial prior term is set, that is, the piecewise smoothness feature of the solution target I is considered, and a total variation constraint is added to I;

[0054] Finally, the block scanning image reconstruction problem is described as the following mathematical model:

[0055]

[0056] Wherein, k is the modulation times, F refers to F norm, defined as the arithmetic square root of the sum of squares of all elements of the matrix, TV is the total variation constraint.

[0057] Compared with the prior art, the advantages and beneficial effects of the present application are as follows: the present application considers using a block scanning based optical multiplexing technology, and proposes a new imaging framework. Different spatial position pixel information of the target is obtained through the moving block scanning, and a total variation based reconstruction algorithm is proposed to fuse and reconstruct the collected multiple low resolution images, so that the super-resolution imaging of the image is realized, and the grid and stripe noise caused by the non-uniformity of the CCD pixel response in other scanning imaging processes is effectively avoided, and the image reconstruction quality is improved. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 It is a block scanning based image super-resolution method system principle diagram.

[0059] Figure 2 It is a scanning modulation matrix example diagram.

[0060] Figure 3 It is a simulation example diagram of the CCD collected image after modulation.

[0061] Figure 4 It is an initial low resolution image and a final joint reconstruction result diagram.

[0062] Figure 5 It is a comparison diagram of 4 times super-resolution results under different modulation times k. DETAILED DESCRIPTION

[0063] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the examples described herein are only used to explain the present application, and are not used to limit the present application.

[0064] The present application mainly aims at the application demand of image super-resolution. Low resolution detector is used to collect high resolution block scanning target information projected on a spatial light modulator multiple times, and a total variation based image reconstruction method is used to fuse and reconstruct the collected low resolution images, so that a target high resolution image is finally obtained.

[0065] Step 1: design a block scanning sampling matrix, so that each scanning matrix contains blocks with different spatial positions.

[0066] Step 2: make each scanning matrix act on the target high resolution image respectively, simulate the block scanning process of the spatial light modulator on the target light field.

[0067] Step 3: Perform a downsampling operation on the scanned and modulated results to simulate the imaging process of a low-resolution detector.

[0068] Step 4: Construct a reasonable mathematical model based on the image processing procedure and image features.

[0069] Step 5: Solve the constructed mathematical model to obtain the final high-resolution image reconstruction result.

[0070] Furthermore, the specific implementation of step 1 includes the following sub-steps:

[0071] Step 1.1, when the target high-resolution image is The detector resolution is m×n (m=n=64). The resolution enhancement factor S is the ratio of the high-resolution image to the detector resolution, i.e.:

[0072]

[0073] The resolution size of each modulation matrix is ​​set to

[0074] Step 1.2: To obtain pixel information of different spatial locations of the target under different modulations, each block undergoes simultaneous moving scans at intervals of w, resulting in k scanning modulation matrices. Among the k modulation matrices, the i-th scanning matrix Φ... i Represented as:

[0075]

[0076] The modulation unit The specific content is as follows:

[0077]

[0078] in

[0079] k is flexibly adjusted according to the change of the scanning movement interval w, where

[0080] Modulation matrix such as Figure 2 As shown in the figure, we performed image reconstruction in the simulation experiment with k=4, 8, and 16 respectively, and the results are attached. Figure 5 As shown, as k increases, the collected target information becomes richer and the reconstruction quality is higher, but the acquisition time increases to some extent.

[0081] Furthermore, in step 2, to simulate the modulation process of the spatial light modulator on the target light field information, we perform dot multiplication between different scanning matrices and the target high-resolution image to obtain multiple modulated high-resolution images, i.e.

[0082] M i = Φ i I (i = {1, k}) (4)

[0083] Further, in step 3, in order to simulate the imaging process of the low-resolution detector, we down-sample the modulated image M i Further, the down-sampling operator is Ψ, and a plurality of low-resolution images with a resolution of 64x64 containing different spatial modulation information of the target are obtained, and one of the acquired low-resolution images is shown in FIG. 4. Figure 3

[0084] Further, the specific implementation of step 4 includes the following sub-steps:

[0085] Step 4.1, constructing a mathematical model intrinsic term. According to the above steps, the mathematical relationship between the observed image and the high-resolution true value to be reconstructed image is:

[0086] F i = ΨM i I (i = {1, k}) (5)

[0087] Step 4.2, the above equation is an underdetermined problem, in order to solve this problem, we set an artificial priori term. Considering the piecewise smoothness characteristics of the solution target I, we add a total variation constraint to I.

[0088] According to step 4, the block scanning image reconstruction problem is described as the following mathematical model:

[0089]

[0090] Further, in step 5, we can use the augmented Lagrange method to solve the mathematical model in step 4 to obtain the final reconstructed high-resolution result I.

[0091] The application also provides a block scanning-based image super-resolution system, comprising the following modules:

[0092] A sampling matrix design module is used to design a block scanning sampling matrix, so that each scanning matrix contains a block with a value of 1 at different spatial positions;

[0093] A modulation module is used to make each scanning matrix act on a target high-resolution image, respectively, to simulate the block scanning process of a spatial light modulator on a target light field;

[0094] An imaging module is used to down-sample the scanning modulated result to simulate the imaging process of a low-resolution detector;

[0095] A model construction module is used to construct a reasonable mathematical model according to the image processing process and image characteristics;​

[0096] a reconstruction module configured to solve the constructed mathematical model to obtain a final high-resolution image reconstruction result.

[0097] The specific implementation of each module and each step is corresponding, and the present application is not described.

[0098] By the accompanying Figure 4 and the accompanying Figure 5 It can be seen that our method can obtain effective reconstruction results under different sampling times, and the reconstruction quality is obviously improved with the increase of the sampling times, so it can be seen that our method has obvious effect in avoiding the noise problem caused by the scanning process, so as to obtain high-quality reconstruction results, and the method has significant effect and important significance for realizing high-resolution image reconstruction by using low-resolution detectors.

[0099] It should be understood that parts not described in detail in the specification are all prior art.

[0100] It should be understood that the above description of the embodiments is more detailed, and therefore cannot be considered as a limitation on the scope of patent protection of the present application. Ordinary skilled persons in the art can make substitutions or modifications under the inspiration of the present application without departing from the scope of protection claimed by the present application, and all fall within the scope of protection of the present application. The scope of protection claimed by the present application shall be subject to the appended claims.

[0101] References

[0102] [1] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “SwinIR: Image restoration using swin transformer,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, (2021), pp. 1833-1844.

[0103] [2] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis Mach. Intell. (2022).

[0104] [3] Y. Xu, W. Luo, A. Hu, Z. Xie, X. Xie, and L. Tao, “TE-SAGAN: An improved generative adversarial network for remote sensing super-resolution images,” Remote. Sens. 14, 2425 (2022).

[0105] [4] W. Yan, Y. Yang, Y. Tan, X. Chen, Y. Li, J. Qu, and T. Ye, “Coherent optical adaptive technique improves the spatial resolution of STED microscopy in thick samples,” Photonics Res. 5, 176-181 (2017).

[0106] [5] J. Liu, Z. Gan, and X. Zhu, “Directional bicubic interpolation—a new method of image super-resolution,” in 3rd International Conference on Multimedia Technology (ICMT-13), (Atlantis Press, 2013), pp. 463-470.

[0107] [6] P. Rasti, H. Demirel, and G. Anbarjafari, “Image resolution enhancement by using interpolation followed by iterative back projection,” in 2013 21st Signal Processing and Communications Applications Conference (SIU), (IEEE, 2013), pp. 1-4.

[0108] [7] W. Dong, G. Shi, X. Li, Y. Ma, and F. Huang, “Compressive sensing via nonlocal low-rank regularization,” IEEE Transactions on Image Process. 23, 3618-3632 (2014).

[0109] [8] P. Yi, Z. Wang, K. Jiang, J. Jiang, T. Lu, and J. Ma, “A progressive fusion generative adversarial network for realistic and consistent video super-resolution,” IEEE Transactions on Pattern Analysis Mach. Intell. (2020).

[0110] [9] M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back-projection networks for super-resolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2018), pp. 1664-1673.

[0111]

[10] K. Zhang, L. V. Gool, and R. Timofte, “Deep unfolding network for image super-resolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2020), pp. 3217-3226.

[0112]

[11] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2018), pp. 9446-9454.

[0113]

[12] H. Chen, M. Salman Asif, A. C. Sankaranarayanan, and A. Veeraraghavan, “Fpa-cs: Focal plane array-based compressive imaging in short wave infrared,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2015), pp. 2358-2366.

[0114]

[13] F. Li, H. Chen, A. Pediredla, C. Yeh, K. He, A. Veeraraghavan, and O. Cossairt, “CS-ToF: High-resolution compressive time-of-flight imaging,” Opt. Express 25, 31096-31110 (2017).

[0115]

[14] Q. Sun, X. Dun, Y. Peng, and W. Heidrich, “Depth and transient imaging with compressive SPAD array cameras,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2018), pp. 273-282.

[0116]

[15] X. Tian, Y. Xiao, R. Liu, F. He, and J. Ma, “Line-wise scanning-based super-resolution imaging,” Opt. Lett. 47, 2230-2233 (2022).

[0117]

[16] S. H. Chan, R. Khoshabeh, K. B. Gibson, P. E. Gill, and T. Q. Nguyen, IEEE Transactions on Image Process. 20, 3097 (2011).

Claims

1. A block-scan-based image super-resolution method, characterized in that, Includes the following steps: Step 1: Design a block scan sampling matrix such that blocks with a value of 1 in each scan matrix contain different spatial locations; Step 1 includes the following sub-steps: Step 1.1, when the target high-resolution image is I M and N represent the size of the high-resolution image of the target, and the detector resolution size is m×n. The resolution enhancement factor is [missing value]. S This is the ratio of the high-resolution image to the detector resolution, i.e.: (1) The resolution size of each modulation matrix is ​​set to , ; Step 1.2: To obtain pixel information of different spatial locations of the target under different modulations, each block simultaneously undergoes a moving scan with an interval of w, resulting in k scanning modulation matrices. Among the k modulation matrices, the i-th scanning matrix... Represented as: (2) The modulation unit The specific content is as follows: (3) in ; Step 2: Each scanning matrix is ​​applied to the target high-resolution image to simulate the block scanning process of the spatial light modulator on the target light field. Step 3: Perform downsampling on the scanned and modulated results to simulate the imaging process of a low-resolution detector; Step 4: Construct a reasonable mathematical model; the specific implementation includes the following sub-steps: Step 4.1: Construct the eigenterms of the mathematical model and observe the image. The mathematical relationship between the image and the high-resolution ground truth image to be reconstructed is as follows: (5) in, For downsampling operators, The image modulated in step 2; Step 4.2, the above equation (5) is an underdetermined problem. In order to solve this problem, artificial prior terms are set, that is, the piecewise smooth characteristics of the target I are considered and total variation constraints are added to I. Ultimately, this block scan image reconstruction problem can be described by the following mathematical model: Where k is the modulation number, F refers to the F norm, which is defined as the arithmetic square root of the sum of squares of all elements of the matrix, and TV is the total variation constraint; Step 5: Solve the constructed mathematical model to obtain the final high-resolution image reconstruction result.

2. The image super-resolution method based on block scanning as described in claim 1, characterized in that: k is flexibly adjusted according to the change of the scanning movement interval w, where As k increases, the collected target information becomes richer and the reconstruction quality is higher, but the collection time increases to some extent.

3. The image super-resolution method based on block scanning as described in claim 1, characterized in that: In step 2, to simulate the modulation process of the spatial light modulator on the target light field information, different scanning matrices are multiplied with the target high-resolution image to obtain multiple modulated high-resolution images, i.e. (4) Where k is the modulation number.

4. The image super-resolution method based on block scanning as described in claim 1, characterized in that: In step 3, to simulate the imaging process of a low-resolution detector, the image modulated in step 2 is processed. Further downsampling is performed; the downsampling operator is: This results in multiple low-resolution images with a resolution of m×n, each containing different spatial modulation information of the target.

5. The image super-resolution method based on block scanning as described in claim 1, characterized in that: In step 5, the augmented Lagrange method is directly used to solve the mathematical model in step 4 to obtain the final high-resolution reconstruction result.

6. A block-scan-based image super-resolution system, characterized in that, Includes the following modules: The sampling matrix design module is used to design the block scan sampling matrix so that blocks with a value of 1 in each scan matrix contain different spatial locations. The specific implementation of the sampling matrix design module includes the following sub-steps: Step 1.1, when the target high-resolution image is I M and N represent the size of the high-resolution image of the target, and the detector resolution size is m×n. The resolution enhancement factor is [missing value]. S This is the ratio of the high-resolution image to the detector resolution, i.e.: (1) The resolution size of each modulation matrix is ​​set to , ; Step 1.2: To obtain pixel information of different spatial locations of the target under different modulations, each block simultaneously undergoes a moving scan with an interval of w, resulting in k scanning modulation matrices. Among the k modulation matrices, the i-th scanning matrix... Represented as: (2) The modulation unit The specific content is as follows: (3) in ; The modulation module is used to enable each scanning matrix to interact with the target high-resolution image separately, simulating the block scanning process of the spatial light modulator on the target light field; The imaging module is used to downsample the scanned and modulated results to simulate the imaging process of a low-resolution detector. The model building module is used to construct a reasonable mathematical model; its implementation includes the following sub-steps: Step 4.1: Construct the eigenterms of the mathematical model and observe the image. The mathematical relationship between the image and the high-resolution ground truth image to be reconstructed is as follows: (5) in, For downsampling operators, The image is modulated; Step 4.2, the above equation (5) is an underdetermined problem. In order to solve this problem, artificial prior terms are set, that is, the piecewise smooth characteristics of the target I are considered and total variation constraints are added to I. Ultimately, this block scan image reconstruction problem can be described by the following mathematical model: Where k is the modulation number, F refers to the F norm, which is defined as the arithmetic square root of the sum of squares of all elements of the matrix, and TV is the total variation constraint; The reconstruction module is used to solve the constructed mathematical model to obtain the final high-resolution image reconstruction result.

Images