A synthetic aperture system image restoration method of non-paired data supervised optimization

Abstract

The application discloses a kind of non-paired data supervision optimization synthetic aperture system image restoration methods, it is related to image restoration field, its technical features are: for the imaging blur and noise interference problem of optical synthetic aperture imaging system, the degraded target A obtained by imaging is combined with the clear target B easily obtained to form non-paired image data.Further, the degraded target B obtained by calculation simulation is input into the network together with the degraded target A, the clear target B is used as supervision, and the regular constraint is applied to the restored image, the network parameters are supervised and optimized, and finally the image restoration of the degraded target is realized.The advantage of this method is that it has stronger noise resistance, does not require the point spread function of the system to be known and manual parameter adjustment, and does not require the corresponding clear target A. Image restoration can be realized only with non-paired degraded target A and clear target B, and can be generalized to other targets and systems, suitable for direct imaging scenarios of optical synthetic aperture systems.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image restoration, and particularly relates to an image restoration processing method based on an optical synthetic aperture imaging system. BACKGROUND

[0002] In optical imaging, the system aperture is the main factor limiting the imaging resolution. In the visible light band, especially in astronomical observation, increasing the system aperture is the main means to improve the spatial resolution. Due to the fact that the manufacturing cost of a large-aperture imaging system increases exponentially with the aperture, the idea of using a precisely positioned sub-aperture lens array to replace the traditional large-aperture lens for imaging is proposed, and the optical synthetic aperture imaging technology is developed.

[0003] The sub-aperture lens array that is pre-adjusted can accurately converge the transmitted light into an image on the target surface. Due to the satisfaction of the co-phase condition, the light beam self-interferes to form an interference image, thereby improving the actual imaging resolution. However, due to the fact that the actual light transmission area of the sub-aperture lens array is still small compared with a single large-aperture lens, and different arrangement modes will cause the response of the imaging system to change. In actual imaging, the point spread function of the optical synthetic aperture imaging system will be expanded, and the mid-frequency response of the amplitude modulation transfer function is often low, resulting in unavoidable blurring in direct imaging, which affects the imaging resolution. Moreover, due to the limited light transmission area of the sub-aperture lens array, the light in the imaging process is dark, and the camera often accompanies imaging noise in digital imaging.

[0004] For the classical image restoration problem of such incoherent imaging, traditional image restoration algorithms can play a certain role. The traditional algorithms include Wiener filter, Lucy-Richardson algorithm, blind deconvolution algorithm, constrained least squares algorithm, super-Laplacian prior algorithm, etc. However, these algorithms often need to know or estimate the point spread function of the imaging process, and the imaging noise will seriously affect the restoration quality, so the application range is limited in actual use. At present, there are also algorithms based on deep neural networks for image restoration, but these methods are based on supervised learning, and the cost of constructing the data set and training is high, and it is difficult to generalize when the target, lighting conditions, imaging system, etc. change.

[0005] Therefore, the shortcomings of the traditional image restoration algorithm are: 1. the point spread function is known or can be approximately estimated; 2. the influence of noise leads to poor restoration effect; 3. manual parameter adjustment is required, and the effect is unstable. The defects of the deep neural network restoration algorithm under supervised learning are: 1. high cost of data set construction; 2. low generalization ability; 3. the clear image label corresponding to the blurred image cannot be obtained. In view of the above problems, it is necessary to propose an image restoration method that meets the actual use scene and demand and has a lower cost. SUMMARY

[0006] The application provides a non-paired data supervised optimization synthetic aperture system image restoration method, which comprises a degraded target A obtained by system imaging and another clear target B. The clear target B is subjected to a calculation simulation process to obtain a similar degradation effect as the degraded target A. The degraded target A and the degraded target B are used as network inputs, the clear target B is used as a supervised label, and the parameters of the network are supervised and optimized by using regular constraints on the restored image, so that the image restoration of the degraded target in the synthetic aperture system imaging is finally realized. Compared with the existing algorithm, the noise resistance of the application is stronger, the point spread function of the known system and the artificial targeted parameter adjustment are not required, the corresponding clear target A is not required to be obtained for supervised optimization, only a single degraded target A and the easily obtained clear target B are required to realize the image restoration, and the application can be randomly generalized to other conditions and is suitable for the direct imaging scene of the optical synthetic aperture system.

[0007] The specific technical scheme of the application is:

[0008] A non-paired data supervised optimization synthetic aperture system image restoration method, characterized in that the method comprises the following steps:

[0009] S1. A single degraded target A image is obtained by using an optical synthetic aperture imaging system, and another clear target B image is obtained by other means to form non-paired data, wherein the clear target B should have a certain image information similarity with the degraded target A, and the image mutual information is used for measurement.

[0010] S2. A neural network model is established, which comprises network 1 for noise suppression and network 2 for image restoration.

[0011] S3. First, the optimization of network 1 is performed, random noise is added to the degraded target A, and the images before and after the noise is added are used to train network 1, so that network 1 learns the noise suppression ability.

[0012] S4. Then, the optimization of network 2 is performed, the degraded target B is obtained by processing the clear target B through a calculation simulation process, the degraded target A and the degraded target B are input into network 1 to obtain the denoised target A and the denoised target B after noise suppression, and then the denoised target A and the denoised target B are input into network 2 to obtain the restored target A and the restored target B. The clear target B and the restored target B are used for loss function constraint, and the restored target A is subjected to regularization constraint, so that the parameter optimization of network 2 is realized. After sufficient optimization, the final restored target A is obtained.

[0013] The other means of the step S1 include re-shooting or selecting from existing image data, and the cost is relatively low.

[0014] The network 1 and the network 2 of the step S2 adopt one of a U-shaped neural network, a full connection network, a full convolution network, and a Transformer network.

[0015] The random noise of the step S3 refers to Gaussian noise or Poisson noise similar to the noise distribution in the degraded target A.

[0016] The calculation simulation process of the step S4 refers to a calculation means capable of achieving similar imaging effects of the degraded target A, including calculation imaging by using a simulation system response or system imaging simulation by using optical simulation software such as Zemax.

[0017] The beneficial effects of the present application are that in the face of actual use requirements of an optical synthetic aperture imaging system, a non-paired data supervised optimization synthetic aperture system image restoration method is proposed. The method has stronger anti-noise ability than traditional image restoration algorithms, has better restoration effect under noise, and does not need to know the point spread function of the system and manually adjust the parameters. Compared with the deep neural network image restoration algorithm based on supervised learning, the corresponding clear target A is not needed to supervise and optimize, only a single degraded target A and a clear target B which can be easily obtained are needed to realize image restoration, and the method can be randomly generalized to other targets, systems and the like. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings described herein are used to provide further understanding of the embodiments of the present application, constitute a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings:

[0019] Figure 1 It is a flowchart of the non-paired data supervised optimization synthetic aperture system image restoration method in the embodiments of the present application;

[0020] Figure 2 It is a deep convolutional neural network structure schematic diagram of the network 1 and the network 2 in the embodiments of the present application;

[0021] Figure 3 It is a schematic diagram of image data flow and parameter closed-loop optimization in the embodiments of the present application;

[0022] Figure 4 It is a comparison schematic diagram of the results of the degraded target A after being restored by a non-paired data supervised optimization synthetic aperture system image restoration method constructed by the present application under different signal-to-noise ratio levels, a Wiener filter, a Lucy-Richardson algorithm, a blind deconvolution algorithm, a constrained least squares algorithm, a super Laplace prior algorithm, and the present application scheme;

[0023] Figure 3 In the above, S refers to an image transformation operation, including 0°, 90°, 180°, 270° rotation and horizontal and vertical flipping.

[0024] Figure 3 Middle: solid line refers to the flow of image data, and the dashed line refers to the optimization of parameters. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given below in combination with examples and drawings, the illustrative embodiments of the present application and the description thereof are only used to explain the present application, and do not limit the present application.

[0026] A synthetic aperture system image restoration method for unpaired data supervised optimization, characterized in that the method comprises the following steps:

[0027] S1. Using an optical synthetic aperture imaging system, obtaining a single degraded target A image, and obtaining another clear target B image by other means to form unpaired data, wherein the clear target B should have certain image information similarity with the degraded target A, and the image mutual information is used for measurement;

[0028] S2. Establishing a neural network model, which comprises network 1 for noise suppression and network 2 for image restoration;

[0029] S3. First, the optimization of network 1 is performed, random noise is added to the degraded target A, and the images before and after adding noise are used to train network 1, so that network 1 learns the noise suppression ability;

[0030] S4. Then, the optimization of network 2 is performed, the degraded target B is obtained by processing the clear target B through a simulation process, the degraded target A and the degraded target B are input into network 1 to obtain the denoised target A and the denoised target B after noise suppression, and then the denoised target A and the denoised target B are input into network 2 to obtain the restored target A and the restored target B, the loss function constraint is performed on the clear target B and the restored target B, and the regularization constraint is performed on the restored target A, so that the parameter optimization of network 2 is realized, and the final restored target A is obtained after sufficient optimization.

[0031] The other means of the step S1 include re-shooting or selecting from existing image data, characterized in that the implementation cost is low.

[0032] The network 1 and the network 2 of the step S2 adopt one of a U-shaped neural network, a fully connected network, a fully convolutional network and a Transformer network.

[0033] The random noise of the step S3 refers to Gaussian noise or Poisson noise similar to the noise distribution in the degraded target A.

[0034] The computational simulation process in step S4 refers to computational methods that can achieve similar imaging effects to the degraded target A, including computational imaging using simulated system response, or system imaging simulation using optical simulation software such as Zemax.

[0035] Example 1: The workflow of a synthetic aperture system image restoration method based on unpaired data-supervised optimization is as follows:

[0036] according to Figure 1 As described in step S1, a single image of the observed target is captured using a visible spectrum camera and an optical synthetic aperture imaging system. The resulting single blurred image is taken as the degraded target A. From the existing image data, a clear target B with similar image information to the degraded target A is easily obtained through image mutual information index selection, thereby constructing unpaired image data.

[0037] according to Figure 1 As described in step S2, a cascaded neural network model is established, which includes network 1 for noise suppression and network 2 for image restoration. Figure 2 As shown, both Network 1 and Network 2 are deep convolutional neural networks employing a hybrid of 1×1 and 3×3 kernel sizes. Network 2 utilizes a U-shaped network structure that combines downsampling and upsampling operations. Additionally, pixel-level addition operations, image channel-scale concatenation operations, and the LeakyReLU activation function are used to assist in network construction.

[0038] according to Figure 1 As described in step S3, network 1 is first optimized. Random Gaussian noise 1 is added to the degraded target A to obtain degraded target A1; then random Gaussian noise 2 is added to degraded target A1 to obtain degraded target A2. Degraded targets A1 and A2 are input into network 1 respectively, and the corresponding outputs are compared with degraded targets A and A1 to calculate the loss function. The parameters of network 1 are optimized using the gradient backpropagation algorithm, iterating for 300 steps. During the iteration process, the Gaussian noise 1 and 2 added before each iteration are randomly generated. The specific formula for calculating the loss function of network 1 is as follows:

[0039] Loss = MSE(Degradation Target A1, Degradation Target A) + MSE(Degradation Target A2, Degradation Target A1)

[0040] MSE(·) represents the mean square error value for two images.

[0041] according to Figure 1The optimization of network 2 is performed after the optimization of network 1 is completed, as described in step S4. The simulation of the incoherent imaging process is achieved by calculation, and the degraded target B with similar imaging effect to the degraded target A is calculated for the clear target B. The degraded target A and the degraded target B are input into network 1 together, and the denoised target A and the denoised target B are obtained, as shown in Figure 3 During the process, network 1 is in a test state and does not perform parameter optimization. The image rotation and flipping of the denoised target B are performed by using the image transformation operation, and the denoised target B and the denoised target A are input into network 2 together, and the recovered target A and the recovered target B are output. The loss function calculation is performed on the recovered target B and the clear target B, and the gradient regularization constraint is performed on the recovered target A, and the parameter optimization of network 2 is performed by using the gradient back propagation algorithm, and the iteration is performed for 2700 steps. During the iteration process, the image transformation operation performed on the denoised target B before each iteration is a random combination of rotation and flipping.

[0042] The calculation model of the incoherent imaging process is

[0043] The degraded target B is equal to the clear target B multiplied by the point spread function plus random noise.

[0044] The point spread function is an arbitrarily set convolution kernel, and is not fixed as the point spread function of the imaging system or an approximate estimation thereof, the * represents the convolution operation, and the random noise represents the simulated noise with similar distribution to the degraded target A, and is generally Gaussian noise.

[0045] The specific calculation formula of the loss function and the gradient regularization constraint of network 2 is

[0046]

[0047] wherein represents the absolute gradient matrix of the image a along the x and y directions, M and N represent the row and column numbers of the two-dimensional image, |·| represents the absolute value, and a represents the hyperparameter, which is 0.0005 here.

[0048] The software and hardware devices used in the method of the application are as follows: Windows 10 operating system, Intel(R) Core i7-10700K CPU, one Nvidia Geforce GTX 1080Ti, RAM 32.0GB, Python 3.7 language environment, and Pytorch 1.12.0 deep learning framework. The initial learning rate of network 1 and network 2 is 0.0002, and the learning rate is set to be reduced to 95% of the original value every 100 steps, and the optimizer is Adam.

[0049] To further verify the actual performance of the non-paired data supervised optimization synthetic aperture system image restoration method involved in the present application, we compared the imaging and restoration effects of the degraded target A under different signal-to-noise ratio conditions. The traditional image restoration algorithms such as Wiener filter, Lucy-Richardson algorithm, blind deconvolution algorithm, constrained least squares algorithm and super-Laplacian prior algorithm were used as comparison, and the comparison results are shown in Figure 4 As can be seen from Figure 4 When the signal-to-noise ratio is 60dB and 50dB, except that the contrast of the restored images of the Lucy-Richardson algorithm and the blind deconvolution algorithm changes, the other algorithms have good restoration effects. When the signal-to-noise ratio is increased to 30dB and 20dB, the traditional image restoration algorithms are severely affected by the strong unknown noise, and thus produce unacceptable artifacts. At this time, the non-paired data supervised optimization synthetic aperture system image restoration method involved in the present application still has good image restoration effect.

[0050] In summary, the non-paired data supervised optimization synthetic aperture system image restoration method involved in the present application can realize image restoration through computational imaging and constrained neural network optimization in the application scene of the optical synthetic aperture imaging system for single-frame imaging of the observed target, only needs a single degraded target A and an easily obtained clear target B to constitute non-paired image data, and has stable effect under different signal-to-noise ratio conditions. In the actual application scene, the supervised learning neural network algorithm which needs a large amount of labeled data and the traditional image restoration algorithm which needs system point spread function prior are invalid, while the method involved in the present application has good image restoration effect, low cost and is suitable for different systems and different observed target imaging.

Claims

1. A synthetic aperture system image reconstruction method for non-paired data supervised optimization, characterized in that The method comprises the following steps: S1. Obtain a single degraded target A image using an optical synthetic aperture imaging system, and obtain another clear target B image by other means to form unpaired data, wherein the clear target B should have certain image information similarity with the degraded target A, and image mutual information is used for measurement; S2. Establish a neural network model, which comprises network 1 for noise suppression and network 2 for image restoration; S3. First, optimize network 1, add random noise to the degraded target A, and use the images before and after adding noise to form data pairs to train network 1, so that network 1 learns the noise suppression ability; S4. Then, optimize network 2, obtain degraded target B by processing clear target B through a calculation simulation process, input degraded target A and degraded target B into network 1 to obtain denoised target A and denoised target B after noise suppression, and then input them into network 2 to obtain restored target A and restored target B, use clear target B and restored target B to constrain the loss function, and constrain the restored target A to realize the parameter optimization of network 2, and finally obtain the restored target A after sufficient optimization.

2. The non-paired data supervised optimized synthetic aperture system image reconstruction method according to claim 1, wherein, The other means of step S1 include re-shooting or selecting from existing image data, characterized by low implementation cost.

3. The non-paired data supervised optimized synthetic aperture system image reconstruction method of claim 1, wherein, Network 1 and network 2 of step S2 use one of U-shaped neural network, fully connected network, fully convolutional network and Transformer network.

4. The non-paired data supervised optimized synthetic aperture system image reconstruction method of claim 1, wherein, The random noise of step S3 refers to Gaussian noise or Poisson noise similar to the noise distribution in the degraded target A.

5. The non-paired data supervised optimized synthetic aperture system image reconstruction method of claim 1, wherein, The calculation simulation process of step S4 refers to a calculation means that can achieve similar imaging effect as the degraded target A, including using simulation system response for calculation imaging, or using optical simulation software such as Zemax for system imaging simulation.

Images