Pupil arrangement optimization method of optical sparse aperture imaging system under multiple constraints

Abstract

The invention discloses a pupil arrangement optimization method of an optical sparse aperture imaging system based on multiple constraints, and belongs to the technical field of optical synthetic aperture. The implementation method comprises the following steps: simulating an optical sparse aperture imaging process by using a point spread function (PSF), and making a remote sensing image fuzzy data set; sending the blurred image into an image restoration network to output a restored image, and performing image restoration based on a two-stage two-order restoration network TRN, so that the front-end OSA imaging system and the rear-end image restoration neural network are complementary; the filling factor of the optical sparse aperture system, the frequency domain signal-to-noise ratio and the high-pass enhancement mean square error of the restored image are used as indexes, pupil arrangement and image restoration network parameters are jointly optimized, and pupil arrangement giving consideration to the imaging quality and the system theoretical cut-off frequency and an image restoration network matched with the pupil arrangement are obtained. According to the trained two-stage two-order recovery network, the pupil arrangement optimization result of the optical sparse aperture imaging system under multiple constraints is obtained through prediction, and the imaging quality is remarkably improved.

Description

Technical Field

[0001] The present invention relates to a method for optimizing the pupil arrangement of an optical sparse aperture imaging system under multiple constraints, and belongs to the technical field of optical synthetic aperture. Background Art

[0002] High-resolution imaging plays a crucial role in space remote sensing and astronomical observations. Due to the limitation of the aperture diffraction effect, for a given optical wavelength, the larger the aperture of the optical system, the higher the limit angular resolution. However, the development of optical system manufacturing technology and space payload limitations restrict the size of the optical system. In order to break through the resolution limitation of single-aperture imaging, the optical sparse aperture (OSA) imaging technology has emerged. Without changing the manufacturing material of the imaging mirror, it uses a series of sub-aperture imaging systems with sparse arrangements and smaller apertures to collect object information at different frequencies, thereby achieving an imaging effect equivalent to that of a single large-aperture imaging system. It has the characteristics of high resolution, light system mass, relatively low manufacturing cost, and easy implementation of manufacturing processes. However, the OSA imaging technology also introduces new problems. Currently, the main key problems are: (1) Due to the discreteness of the aperture distribution of the OSA imaging system, its modulation transfer function drops significantly in the mid-low frequencies. Coupled with the reduction of the light-passing area and the decrease of the image signal-to-noise ratio, the image formed on the image plane becomes blurred and degraded, and image post-processing technology must be used to restore the image formed by the OSA imaging system. (2) Different sub-aperture arrangement methods determine the modulation ability of the OSA imaging system for object spatial frequency information. By adjusting the parameters of the sub-aperture system (the parameters of the aperture system refer to the position coordinates and aperture sizes of the sub-apertures), the frequency missing problem can be improved, and the acquisition of effective frequency information can be enhanced.

[0003] In recent years, most of the domestic and foreign scholars' research on pupil arrangement and image restoration has been carried out separately and independently. For example, the pupil arrangement design of the OSA imaging system usually imposes certain constraints on the sub-aperture parameters, and then designs one or several optimization functions directly related to the characteristic parameters of the OSA imaging system (such as resolution, modulation transfer function (MTF), etc.), and uses algorithms such as traversal algorithms, genetic algorithms, or gradient-based algorithms to find the sub-aperture parameters that make the optimization function reach an extreme value. With the proposal of the end-to-end design concept that connects the optical lens and the backend image processing algorithm in the field of optical design, researchers have also begun to consider combining the aperture arrangement of the OSA imaging system with subsequent image restoration for research, aiming to improve the quality of the final image restoration and optimize the parameters of the sub-apertures. Summary of the Invention

[0004] The object of the present invention is to provide an optimization method for the pupil arrangement of an optical sparse aperture imaging system under multiple constraints. Taking the filling factor of the optical sparse aperture system, the signal-to-noise ratio in the frequency domain, and the high-pass enhanced mean square error of the restored image as indicators, the pupil arrangement and the parameters of the image restoration network are jointly optimized to obtain a pupil arrangement that takes into account the imaging quality and the theoretical cut-off frequency of the system and a matching image restoration network. According to the prediction of the trained two-stage two-order restoration network, the optimization result of the pupil arrangement of the optical sparse aperture imaging system under multiple constraints is obtained, significantly improving the final imaging quality.

[0005] The object of the present invention is achieved by the following technical solutions.

[0006] The optimization method for the pupil arrangement of the optical sparse aperture imaging system under multiple constraints disclosed in the present invention includes the following steps.

[0007] Step 1: Use the point spread function PSF to simulate the optical sparse aperture imaging process and produce a remote sensing image blur dataset.

[0008] The expression of the point spread function PSF satisfies the condition of being differentiable with respect to the coordinates and aperture variables of the sub-apertures.

[0009] Step 1.1: Characterize the point spread function PSF of the OSA imaging system suitable for inverse optimization.

[0010] Establish a pupil coordinate system on the pupil plane. The amplitude transmittance function of a single aperture in the pupil coordinate system is described using the circle function as follows:

[0011]

[0012] where (x 0 , y 0 ) represents the center coordinates of the sub-aperture, (x, y) represents the coordinates of the pupil plane; d 0 represents the diameter of the sub-aperture; the pupil arrangement is quantitatively described by the number of sub-apertures k, the diameter d i of each sub-aperture, and the coordinates (x i , y i ) of the sub-aperture in the pupil coordinate system; the amplitude transmittance function of the OSA imaging system containing multiple apertures is:

[0013]

[0014] where i is the sub-aperture number, k is the total number of apertures, which is an integer and i ∈ [1, k], (x i , y i ) is the center coordinates of the i-th sub-aperture, and d i is the diameter of the i-th sub-aperture.

[0015] The pupil function of the OSA imaging system is:

[0016]

[0017] Among them, represents the phase delay of the system; j is the imaginary unit; in practical applications, the phase delay of the optical system will be corrected to a level that does not affect imaging, and under such conditions, it is determined as

[0018] The pupil function of the OSA imaging system is:

[0019] T(x,y) = P(x,y) (4)

[0020] The point spread function PSF of the OSA imaging system is the square of the modulus of the Fourier transform of the pupil function:

[0021]

[0022] Since the Fourier transform of the circle function is the Bessel function, the point spread function PSF of the OSA imaging system suitable for inverse optimization is finally expressed as:

[0023]

[0024] Among them, (x′,y′) are the coordinates on the image plane, λ is the working wavelength of the system, f is the focal length of the system; J 1 () represents the first-order Bessel function of the first kind; the partial derivative values of the PSF with respect to the sub-aperture coordinates (x i ,y i ) and the aperture diameter d i are calculated using Equation (4), so as to ensure the progress of inverse optimization;

[0025] Step 1.2: Use Equation (6) obtained in Step 1.1 to simulate the optical sparse aperture imaging process and produce a blurred remote sensing image;

[0026] The imaging process of the OSA system is expressed as:

[0027] g(x′,y′) = (I 0 *PSF)(x′,y′) + n(x′,y′) (7)

[0028] Among them, I 0 is the clear image, * represents convolution, n(x′,y′) represents noise, g(x′,y′) represents the blurred image formed by the OSA imaging system, and g(x′,y′) and I 0 constitute a clear-blurred image pair;

[0029] Step 2: The blurred image is fed into the image restoration network to output the restored image. Image restoration is performed based on the two-stage two-order restoration network TRN. That is, the blurred image as the input will be fed into the upper and lower layer networks in the TRN respectively. The processing result of the lower layer network will be fused into the upper layer network, and the output of the upper layer is the restored image; according to the restored image and d in Step 1 i 、(x i ,y i )、I 0 , calculate the value of the joint loss function;

[0030] Step 2.1: Take the blurred image made in Step 1 as the input image, and perform image restoration based on the two-stage two-order restoration network TRN. That is, the blurred image as the input will be fed into the upper and lower layer networks in the TRN respectively. The processing result of the lower layer network will be fused into the upper layer network, and the output of the upper layer is the restored image;

[0031] Step 2.2: According to d in Step 1 i 、(x i ,y i )、I 0 and the restored image obtained in Step 2.1, calculate the value of the joint loss function;

[0032] The joint loss function is as follows:

[0033] Loss=OSA_loss+Img_loss (8)

[0034] Loss represents the joint loss function. The decrease of OSA_loss indicates that the OSA imaging system parameters are approaching the direction of interest;

[0035] Perform Fourier transform on the blurred image:

[0036]

[0037] where I 0spectral (μ,ν) represents the object spectrum distribution, OTF(μ,ν) is the optical transfer function of the OSA imaging system, and N(μ,ν) represents the Gaussian white noise spectrum distribution;

[0038] Take the modulus of G(μ,ν) to obtain the spectrum distribution of the blurred image, ignoring the coordinates (μ,ν), as shown in Equation (10):

[0039]

[0040] where real represents taking the real part, imag represents taking the imaginary part, and MTF represents the modulation transfer function of the OSA imaging system;

[0041] According to Equation (10), define the spectrum signal-to-noise ratio SSNR as follows:

[0042]

[0043] Among them, MTF and OTF both represent the MTF and OTF of the current OSA imaging system; |I 0spectral | represents the object spectrum, which can be represented by the average spectrum of the training set images or the spectrum of a single training set image. The calculation process is shown in Equation (12):

[0044]

[0045] m is the total number of images included in the training set, I i represents an image in the training set, and further obtains the spectral signal-to-noise ratio loss function, as shown in Equation (13):

[0046] loss_SSNR = ∑∑(SSNR j -desire_SSNR) 2 (13)

[0047] OSA_loss consists of the fill factor of the OSA imaging system and the SSNR loss, as shown in Equation (14):

[0048]

[0049] Among them, ρ is the fill factor of the OSA system. The decrease of Img_loss in Equation (8) indicates that the restored image approaches the clear image, which is represented by the high-pass enhanced mean square error loss function HEMSE, as shown in Equation (15). represents high-pass filtering, represent the restored image and the clear image respectively, β 1 , β 2 represent the weighting coefficients of the high-pass filtering part and MSE respectively. MSE represents the mean square error. The decrease of the HEMSE value indicates that the restored image is getting closer and closer to the true value image;

[0050]

[0051] In summary, the joint loss function of Equation (8) changes to:

[0052]

[0053] In the formula, λ 1 , λ 2 , λ 3 are the weight factors of each loss function. Selecting the appropriate weights between the loss functions can improve the training efficiency;

[0054] Step 3: For the sub-aperture coordinates (x i , y i ) and the aperture diameter di Initialize to obtain the initial aperture arrangement; optimize d i and (x i , y i ), and the parameters in the two-stage two-step restoration network, that is, continuously iterate the processes of step one and step two until the combined loss no longer decreases significantly or reaches the predetermined number of training times; obtain the trained two-stage two-step restoration network after training; predict the optimized result of the pupil arrangement of the optical sparse aperture imaging system under multiple constraints according to the trained two-stage two-step restoration network.

[0055] Preferably, step one, step two, and step three are implemented based on the pytorch deep learning framework and carried out on a single GTX4090 GPU; during the training process, the Adam optimizer is selected to adaptively adjust the learning step size to achieve the best training effect. The initial learning rate of the sub-aperture parameters is set to 0.0025, and the initial learning rate of the neural network parameters is set to 0.0001; the NWPU-RESISC45 remote sensing image dataset is used during the training and testing processes. This dataset contains 45 scenarios with a total of 31,500 images. Randomly select 1,200 remote sensing image pictures containing 24 scenarios as the training set, and the testing is a generalization test on the entire NWPU-RESISC45 dataset; the input and output image sizes are both 256×256 pixels; the batch size is 10 images. Traversing 1,200 training sets is regarded as one training process, and repeat this process until the number of training times reaches 1,000 times, that is, complete the joint optimization design of step one, step two, and step three.

[0056] Preferably, in step three, the method for optimizing d i and (x i , y i ) and the parameters in the two-stage two-step restoration network is as follows:

[0057] The parameters d i and (x i , y i ) of the sub-aperture are continuously optimized, and the parameter v of the two-stage two-step restoration network is also continuously optimized. The parameters d i and (x i , y i ) of the sub-aperture and the parameters of the two-stage two-step restoration network are optimized in the direction of making the combined loss function value reach the minimum, that is:

[0058]

[0059] where x * , y * , d * , v *It represents the sub-aperture parameters and the parameters of the image restoration neural network after 1000 times of training.

[0060] Preferably, it further includes Step Four: quantitatively evaluating the result obtained in Step Three:

[0061] According to the sub-aperture parameters obtained in Step Three, give the sub-aperture arrangement of the OSA system, simulate the optical sparse aperture imaging process under this arrangement, and obtain a blurred image; use the two-stage two-order restoration network corresponding to this aperture arrangement to perform image restoration and obtain a clear image; on the one hand, evaluate the sub-aperture arrangement of the OSA system and calculate its MTF and the theoretical cut-off frequency of the optical system; on the other hand, evaluate the image restoration result, and use the two indicators of peak signal-to-noise ratio PSNR and structural similarity SSIM in the quantitative evaluation indicators; calculate the average SSIM and PSNR of the restored images on the entire NWPU-RESISC45 dataset, and perform imaging restoration tests on the checkerboard images; verify the optimized result of the pupil arrangement of the optical sparse aperture imaging system under multiple constraints from multiple dimensions including the evaluation result of the sub-aperture arrangement of the OSA system, the image restoration evaluation result, and the imaging restoration test; according to the optimized result of the pupil arrangement of the optimal optical sparse aperture imaging system, the theoretical cut-off frequency of the optimized OSA system reaches the cut-off frequency of the equivalent full aperture, and the details of the restoration effect on the checkerboard image are very clear, and the imaging quality is significantly improved.

[0062] Beneficial effects:

[0063] 1. An optimization method for the pupil arrangement of an optical sparse aperture imaging system based on multiple constraints disclosed by the present invention uses the filling factor of the optical sparse aperture system, the signal-to-noise ratio in the frequency domain, and the high-pass enhanced mean square error of the restored image as indicators to jointly optimize the pupil arrangement and the parameters of the image restoration network, and obtains a pupil arrangement that takes into account the imaging quality and the theoretical cut-off frequency of the system and a matching image restoration network. According to the trained two-stage two-order restoration network, the optimized result of the pupil arrangement of the optical sparse aperture imaging system under multiple constraints is predicted, so as to significantly improve the final imaging quality.

[0064] 2. An optimization method for the pupil arrangement of an optical sparse aperture imaging system based on multiple constraints disclosed by the present invention has the following advantages compared with the traditional separate design and the current joint optimization design technology:

[0065] ① Compared with the traditional separate design, the method proposed by the present invention optimizes the parameters of the image restoration neural network while optimizing the pupil arrangement parameters, making the front-end OSA imaging system and the back-end image restoration neural network complementary, reducing design redundancy, and significantly improving the imaging quality.

[0066] ②The current joint optimization design technology cannot take into account both the imaging quality and the system's theoretical cut-off frequency. In contrast, the present invention innovatively proposes a joint loss function composed of a spectral signal-to-noise ratio (SSNR) loss function, a high-pass enhanced mean square error loss function (HEMSE), and a fill factor loss function, which realizes the joint optimization of the pupil arrangement of a sparse aperture imaging system and an image restoration network under multiple constraints. On the premise of meeting the technical indicators of the equivalent full-aperture cut-off frequency, the quality of the final image restoration is significantly improved, and the optimization results of the pupil arrangement of the optical sparse aperture imaging system under multiple constraints are evaluated and verified from multiple dimensions such as the average PSNR, SSIM index, MTF distribution, cut-off frequency of the OSA system, and the imaging restoration effect on the checkerboard pattern; according to the optimal optimization results of the pupil arrangement of the optical sparse aperture imaging system, the theoretical cut-off frequency of the optimized OSA system reaches the cut-off frequency of the equivalent full aperture, and the details of the restoration effect on the checkerboard pattern are very clear, and the imaging quality is significantly improved. Description of the Drawings

[0067] Figure 1 is the Golay-9 pupil arrangement;

[0068] Figure 2 is the imaging process and effect of the Golay-9 imaging system;

[0069] Figure 3 is the network structure of TNR;

[0070] Figure 4 is the schematic diagram of the SSNR distribution. Among them, Figure (a) is the SSNR distribution of the equivalent full aperture, and Figure (b) is the SSNR distribution of the OSA system;

[0071] Figure 5 is the flow chart of the joint optimization of the aperture arrangement and the image restoration neural network;

[0072] Figure 6 is the diagram of the joint optimization design result, Figure 6 (a) is the equivalent full aperture and MTF, Figure 6 (b) is the initial Golay-9 arrangement and MTF distribution, Figure 6 (c) is the arrangement and MTF distribution after the joint optimization design, Figure 6 (d) is the MTF curves of the equivalent full aperture, the initial structure, and the joint optimization design arrangement in the meridional direction;

[0073] Figure 7 is the comparison diagram of the restoration effect of the joint optimization design result on the checkerboard pattern. Detailed Implementation Manner

[0074] To better illustrate the purpose and advantages of the present invention, the following further describes the content of the invention in conjunction with the drawings and examples.

[0075] Example 1:

[0076] As Figure 5 shown, the method for optimizing the pupil arrangement of the optical sparse aperture imaging system based on multiple constraints disclosed in this embodiment is specifically implemented as follows:

[0077] Step 1: Use the point spread function PSF to simulate the optical sparse aperture imaging process and produce a remote sensing image blur dataset;

[0078] The expression of the point spread function PSF must be differentiable with respect to the coordinates and aperture variables of the sub-apertures;

[0079] Step 1.1: Characterize the point spread function PSF of the OSA imaging system suitable for inverse optimization

[0080] Establish a pupil coordinate system on the pupil plane. The amplitude transmittance function of a single aperture in the pupil coordinate system is described using the circle function as follows:

[0081]

[0082] where (x 0 , y 0 ) represents the center coordinates of the sub-aperture, (x, y) represents the coordinates of the pupil plane; d 0 represents the diameter of the sub-aperture; the pupil arrangement is quantitatively described by the number of sub-apertures k, the diameter d i of each sub-aperture, and the coordinates (x i , y i ) of the sub-aperture in the pupil coordinate system; the amplitude transmittance function of the OSA imaging system including multiple apertures is:

[0083]

[0084] where i is the sub-aperture number, k is the total number of apertures, which is an integer and i ∈ [1, k], (x i , y i ) are the center coordinates of the i-th sub-aperture, d i is the diameter of the i-th sub-aperture. In this embodiment, the total number of sub-apertures is 9, that is, k = 9;

[0085] The pupil function of the OSA imaging system is:

[0086]

[0087] where, represents the phase delay of the system; j is the imaginary unit; in practical applications, the phase delay of the optical system is usually corrected to a level that does not affect imaging. Under this condition, it is determined that

[0088] The pupil function of the OSA imaging system is as follows:

[0089] T(x,y) = P(x,y) (4)

[0090] The pupil function of Golay-9, that is, the aperture arrangement is as Figure 1 shown. The white part is the pupil that allows light to pass through, and its amplitude transmittance is 1. The black part does not allow light to pass through, and the amplitude transmittance is 0. The point spread function PSF of the OSA imaging system is the square of the modulus of the Fourier transform of the pupil function:

[0091]

[0092] where F represents the Fourier transform. Since the Fourier transform of the circle function is the Bessel function, the point spread function PSF of the OSA imaging system suitable for reverse optimization is finally expressed as:

[0093]

[0094] where (x′, y′) are the coordinates on the image plane, λ is the working wavelength of the system, f is the focal length of the system; J 1 () represents the first-order Bessel function of the first kind; using Equation (6), the partial derivative values of the PSF with respect to the sub-aperture coordinates (x i , y i ) and the aperture diameter d i can be calculated, so as to ensure that the reverse optimization can be carried out;

[0095] Step 1.2: Use Equation (6) obtained in Step 1.1 to simulate the optical sparse aperture imaging process and produce a blurred remote sensing image;

[0096] The imaging process of the OSA system is expressed as the convolution of the original image and the system PSF in the spatial domain plus noise:

[0097] g(x′, y′) = (I 0 *PSF)(x′, y′) + n(x′, y′) (7)

[0098] where, I 0 is the clear image, * represents convolution, n(x′, y′) represents noise, g(x′, y′) represents the blurred image formed by the OSA imaging system, and g(x′, y′) and I 0 constitute a clear-blurred image pair, as Figure 2 shown. The clear image is simulated to obtain the blurred image passing through the Golay-9 system through the calculation process of Equation (7), and the two constitute a clear-blurred image pair.

[0099] Step 2: The blurred image is fed into the image restoration network to output the restored image. According to d in Step 1 i , (x i , y i ), I 0 and the restored image obtained in Step 2.1, calculate the value of the joint loss function;

[0100] Step 2.1: Use the blurred image produced in Step 1 as the input image and perform image restoration based on the two-stage restoration network (TRN). The network structure of TRN is as Figure 3 shown. The network is divided into upper and lower layers. The main body of its lower layer is an encoder-decoder neural network, called the encoder-decoder stage, and the upper layer is a pixel-level information processing neural network, which can avoid the loss of image information caused by the downsampling process, called the pixel-to-pixel stage. The encoder-decoder layer at the bottom is based on the residual U-Net network and introduces a Channel Attention Block (CAB) and a Supervised Attention Module (SAM). The downsampling and upsampling operations of the encoder-decoder layer enable the network to have the ability to extract and process image features on a larger scale, ensuring accurate capture of large-scale target information in the image (such as the overall shape of an airplane). The Channel Attention Block can give different weights to the features on the basis of the image features extracted by the convolution operation, adjust the influence of different feature information on the restoration result, and enable the network to retain more important feature information. The upper layer of the network is the pixel-to-pixel layer, which uses convolution operations and an Original Resolution Block (ORB). ORB consists of several CABs. The entire upper-layer network does not reduce or increase the size of the feature map, which avoids the loss of image detail information during the downsampling process and enables the network to focus more on processing the detail ability of the image. The upper-layer network can well compensate for the loss of details and boundaries in the lower-layer network. SAM is introduced between the two layers. The feature map calculated by the encoder-decoder layer will be transmitted to different parts of the upper-layer network under the supervision of the ground truth image, realizing the information fusion between the two layers. The convolution layer of the network uses Leaky ReLU as the activation function to avoid the situation where the gradient is zero during the reverse calculation process of the network. The blurred image is used as the input and will be fed into the upper and lower layer networks in TRN respectively. The processing result of the lower-layer network will be fused into the upper-layer network, and the output of the upper layer is the restored image;

[0101] Step 2.2: According to d in Step 1 i , (x i , yi )、I 0 And the restored image obtained in step 2.1, calculate the value of the joint loss function. The loss function is used to guide the update direction of the OSA imaging system parameters and network parameters. Therefore, it is necessary to propose a loss function that can not only reflect the restored imaging quality but also characterize the frequency characteristics of the OSA imaging system. The joint loss function is as follows:

[0102] Loss = OSA_loss + Img_loss (8)

[0103] Loss represents the joint loss function. The reduction of OSA_loss indicates that the OSA imaging system parameters are approaching the direction of interest. Considering the actual engineering needs, it is hoped that when the OSA imaging system meets the cut-off frequency of the equivalent full-aperture system, neither its sub-aperture diameter nor the filling factor is too large. According to the OSA imaging system model in equation (7), the quality of the final imaging not only depends on the PSF of the OSA imaging system, but also on the object spectrum and noise level. The present invention innovatively proposes a spectral signal-to-noise ratio (Spectral SNR, SSNR) loss function, which is described in detail below.

[0104] According to the imaging model of the OSA imaging system, the Fourier transform of the degraded image consists of three parts, as shown in equation (9):

[0105] G(μ, ν) = F{g(x′, y′)}

[0106] = F{I 0 (x, y)*PFS(x′, y′) + n(x′, y′)}

[0107] = I 0spectral (μ, ν)·OTF(μ, ν) + N(μ, ν) (9)

[0108] where I 0spectral (μ, ν) represents the object spectrum distribution, OTF(μ, ν) is the optical transfer function of the OSA imaging system, and N(μ, ν) represents the Gaussian white noise spectrum distribution. Taking the modulus of G(μ, ν) gives the spectrum distribution of the blurred image, ignoring the coordinates (μ, ν), as shown in equation (10):

[0109]

[0110] where real represents taking the real part, imag represents taking the imaginary part, and MTF represents the modulation transfer function of the OSA imaging system. From equation (10), it can be seen that the spectrum distribution of the final degraded image is not only related to the MTF of the OSA imaging system, but also related to the object spectrum and noise spectrum. Define the spectral signal-to-noise ratio SSNR as follows:

[0111]

[0112] where MTF and OTF both represent the MTF and OTF of the current OSA imaging system; |I 0spectral | represents the object spectrum, which can be represented by the average spectrum of the training set images or the spectrum of a single training set image. The calculation process is shown in Equation (12):

[0113]

[0114] m is the total number of images included in the training set, I i represents an image in the training set. The SSNR of the system is similar to the MTF distribution. Due to the introduction of noise and the object spectrum, SSNR better reflects the modulation effect on different frequency information of the training set compared to MTF. As Figure 4 shown, Figure 4 (a) is the SSNR distribution of the equivalent full aperture, Figure 4 (b) is the SSNR distribution of the OSA system. Obviously, the influence brought by noise cannot exceed the information itself, that is, when SSNR > 0 dB, the corresponding frequency information of the degraded image is effective frequency information. Based on this criterion, using the defined SSNR to give the spectral signal-to-noise ratio loss function, and further obtaining the spectral signal-to-noise ratio loss function, as shown in Equation (13), and further obtaining the spectral signal-to-noise ratio loss function, as shown in Equation (13):

[0115] loss_SSNR = ∑∑(SSNR j -desire_SSNR) 2 (13)

[0116] Therefore, OSA_loss consists of the fill factor of the OSA imaging system and the SSNR loss, as shown in Equation (14):

[0117]

[0118] where ρ is the fill factor of the OSA system. In this example, the fill factor of the OSA system is 13.5%, that is, ρ = 13.5%. The reduction of Img_loss in Equation (8) indicates that the final restored image approaches the clear image. Since the high-frequency information in the data set accounts for a small proportion and the mid-low frequency information accounts for a large proportion, the high-pass enhanced mean square error loss function (High-pass Enhanced Mean Square Error, HEMSE) proposed in the present invention, as shown in Equation (15), represents high-pass filtering, represent the restored image and the clear image respectively, β 1 , β 2respectively represent the weighting coefficients of the high-pass filtering part and MSE, where MSE represents the mean square error. A decreasing HEMSE value indicates that the restored image is getting closer to the true image. In this example, β 1 = 9, β 2 = 1;

[0119]

[0120] In summary, the change in the joint loss function of Equation (8) is as follows:

[0121]

[0122] where λ 1 , λ 2 , λ 3 are the weight factors of each loss function. Selecting appropriate weights between loss functions can improve the training efficiency. In this example, λ 1 = 1, λ 2 = 0.1, λ 3 = 1;

[0123] Step 3: Initialize d i and (x i , y i ) to obtain the initial Golay-9 aperture arrangement. Based on the pytorch deep learning framework, use the gradient descent method to optimize d i and (x i , y i ) and the parameters in the two-stage two-step restoration network, that is, continuously loop through the processes of Step 1 and Step 2 until the joint loss no longer decreases significantly or reaches a certain number of training times; after training, the optimized result of the aperture arrangement and the trained two-stage two-step restoration network can be finally obtained;

[0124] The joint optimization design framework of the aperture arrangement and the image restoration neural network is as Figure 5 shown. First, initialize the sub-aperture parameters. In this example, the classic Golay-9 aperture arrangement is used. Then, as described in Step 1, simulate the OSA system PSF applicable to reverse optimization according to the initial sub-aperture parameters, convolve it with the small batch of clear images in the dataset to obtain the blurred image. Then, as described in Step 2, use the simulated blurred image as the input image and send it into the image restoration neural network for image restoration to obtain the restored image. Then, calculate the value of the joint loss function according to Equation (16). Finally, the backpropagation process of this framework is shown by the red arrows in the figure. Based on the pytorch deep learning framework, automatically calculate the gradient values of the loss function with respect to the sub-aperture parameters and the image restoration neural network parameters, and optimize the sub-aperture parameters and the image restoration network parameters based on the gradient descent method. The joint optimization design is a process of continuously looping through Step 1 and Step 2, and the parameter d of the sub-aperturei and (x i , y i ) are continuously optimized, and the parameters v of the two-stage restoration network are also continuously optimized. The two are optimized in the direction of minimizing the value of the joint loss function until the joint loss no longer decreases significantly or reaches a certain number of training times. That is:

[0125]

[0126] After training, the optimized result of the aperture arrangement and the image restoration neural network can be finally obtained. Among them, x * , y * , d * , v * represent the sub-aperture parameters and the image restoration neural network parameters after training is completed.

[0127] The entire joint optimization design process is implemented using the pytorch deep learning framework on a single GTX4090 GPU. During the training process, the Adam optimizer is selected to adaptively adjust the learning step size to achieve the best training effect. The initial learning rate of the sub-aperture parameters is set to 0.0025, and the initial learning rate of the neural network parameters is set to 0.0001. The NWPU-RESISC45 remote sensing image dataset is used during the training and testing processes. This dataset contains 45 scenarios with a total of 31,500 images. Randomly select 1,200 remote sensing image pictures containing 24 scenarios as the training set, and the testing is carried out on the entire NWPU-RESISC45 dataset for generalization testing. The sizes of the input and output pictures are both 256×256 pixels. The batch size is 10 pictures, and traversing 1,200 training sets is regarded as one training process. Repeat this process until the number of training times reaches 1,000 times, that is, complete the joint optimization design of steps one, two, and three.

[0128] Step four: quantitatively evaluate the results obtained in step three:

[0129] The joint optimization design in Step 3 yielded the sub-aperture parameters and the corresponding image restoration neural network. Based on the sub-aperture parameters, the sub-aperture arrangement of the optimized OSA system was given, and the optical sparse aperture imaging process under this arrangement was simulated to obtain a blurred image. The TNR restoration network corresponding to this aperture arrangement was used for image restoration to obtain a clear image. On the one hand, the sub-aperture arrangement of the OSA system was evaluated by calculating its MTF and the theoretical cut-off frequency of the optical system. On the other hand, the image restoration results were evaluated, and two indicators, Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM), were used in the quantitative evaluation indicators. The average SSIM and PSNR of the restored images on the entire NWPU-RESISC45 dataset were calculated, and the imaging restoration test was carried out on the checkerboard image to evaluate and verify the optimization results of the pupil arrangement of the optical sparse aperture imaging system under multiple constraints from multiple dimensions.

[0130] The average SSIM and PSNR indicators on the entire NWPU-RESISC45 dataset are shown in Table 1. It can be seen that compared with the direct imaging of OSA, the imaging indicators have increased significantly. The MTF distribution of the optimized OSA and the imaging effect of the checkerboard pattern are as Figure 6 、 Figure 7 shown, Figure 6 (a) is the equivalent full aperture and MTF, Figure 6 (b) is the initial Golay-9 arrangement and MTF distribution, Figure 6 (c) is the arrangement and MTF distribution after the joint optimization design, Figure 6 (d) is the MTF curves of the equivalent full aperture, the initial structure, and the arrangement after the joint optimization design in the meridional direction. The blue curve is the result of the joint optimization design. It can be seen that after the joint optimization, the theoretical cut-off frequency of the OSA system reaches the cut-off frequency of the equivalent full aperture. Figure 7 The original image of the checkerboard and the restoration effect of the joint optimization design on the checkerboard pattern are shown. The details of the restored image are very clear, verifying the effectiveness of the present invention.

[0131] Table 1

[0132]

[0133] The above specific description further elaborates on the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A pupil arrangement optimization method for an optical sparse aperture imaging system under multiple constraints, characterized in that: The following steps are included: Step 1: Use the point spread function PSF to simulate the optical sparse aperture imaging process and produce a remote sensing image blur dataset; The expression of the point spread function PSF satisfies the condition that the coordinates and aperture variables of the sub-aperture are differentiable; Step 1.1, characterize the point spread function (PSF) of the OSA imaging system suitable for inverse optimization; A pupil coordinate system is established on the pupil plane. The amplitude transmittance function of a single aperture in the pupil coordinate system is described using the circle function as follows: Where (x0, y0) represents the center coordinates of the subaperture, (x, y) represents the coordinates of the pupil plane; d0 represents the diameter of the subaperture; the pupil is arranged through the number of subapertures k and the diameter of each subaperture d i and the coordinates of the subaperture in the pupil coordinate system (x i ,y i ) is quantitatively described; the amplitude transmittance function of the OSA imaging system containing multiple apertures is: Where i is the subaperture number, k is the total number of apertures, which is an integer and i∈[1,k], (x i ,y i ) is the center coordinate of the i-th subaperture, d i is the diameter of the i-th sub-aperture; The pupil function of the OSA imaging system is: in, represents the phase delay of the system; j is an imaginary unit; in practical applications, the phase delay of the optical system will be corrected to a level that does not affect the imaging. Under this condition, it is determined to be The pupil function of the OSA imaging system is: T(x,y)=P(x,y) (4) The point spread function (PSF) of the OSA imaging system is the Fourier transform of the pupil function and the square of the modulus: Since the Fourier transform of the circle function is a Bessel function, the point spread function PSF of the OSA imaging system suitable for inverse optimization is finally expressed as: Where (x′, y′) is the coordinate on the image plane, λ is the operating wavelength of the system, and f is the focal length of the system; J1() represents the first-order Bessel function of the first kind; the PSF subaperture coordinate (x i ,y i ) and aperture diameter d i The partial derivative value of , thus ensuring that the reverse optimization can be carried out; Step 1.2: Use step 1.1 to obtain equation (6), simulate the optical sparse aperture imaging process, and produce a blurred image of the remote sensing image; The imaging process of the OSA system is expressed as: g(x′,y′)=(I0*PSF)(x′,y′)+n(x′,y′) (7) Where I0 is the clear image, * represents convolution, n(x′, y′) represents noise, g(x′, y′) represents the blurred image formed by the OSA imaging system, and g(x′, y′) and I0 form a clear-blurred image pair; Step 2: The blurred image is sent to the image restoration network to output the restored image. Image restoration is performed based on the two-stage two-stage restoration network TRN. That is, the blurred image is sent to the upper and lower layers of the TRN as input, and the processing results of the lower layer network are fused into the upper layer network. The output of the upper layer is the restored image. According to the restored image and the d i 、(x i ,y i ), I0, calculate the value of the joint loss function; Step 3: Sub-aperture coordinates (x i ,y i ) and aperture diameter d i Initialize to get the initial aperture arrangement; based on the gradient descent method, i and (x i ,y i ) and the parameters in the two-stage two-order restoration network, that is, continuously iterating the process of step one and step two until the joint loss decreases no longer significantly or reaches a predetermined number of training times; after training, a trained two-stage two-order restoration network is obtained; and according to the trained two-stage two-order restoration network, the pupil arrangement optimization result of the optical sparse aperture imaging system under multiple constraints is predicted.

2. The method according to claim 1, characterized in that: The implementation method of step 2 is: Step 2.1: Use the blurred image produced in step 1 as the input image, and perform image restoration based on the two-stage two-order restoration network TRN. That is, the blurred image as input will be sent to the upper and lower layers of the TRN respectively, and the processing results of the lower layer will be fused into the upper layer, and the output of the upper layer is the restored image. Step 2.2: According to step 1, i 、(x i ,y i ), I0 and the restored image obtained in step 2.1, calculate the value of the joint loss function; The joint loss function is as follows: Loss = OSA_loss + Img_loss (8) Loss represents the joint loss function, where the decrease of OSA_loss indicates that the OSA imaging system parameters are approaching the direction of interest; Take the Fourier transform of the blurred image: Among them I 0spectral (μ, ν) represents the object spectrum distribution, OTF(μ, ν) is the optical transfer function of the OSA imaging system, and N(μ, ν) represents the Gaussian white noise spectrum distribution; The spectral distribution of the blurred image is obtained by taking the modulus of G(μ,ν) and ignoring the coordinates (μ,ν), as shown in formula (10): Where real means taking the real part, imag means taking the imaginary part, and MTF means the modulation transfer function of the OSA imaging system; According to formula (10), the spectrum signal-to-noise ratio SSNR is defined as follows: Where MTF and OTF represent the MTF and OTF of the current OSA imaging system; |I 0spectral | represents the object spectrum, and the calculation process is shown in formula (12): m is the total number of images in the training set, I i represents a picture in the training set, and further obtains the spectrum signal-to-noise ratio loss function, as shown in formula (13): loss_SSNR=ΣΣ(SSNR j -desire_SSNR) 2 (13) OSA_loss is composed of the filling factor and SSNR loss of the OSA imaging system, as shown in equation (14): Where ρ is the filling factor of the OSA system. The decrease of Img_loss in equation (8) indicates that the restored image is close to the clear image. It is represented by the high-pass enhanced mean square error loss function HEMSE, as shown in equation (15). H{·} represents high-pass filtering. I0 represents the restored image and the clear image, β1 and β2 represent the weighting coefficients of the high-pass filter and MSE, respectively. MSE represents the mean square error. The decrease of HEMSE value indicates that the restored image is getting closer and closer to the true image. In summary, the joint loss function of formula (8) changes to: In the formula, λ1, λ2, λ3 are the weight factors of each loss function. Selecting appropriate weights between loss functions can improve training efficiency.

3. The method according to claim 1 or 2, characterized in that: Steps 1, 2 and 3 are implemented based on the pytorch deep learning framework and are performed on a single GTX 4090 GPU. During the training process, the Adam optimizer is selected to adaptively adjust the learning step size to achieve the best training effect. The initial learning rate of the subaperture parameter is set to 0.0025, and the initial learning rate of the neural network parameter is set to 0.0001. The NWPU-RESISC45 remote sensing image dataset is used during training and testing, which contains 45 scenes with a total of 31,500 images. 1,200 remote sensing image images containing 24 scenes are randomly selected as the training set, and the test is a generalization test on the entire NWPU-RESISC45 dataset. The input and output image sizes are both 256×256 pixels. The batch size is 10 images, and traversing 1,200 training sets is regarded as a training process. This process is repeated until the number of training times reaches 1,000, completing the joint optimization design of steps 1, 2 and 3.

4. The method according to claim 2, characterized in that: In step 3, based on the gradient descent method, i and (x i ,y i ) and the method for optimizing the parameters in the two-stage two-stage restoration network is: Subaperture parameter d i and (x i ,y i ) is continuously optimized, the parameter v of the two-stage two-stage restoration network is also continuously optimized, the parameter v of the two-stage two-stage restoration network is also continuously optimized, and the parameter d of the sub-aperture i and (x i ,y i ) and the parameters of the two-stage two-stage restoration network are optimized in the direction of minimizing the value of the joint loss function, namely: where x * ,y * ,d * ,v * Represents the sub-aperture parameters and image restoration neural network parameters after training 1000 times.

5. The method according to claim 1, 2 or 4, characterized in that: The method also includes step 4: according to the sub-aperture parameters obtained in step 3, the sub-aperture arrangement of the OSA system is given, and the optical sparse aperture imaging process under the arrangement is simulated to obtain a blur map; the image is restored using a two-stage two-stage restoration network corresponding to the aperture arrangement to obtain a clear image; on the one hand, the sub-aperture arrangement of the OSA system is evaluated, and its MTF and the theoretical cutoff frequency of the optical system are calculated; on the other hand, the image restoration result is evaluated, and the peak signal-to-noise ratio PSNR and the structural similarity SSIM in the quantitative evaluation index are used; the entire NWP is calculated. The average SSIM and PSNR of the restored images on the U-RESISC45 dataset are calculated, and an imaging restoration test is performed on the chessboard image. The pupil arrangement optimization results of the optical sparse aperture imaging system under multiple constraints are verified in multiple dimensions based on the sub-aperture arrangement evaluation results, image restoration evaluation results, and imaging restoration tests of the OSA system. According to the optimal pupil arrangement optimization results of the optical sparse aperture imaging system, the theoretical cutoff frequency of the optimized OSA system reaches the cutoff frequency equivalent to the full-aperture, the details of the restoration effect on the chessboard image are very clear, and the imaging quality is significantly improved.

Images