Optimization Method for Pupil Arrangement in Optical Sparse Aperture Imaging Systems under Multiple Constraints

Abstract

This invention relates to a method for optimizing the pupil arrangement of an optical sparse aperture imaging system under multiple constraints, belonging to the field of optical synthetic aperture technology. The method involves: simulating the optical sparse aperture imaging process using a point spread function (PSF) to create a blurred dataset of remote sensing images; feeding the blurred images into an image restoration network to output restored images; performing image restoration based on a two-stage, two-order restoration network (TRN) to complement the front-end OSA imaging system and the back-end image restoration neural network; using the fill factor of the optical sparse aperture system, the frequency domain signal-to-noise ratio, and the high-pass enhancement mean square error of the restored image as indicators, jointly optimizing the pupil arrangement and image restoration network parameters to obtain a pupil arrangement that balances imaging quality and the system's theoretical cutoff frequency, along with a matching image restoration network; and predicting the optimized pupil arrangement of the optical sparse aperture imaging system under multiple constraints based on the trained two-stage, two-order restoration network, significantly improving imaging quality.

Description

Technical Field

[0001] This invention relates to a method for optimizing the pupil arrangement of an optical sparse aperture imaging system under multiple constraints, belonging to the field of optical synthetic aperture technology. Background Technology

[0002] High-resolution imaging plays a crucial role in space remote sensing and astronomical observation. Due to the limitations of aperture diffraction, for a given wavelength of light, the larger the aperture of an optical system, the higher the limiting angular resolution. However, the development of optical system manufacturing technology and space payload constraints limit the size of optical systems. To overcome the resolution limitations of single-aperture imaging, optical sparse aperture (OSA) imaging technology has emerged. Without changing the manufacturing material of the imaging mirror, it utilizes a series of sparsely arranged, small-aperture sub-aperture imaging systems to collect information about objects at different frequencies, thereby achieving an imaging effect with resolution equivalent to a single large-aperture imaging system. It features high resolution, lightweight system, relatively low manufacturing cost, and easy manufacturing processes. However, OSA imaging technology has also introduced new problems. Currently, its main key problems are: (1) Due to the discreteness of the aperture distribution of the OSA imaging system, its modulation transfer function drops significantly in the low and medium frequencies. In addition, the reduced light-transmitting area and the decrease in image signal-to-noise ratio lead to blurring and degradation of the image on the image plane. Image post-processing techniques are required to restore the image formed by the OSA imaging system. (2) Different sub-aperture arrangements determine the modulation capability of the OSA imaging system for the spatial frequency information of the object. By adjusting the parameters of the sub-aperture system (the parameters of the aperture system refer to the position coordinates and aperture size of the sub-apertures), the frequency missing problem can be improved and the acquisition of effective frequency information can be enhanced.

[0003] In recent years, research on pupil arrangement and image restoration by scholars both domestically and internationally has mostly been conducted separately. For example, the pupil arrangement design of OSA imaging systems typically involves imposing certain constraints on sub-aperture parameters, and then designing one or more optimization functions directly related to OSA imaging system characteristic parameters (such as resolution and modulation transfer function, MTF). Sub-aperture parameters that maximize the optimization function are then found using traversal algorithms, genetic algorithms, or gradient-based algorithms. However, with the emergence of end-to-end design concepts that connect optical lenses with backend image processing algorithms in the field of optical design, researchers have begun to consider combining OSA imaging system aperture arrangement with subsequent image restoration, aiming to improve the quality of the final image restoration by optimizing the sub-aperture parameters. Summary of the Invention

[0004] The purpose of this invention is to provide a method for optimizing the pupil arrangement of an optical sparse aperture imaging system under multiple constraints. Using the fill factor, frequency domain signal-to-noise ratio, and high-pass enhancement mean square error of the restored image as indicators, the method jointly optimizes the pupil arrangement and image restoration network parameters to obtain a pupil arrangement that balances imaging quality and the system's theoretical cutoff frequency, along with a matching image restoration network. Based on the trained two-stage, two-order restoration network, the optimized pupil arrangement result of the optical sparse aperture imaging system under multiple constraints is predicted, significantly improving the final imaging quality.

[0005] The objective of this invention is achieved through the following technical solution.

[0006] The present invention discloses a method for optimizing the pupil arrangement of an optical sparse aperture imaging system under multiple constraints, comprising the following steps.

[0007] Step 1: Use the point spread function (PSF) to simulate the optical sparse aperture imaging process and create a blurred dataset of remote sensing images;

[0008] The expression for the point diffusion function (PSF) satisfies the condition that the coordinates and aperture variables of the sub-aperture are differentiable.

[0009] Step 1.1: Characterize the point spread function (PSF) of the OSA imaging system suitable for inverse optimization;

[0010] Establish a pupil coordinate system on the pupil plane. The amplitude transmittance function of a single aperture in the pupil coordinate system is described using the circle function as follows:

[0011]

[0012] Where (x0, y0) represents the center coordinates of the sub-aperture, (x, y) represents the coordinates of the pupil surface; d0 represents the diameter of the sub-aperture; the pupil arrangement passes through the number of sub-apertures k and the diameter d of each sub-aperture. i and the coordinates of the sub-aperture in the pupil coordinate system (x i ,y i Quantitative description is performed; the amplitude transmittance function of an OSA imaging system with multiple apertures is:

[0013]

[0014] Where i is the sub-aperture number, k is the total number of apertures, which is an integer and i∈[1,k], (x i ,y i ) represents the center coordinates of the i-th sub-aperture, d i Let be the diameter of the i-th sub-aperture;

[0015] The pupil function of the OSA imaging system is:

[0016]

[0017] in, This represents the phase delay of the system; j is the imaginary unit; in practical applications, the phase delay of an optical system is corrected to a level that does not affect imaging, and under this condition, it is determined to be...

[0018] The pupil function of the OSA imaging system is:

[0019] T(x,y)=P(x,y) (4)

[0020] The point spread function (PSF) of an OSA imaging system is the square of the modulus of the Fourier transform of the pupil function:

[0021]

[0022] Since the Fourier transform of the circle function is a Bessel function, the final point spread function (PSF) of the OSA imaging system suitable for inverse optimization is expressed as:

[0023]

[0024] Where (x′, y′) are the coordinates on the image plane, λ is the operating wavelength of the system, and f is the focal length of the system; J1() represents the first-order Bessel function of the first kind; the PSF pair aperture coordinates (x′, y′) are calculated using equation (4). i ,y i and aperture diameter d i The partial derivative values ​​are used to ensure that inverse optimization can be performed;

[0025] Step 1.2: Using Equation (6) obtained in Step 1.1, simulate the optical sparse aperture imaging process and create a blurred image of the remote sensing image;

[0026] The OSA system imaging process is represented as follows:

[0027] g(x′,y′)=(I0*PSF)(x′,y′)+n(x′,y′) (7)

[0028] Where I0 is the sharp image, * represents convolution, n(x′,y′) represents noise, and g(x′,y′) represents the blurred image generated by the OSA imaging system. g(x′,y′) and I0 form a sharp-blurred image pair.

[0029] Step 2: The blurred image is fed into the image restoration network, which outputs the restored image. Image restoration is performed based on a two-stage, two-order restoration network (TRN). The blurred image is fed into both the upper and lower layers of the TRN network. The processing results from the lower layer are fused into the upper layer, and the output of the upper layer is the restored image. Based on the restored image and the d from Step 1... i 、(x i ,y i ), I0, calculate the value of the joint loss function;

[0030] Step 2.1: Take the blurred image created in Step 1 as the input image and perform image restoration based on the two-stage two-order restoration network TRN. That is, the blurred image as input will be fed into the upper and lower layers of the TRN network respectively. The processing result of the lower layer network will be fused into the upper layer network, and the output of the upper layer is the restored image.

[0031] Step 2.2, according to d in Step 1 i 、(x i ,y i ), I0 and the restored image obtained in step 2.1, and calculate the value of the joint loss function;

[0032] The joint loss function is as follows:

[0033] Loss = OSA_loss + Img_loss (8)

[0034] Loss represents the joint loss function, where a decrease in OSA_loss indicates that the OSA imaging system parameters are moving closer to the direction of interest;

[0035] Perform Fourier transform on a blurred image:

[0036]

[0037] Where I 0spectral (μ,ν) represents the object's spectral distribution, OTF(μ,ν) is the optical transfer function of the OSA imaging system, and N(μ,ν) represents the Gaussian white noise spectral distribution;

[0038] The spectral distribution of the blurred image is obtained by taking the modulus of G(μ,ν), ignoring the coordinates (μ,ν), as shown in Equation (10):

[0039]

[0040] Where real represents taking the real part, imag represents taking the imaginary part, and MTF represents the modulation transfer function of the OSA imaging system;

[0041] Based on equation (10), the spectral signal-to-noise ratio (SSNR) is defined as follows:

[0042]

[0043] Where MTF and OTF both represent the MTF and OTF of the current OSA imaging system; |I 0spectral | represents the spectrum of an object, which can be represented by the average spectrum of the training set images or by the spectrum of a single training set image. The calculation process is shown in equation (12):

[0044]

[0045] m is the total number of images in the training set, I i Let represent an image in the training set. Further, the spectral signal-to-noise ratio loss function is obtained, as shown in Equation (13):

[0046] loss_SSNR=∑∑(SSNR j -desire_SSNR) 2 (13)

[0047] OSA_loss consists of the fill factor and SSNR loss of the OSA imaging system, as shown in Equation (14):

[0048]

[0049] Where ρ is the fill factor of the OSA system, the decrease of Img_loss in equation (8) indicates that the restored image is closer to the clear image, which is represented by the high-pass enhancement mean square error loss function HEMSE, as shown in equation (15). Indicates high-pass filtering. β1 and β2 represent the restored image and the clear image, respectively. β1 and β2 represent the weighting coefficients of the high-pass filter and MSE, respectively. MSE represents the mean square error. A decrease in the HEMSE value indicates that the restored image is getting closer and closer to the true image.

[0050]

[0051] In summary, the joint loss function of equation (8) changes as follows:

[0052]

[0053] In the formula, λ1, λ2, and λ3 are the weight factors of each loss function. Selecting appropriate weights between loss functions can improve training efficiency.

[0054] Step 3: Pair the aperture coordinates (x) i ,y i and aperture diameter d i Initialization is performed to obtain the initial aperture arrangement; based on the gradient d method, the aperture arrangement is then determined. i and (x) i ,yi The parameters in the two-stage two-order restoration network are optimized by iterating through steps one and two until the joint loss decreases no longer significantly or the predetermined number of training iterations is reached. After training, a trained two-stage two-order restoration network is obtained. Based on the trained two-stage two-order restoration network, the optimized pupil arrangement of the optical sparse aperture imaging system under multiple constraints is predicted.

[0055] Preferably, steps one, two, and three are implemented using the PyTorch deep learning framework on a single GTX4090 GPU. During training, the Adam optimizer is selected to adaptively adjust the learning step size to achieve the best training effect. The initial learning rate for the sub-aperture parameter is set to 0.0025, and the initial learning rate for the neural network parameter is set to 0.0001. The NWPU-RESISC45 remote sensing image dataset is used for both training and testing. This dataset contains 45 scenes and a total of 31,500 images. 1,200 remote sensing images containing 24 scenes are randomly selected from this dataset as the training set. The testing is performed on the entire NWPU-RESISC45 dataset to test generalization. The input and output image sizes are both 256×256 pixels. The batch size is 10 images. Traversing 1,200 images in the training set is considered as one training cycle. This process is repeated until the number of training iterations reaches 1,000, thus completing the joint optimization design of steps one, two, and three.

[0056] As a preferred option, step three involves using the gradient descent method to analyze d. i and (x) i ,y i The method for optimizing the parameters in the two-stage, two-order restoration network is as follows:

[0057] The parameter d of the sub-aperture i and (x) i ,y i With continuous optimization, the parameters v of the two-stage, two-order restoration network are also continuously optimized, as are the parameters d of the sub-aperture. i and (x) i ,y i The parameters of the two-stage, two-order restoration network are optimized in the direction of minimizing the joint loss function, i.e.:

[0058]

[0059] Where x * ,y * ,d * ,v * This represents the sub-aperture parameters and image restoration neural network parameters after 1000 training iterations.

[0060] As a preferred option, step four is also included: quantitatively evaluating the results obtained in step three.

[0061] Based on the sub-aperture parameters obtained in step three, the sub-aperture arrangement of the OSA system is given, and the optical sparse aperture imaging process under this arrangement is simulated to obtain a blurred image. The image is then restored using the two-stage, two-order restoration network corresponding to this aperture arrangement, resulting in a clear image. On one hand, the sub-aperture arrangement of the OSA system is evaluated, and its MTF and the theoretical cutoff frequency of the optical system are calculated. On the other hand, the image restoration results are evaluated, with the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) among the quantitative evaluation indicators. The average SSIM and PSNR of the restored images on the entire NWPU-RESISC45 dataset are calculated, and an imaging restoration test is performed on a chessboard image. Based on the evaluation results of the OSA system's sub-aperture arrangement, the image restoration evaluation results, and the imaging restoration test, the pupil arrangement optimization results of the optical sparse aperture imaging system under multiple constraints are verified in multiple dimensions. According to the optimal pupil arrangement optimization results of the optical sparse aperture imaging system, the theoretical cutoff frequency of the optimized OSA system reaches the equivalent full-aperture cutoff frequency, and the details of the restored effect on the chessboard image are very clear, significantly improving the imaging quality.

[0062] Beneficial effects:

[0063] 1. This invention discloses a pupil arrangement optimization method for an optical sparse aperture imaging system based on multiple constraints. The method uses the fill factor, frequency domain signal-to-noise ratio, and high-pass enhancement mean square error of the restored image of the optical sparse aperture system as indicators to jointly optimize the pupil arrangement and image restoration network parameters. This results in a pupil arrangement that balances imaging quality and the theoretical cutoff frequency of the system, along with a matching image restoration network. The optimized pupil arrangement of the optical sparse aperture imaging system under multiple constraints is predicted based on the trained two-stage two-order restoration network, thereby significantly improving the final imaging quality.

[0064] 2. The pupil arrangement optimization method for an optical sparse aperture imaging system based on multiple constraints disclosed in this invention has the following advantages compared with traditional separate design and current joint optimization design techniques:

[0065] ① Compared with the traditional separate design, the method proposed in this invention optimizes the pupil arrangement parameters and the image restoration neural network parameters at the same time, so that the front-end OSA imaging system and the back-end image restoration neural network complement each other, reduce design redundancy, and significantly improve imaging quality.

[0066] ② Current joint optimization design techniques cannot simultaneously achieve both imaging quality and the theoretical cutoff frequency of the system. In contrast, this invention innovatively proposes a joint loss function composed of the spectral signal-to-noise ratio (SSNR) loss function, the high-pass enhancement mean square error loss function (HEMSE) loss function, and the fill factor loss function. Under multiple constraints, this function achieves joint optimization of the pupil arrangement and image restoration network of the sparse aperture imaging system. While meeting the technical specification of the equivalent full-aperture cutoff frequency, it significantly improves the quality of the final image restoration. Furthermore, the optimization results of the pupil arrangement of the optical sparse aperture imaging system under multiple constraints are evaluated and verified from multiple dimensions, including the average PSNR, SSIM index, MTF distribution of the OSA system, cutoff frequency, and imaging restoration effect on the checkerboard pattern. According to the optimal optimization results of the pupil arrangement of the optical sparse aperture imaging system, the theoretical cutoff frequency of the optimized OSA system reaches the equivalent full-aperture cutoff frequency, and the details of the restoration effect on the checkerboard pattern are very clear, resulting in a significant improvement in imaging quality. Attached Figure Description

[0067] Figure 1 For Golay-9 pupil arrangement;

[0068] Figure 2 The imaging process and effects of the Golay-9 imaging system;

[0069] Figure 3 The network structure is for TNR;

[0070] Figure 4 Figure 1 shows a schematic diagram of SSNR distribution, where Figure (a) shows the SSNR distribution of the equivalent full aperture, and Figure (b) shows the SSNR distribution of the OSA system.

[0071] Figure 5 Flowchart for joint optimization of aperture arrangement and image restoration neural network;

[0072] Figure 6 To jointly optimize the design results diagram, Figure 6 (a) is the equivalent full caliber and MTF. Figure 6 (b) shows the initial Golay-9 layout and MTF distribution. Figure 6 (c) shows the layout and MTF distribution after joint optimization design. Figure 6 (d) is the MTF curve of the equivalent full caliber, initial structure, and joint optimization design arrangement in the meridional direction;

[0073] Figure 7 A comparison of the restoration effect of the joint optimization design results on the chessboard diagram. Detailed Implementation

[0074] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.

[0075] Example 1:

[0076] like Figure 5 As shown in the figure, the pupil arrangement optimization method for an optical sparse aperture imaging system based on multiple constraints disclosed in this embodiment is implemented in the following specific steps:

[0077] Step 1: Use the point spread function (PSF) to simulate the optical sparse aperture imaging process and create a blurred dataset of remote sensing images;

[0078] The expression for the point diffusion function (PSF) must be differentiable with respect to the coordinates of the sub-aperture and the aperture variable.

[0079] Step 1.1: Characterize the point spread function (PSF) of the OSA imaging system suitable for inverse optimization.

[0080] Establish a pupil coordinate system on the pupil plane. The amplitude transmittance function of a single aperture in the pupil coordinate system is described using the circle function as follows:

[0081]

[0082] Where (x0, y0) represents the center coordinates of the sub-aperture, (x, y) represents the coordinates of the pupil surface; d0 represents the diameter of the sub-aperture; the pupil arrangement passes through the number of sub-apertures k and the diameter d of each sub-aperture. i and the coordinates of the sub-aperture in the pupil coordinate system (x i ,y i Quantitative description is performed; the amplitude transmittance function of an OSA imaging system with multiple apertures is:

[0083]

[0084] Where i is the sub-aperture number, k is the total number of apertures, which is an integer and i∈[1,k], (x i ,y i ) represents the center coordinates of the i-th sub-aperture, d i Let k be the diameter of the i-th sub-aperture. In this embodiment, the total number of sub-apertures is 9, i.e., k = 9.

[0085] The pupil function of the OSA imaging system is:

[0086]

[0087] in, This represents the phase delay of the system; j is the imaginary unit; in practical applications, the phase delay of an optical system is usually corrected to a level that does not affect imaging, under which condition is determined...

[0088] The pupil function of the OSA imaging system is:

[0089] T(x,y)=P(x,y) (4)

[0090] Golay-9's pupil function, i.e., aperture arrangement, is as follows: Figure 1 As shown, the white portion is the light-transmitting pupil with an amplitude transmittance of 1, while the black portion is opaque with an amplitude transmittance of 0. The point spread function (PSF) of the OSA imaging system is the square of the Fourier transform of the pupil function:

[0091]

[0092] Where F represents the Fourier transform, and since the Fourier transform of the circle function is a Bessel function, the point spread function PSF of the OSA imaging system, which is suitable for inverse optimization, is expressed as:

[0093]

[0094] Where (x′, y′) are the coordinates on the image plane, λ is the operating wavelength of the system, and f is the focal length of the system; J1() represents the first-order Bessel function of the first kind; the PSF pair aperture coordinates (x′, y′) can be calculated using equation (6). i ,y i and aperture diameter d i The partial derivative values ​​are used to ensure that inverse optimization can be performed;

[0095] Step 1.2: Using Equation (6) obtained in Step 1.1, simulate the optical sparse aperture imaging process and create a blurred image of the remote sensing image;

[0096] The OSA system imaging process can be represented as a spatial domain convolution of the original image and the system PSF, with noise added:

[0097] g(x′,y′)=(I0*PSF)(x′,y′)+n(x′,y′) (7)

[0098] Where I0 is the sharp image, * represents convolution, n(x′,y′) represents noise, and g(x′,y′) represents the blurred image generated by the OSA imaging system. g(x′,y′) and I0 form a sharp-blurred image pair, as shown below. Figure 2 As shown, the clear image is obtained by the calculation process of Equation (7), and the blurred image after passing through the Golay-9 system is obtained by simulation. The two form a clear-blurred image pair.

[0099] Step 2: The blurred image is fed into the image restoration network, which outputs the restored image, based on the d from Step 1. i 、(x i ,y i), I0 and the restored image obtained in step 2.1, and calculate the value of the joint loss function;

[0100] Step 2.1: Using the blurred image created in Step 1 as the input image, image restoration is performed based on a two-stage restoration network (TRN). The TRN network structure is as follows: Figure 3 As shown, the network consists of two layers. The lower layer is a main encoder-decoder neural network, called the encoder-decoder stage. The upper layer is a pixel-level information processing neural network, which avoids the loss of image information caused by downsampling and is called the pixel-to-pixel stage. The lower encoder-decoder stage is based on the residual U-Net network and introduces a Channel Attention Block (CAB) and a Supervised Attention Module (SAM). The downsampling and upsampling operations of the encoder-decoder stage enable the network to extract and process image features at a larger scale, ensuring accurate capture of large-scale target information (such as the overall shape of an aircraft). The Channel Attention Block assigns different weights to the image features extracted by convolutional operations, adjusting the influence of different feature information on the restoration result, allowing the network to retain more important feature information. The upper layer of the network is the pixel-to-pixel stage, which uses convolutional operations and an Original Resolution Block (ORB). ORB consists of several CABs. The entire upper-layer network does not shrink or enlarge the feature map size, avoiding the loss of image detail during downsampling and allowing the network to focus more on processing image details. The upper-layer network effectively compensates for the loss of details and boundaries in the lower-layer network. SAM is introduced between the two layers; the feature maps computed by the encoder-decoder layers are passed to different parts of the upper-layer network under the supervision of the ground truth image, achieving information fusion between the two layers. The network's convolutional layers use Leaky ReLU as the activation function to avoid zero gradients during backward computation. The blurred image is fed into both the upper and lower layers of the TRN network. The processing results of the lower layer are fused into the upper layer, and the output of the upper layer is the restored image.

[0101] Step 2.2, according to d in Step 1 i 、(x i ,y iThe joint loss function is calculated using the restored image obtained from step 2.1 and I0. This loss function guides the update direction of the OSA imaging system parameters and network parameters; therefore, a loss function that reflects both the restored image quality and the frequency characteristics of the OSA imaging system needs to be proposed. The joint loss function is as follows:

[0102] Loss = OSA_loss + Img_loss (8)

[0103] Loss represents the joint loss function, where a decrease in OSA_loss indicates that the OSA imaging system parameters are moving closer to the direction of interest. Considering practical engineering needs, it is desirable that the sub-aperture diameter and fill factor of the OSA imaging system are not too large while satisfying the cutoff frequency of the equivalent full-aperture system. As can be seen from the OSA imaging system model in Equation (7), the final imaging quality depends not only on the PSF of the OSA imaging system, but also on the object spectrum and noise level. This invention innovatively proposes a Spectral Signal-to-Noise Ratio (SSNR) loss function, which is detailed below.

[0104] According to the OSA imaging system model, the Fourier transform of the degraded image consists of three parts, as shown in equation (9):

[0105] G(μ,ν)=F{g(x′,y′)}

[0106] =F{I0(x,y)*PFS(x′,y′)+n(x′,y′)}

[0107] =I 0spectral (μ,ν)·OTF(μ,ν)+N(μ,ν) (9)

[0108] Where I 0spectral (μ,ν) represents the object's spectral distribution, OTF(μ,ν) is the optical transfer function of the OSA imaging system, and N(μ,ν) represents the Gaussian white noise spectral distribution. Taking the modulus of G(μ,ν) yields the spectral distribution of the blurred image, ignoring the coordinates (μ,ν), as shown in equation (10):

[0109]

[0110] Where real represents the real part, imag represents the imaginary part, and MTF represents the modulation transfer function of the OSA imaging system. From equation (10), it can be seen that the spectral distribution of the final degraded image is not only related to the OSA imaging system MTF, but also to the object spectrum and noise spectrum. The spectral signal-to-noise ratio (SSNR) is defined as follows:

[0111]

[0112] Where MTF and OTF both represent the MTF and OTF of the current OSA imaging system; |I 0spectral | represents the spectrum of an object, which can be represented by the average spectrum of the training set images or by the spectrum of a single training set image. The calculation process is shown in equation (12):

[0113]

[0114] m is the total number of images in the training set, I i This represents an image from the training set. The system's SSNR distribution is similar to its MTF distribution; however, due to the introduction of noise and object spectra, SSNR better reflects the modulation effect on different frequency information of the training set compared to MTF. For example... Figure 4 As shown, Figure 4 (a) is the equivalent full-caliber SSNR distribution. Figure 4 (b) shows the SSNR distribution of the OSA system. Clearly, the impact of noise cannot exceed the information itself; that is, when SSNR > 0 dB, the corresponding frequency information of the degraded image is the effective frequency information. Based on this criterion, the spectral signal-to-noise ratio loss function is given using the defined SSNR, and further obtained as shown in equation (13):

[0115] loss_SSNR=∑∑(SSNR j -desire_SSNR) 2 (13)

[0116] Therefore, OSA_loss consists of the fill factor and SSNR loss of the OSA imaging system, as shown in Equation (14):

[0117]

[0118] Where ρ is the fill factor of the OSA system, and in this example, the fill factor of the OSA system is 13.5%, i.e., ρ = 13.5%. The decrease in Img_loss in equation (8) indicates that the final restored image is closer to the clear image. Since the high-frequency information in the dataset accounts for a small proportion and the mid-to-low-frequency information accounts for a large proportion, the high-pass enhanced mean square error loss function (HEMSE) proposed in this invention is shown in equation (15). Indicates high-pass filtering. Let HEMSE represent the restored image and the clear image, respectively. β1 and β2 represent the weighting coefficients of the high-pass filter and MSE, respectively. MSE represents the mean square error. A decrease in the HEMSE value indicates that the restored image is getting closer and closer to the true image. In this example, β1 = 9 and β2 = 1.

[0119]

[0120] In summary, the joint loss function of equation (8) changes as follows:

[0121]

[0122] In the formula, λ1, λ2, and λ3 are the weight factors of each loss function. Selecting appropriate weights between loss functions can improve training efficiency. In this example, λ1 = 1, λ2 = 0.1, and λ3 = 1.

[0123] Step 3: For d i and (x) i ,y i Initialization was performed to obtain the initial Golay-9 aperture arrangement. Then, gradient descent was used based on the PyTorch deep learning framework to process the aperture arrangement. i and (x) i ,y i The parameters in the two-stage two-order restoration network are optimized by continuously iterating the first and second steps until the joint loss decreases no longer significantly or a certain number of training iterations are reached. After training, the optimized aperture arrangement and the trained two-stage two-order restoration network can be obtained.

[0124] A framework for joint optimization design of aperture arrangement and image restoration neural network, such as Figure 5 As shown. First, the sub-aperture parameters are initialized. In this example, the classic Golay-9 aperture arrangement is used. Next, according to step one, the OSA system PSF suitable for inverse optimization is simulated based on the initial sub-aperture parameters. It is convolved with a small batch of clear images in the dataset to obtain a blurred image. Then, according to step two, the simulated blurred image is used as the input image and fed into the image restoration neural network for image restoration to obtain the restored image. Then, the value of the joint loss function is calculated according to equation (16). Finally, the backpropagation process of this framework is shown by the red arrow in the figure. Based on the PyTorch deep learning framework, the gradient values ​​of the loss function with respect to the sub-aperture parameters and the image restoration neural network parameters are automatically calculated. The sub-aperture parameters and the image restoration network parameters are optimized based on the gradient descent method. The joint optimization design is a process of continuously iterating between step one and step two. The sub-aperture parameter d i and (x) i ,y i The parameters v of the two-stage, two-order restoration network are continuously optimized, both working towards minimizing the joint loss function, until the decrease in joint loss becomes no longer significant or a certain number of training iterations are reached. That is:

[0125]

[0126] After training, the optimized aperture arrangement and image restoration neural network can be obtained. Where x * ,y * ,d * ,v * This represents the sub-aperture parameters and image restoration neural network parameters after training is complete.

[0127] The entire joint optimization design process was implemented using the PyTorch deep learning framework on a single GTX4090 GPU. During training, the Adam optimizer was selected to adaptively adjust the learning step size to achieve the best training effect. The initial learning rate for the sub-aperture parameter was set to 0.0025, and the initial learning rate for the neural network parameter was set to 0.0001. The NWPU-RESISC45 remote sensing image dataset was used for both training and testing. This dataset contains 45 scenes and a total of 31,500 images. 1,200 remote sensing images containing 24 scenes were randomly selected from this dataset as the training set. The generalization test was performed on the entire NWPU-RESISC45 dataset. The input and output image size were both 256×256 pixels. The batch size was 10 images. Iterating through 1,200 images in the training set was considered as one training cycle. This process was repeated until the training iterations reached 1,000, thus completing the joint optimization design steps one, two, and three.

[0128] Step 4: Quantitatively evaluate the results obtained in Step 3:

[0129] Step 3, the joint optimization design, yielded the sub-aperture parameters and the corresponding image restoration neural network. Based on the sub-aperture parameters, the optimized sub-aperture arrangement of the OSA system was proposed, and the optical sparse aperture imaging process under this arrangement was simulated to obtain a blurred image. The TNR restoration network corresponding to this aperture arrangement was used to restore the image, resulting in a clear image. On the one hand, the sub-aperture arrangement of the OSA system was evaluated, and its MTF and the theoretical cutoff frequency of the optical system were calculated. On the other hand, the image restoration results were evaluated, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used as quantitative evaluation indicators. The average SSIM and PSNR of the restored images on the entire NWPU-RESISC45 dataset were calculated, and imaging restoration tests were performed on chessboard images to evaluate and verify the pupil arrangement optimization results of the optical sparse aperture imaging system under multiple constraints from multiple dimensions.

[0130] The average SSIM and PSNR metrics across the entire NWPU-RESISC45 dataset are shown in Table 1. It is evident that the imaging metrics are significantly improved compared to direct OSA imaging. The MTF distribution and checkerboard pattern imaging effect of the optimized OSA are shown in Table 1. Figure 6 , Figure 7 As shown, Figure 6 (a) is the equivalent full caliber and MTF. Figure 6 (b) shows the initial Golay-9 layout and MTF distribution. Figure 6 (c) shows the layout and MTF distribution after joint optimization design. Figure 6 (d) shows the MTF curves of the equivalent full caliber, initial structure, and joint optimization design arranged along the meridional direction. The blue curve represents the result of the joint optimization design. It can be seen that after joint optimization, the theoretical cutoff frequency of the OSA system reaches the cutoff frequency of the equivalent full caliber. Figure 7 The original image of the chessboard is shown, along with the restoration effect of the joint optimization design on the chessboard. The details of the restored image are very clear, verifying the effectiveness of the invention.

[0131] Table 1

[0132]

[0133] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing the pupil arrangement of an optical sparse aperture imaging system under multiple constraints, characterized in that: The process includes the following steps: Step 1: Use the point spread function (PSF) to simulate the optical sparse aperture imaging process and create a blurred dataset of remote sensing images. The expression for the point diffusion function (PSF) satisfies the condition that the coordinates and aperture variables of the sub-aperture are differentiable. Step 1.1: Characterize the point spread function (PSF) of the OSA imaging system suitable for inverse optimization; Establish a pupil coordinate system on the pupil plane. The amplitude transmittance function of a single aperture in the pupil coordinate system is described using the circle function as follows: Where (x0, y0) represents the center coordinates of the sub-aperture, (x, y) represents the coordinates of the pupil surface; d0 represents the diameter of the sub-aperture; the pupil arrangement passes through the number of sub-apertures k and the diameter d of each sub-aperture. i and the coordinates of the sub-aperture in the pupil coordinate system (x i ,y i Quantitative description is performed; the amplitude transmittance function of an OSA imaging system with multiple apertures is: Where i is the sub-aperture number, k is the total number of apertures, which is an integer and i∈[1,k], (x i ,y i ) represents the center coordinates of the i-th sub-aperture, d i Let be the diameter of the i-th sub-aperture; The pupil function of the OSA imaging system is: in, This represents the phase delay of the system; j is the imaginary unit; in practical applications, the phase delay of an optical system is corrected to a level that does not affect imaging, and under this condition, it is determined to be... The pupil function of the OSA imaging system is: T(x,y)=P(x,y) (4) The point spread function (PSF) of an OSA imaging system is the square of the modulus of the Fourier transform of the pupil function: Since the Fourier transform of the circle function is a Bessel function, the final point spread function (PSF) of the OSA imaging system suitable for inverse optimization is expressed as: Where (x′, y′) are the coordinates on the image plane, λ is the operating wavelength of the system, and f is the focal length of the system; J1() represents the first-order Bessel function of the first kind; the PSF pair aperture coordinates (x′, y′) are calculated using equation (4). i ,y i and aperture diameter d i The partial derivative values ​​are used to ensure that inverse optimization can be performed; Step 1.2: Using Equation (6) obtained in Step 1.1, simulate the optical sparse aperture imaging process and create a blurred image of the remote sensing image; The OSA system imaging process is represented as follows: g(x′,y′)=(I0*PSF)(x′,y′)+n(x′,y′) (7) Where I0 is the sharp image, * represents convolution, n(x′,y′) represents noise, and g(x′,y′) represents the blurred image generated by the OSA imaging system. g(x′,y′) and I0 form a sharp-blurred image pair. Step 2: The blurred image is fed into the image restoration network, which outputs the restored image. Image restoration is performed based on a two-stage, two-order restoration network (TRN). The blurred image is fed into both the upper and lower layers of the TRN network. The processing results from the lower layer are fused into the upper layer, and the output of the upper layer is the restored image. Based on the restored image and the d from Step 1... i 、(x i ,y i ), I0, calculate the value of the joint loss function; The second step is implemented as follows: Step 2.1: Take the blurred image created in Step 1 as the input image and perform image restoration based on the two-stage two-order restoration network TRN. That is, the blurred image as input will be fed into the upper and lower layers of the TRN network respectively. The processing result of the lower layer network will be fused into the upper layer network, and the output of the upper layer is the restored image. Step 2.2, based on d from Step 1 i 、(x i ,y i ), I0 and the restored image obtained in step 2.1, and calculate the value of the joint loss function; The joint loss function is as follows: Loss = OSA_loss + Img_loss (8) Loss represents the joint loss function, where a decrease in OSA_loss indicates that the OSA imaging system parameters are moving closer to the direction of interest; Perform Fourier transform on a blurred image: Where I 0spectral (μ,ν) represents the object's spectral distribution, OTF(μ,ν) is the optical transfer function of the OSA imaging system, and N(μ,ν) represents the Gaussian white noise spectral distribution; The spectral distribution of the blurred image is obtained by taking the modulus of G(μ,ν), ignoring the coordinates (μ,ν), as shown in Equation (10): Where real represents taking the real part, imag represents taking the imaginary part, and MTF represents the modulation transfer function of the OSA imaging system; Based on equation (10), the spectral signal-to-noise ratio (SSNR) is defined as follows: Where MTF and OTF both represent the MTF and OTF of the current OSA imaging system; |I 0spectral | represents the object spectrum, and the calculation process is shown in equation (12): m is the total number of images in the training set, I i Let represent an image in the training set. Further, the spectral signal-to-noise ratio loss function is obtained, as shown in Equation (13): loss_SSNR=∑∑(SSNR j -desire_SSNR) 2 (13) OSA_loss consists of the fill factor and SSNR loss of the OSA imaging system, as shown in Equation (14): Where ρ is the fill factor of the OSA system, the decrease of Img_loss in equation (8) indicates that the restored image is closer to the clear image, which is represented by the high-pass enhancement mean square error loss function HEMSE, as shown in equation (15). Indicates high-pass filtering. I0 represents the restored image and the clear image, respectively; β1 and β2 represent the weighting coefficients of the high-pass filter and MSE, respectively; MSE represents the mean square error; a decrease in the HEMSE value indicates that the restored image is getting closer and closer to the true image. In summary, the joint loss function of equation (8) changes as follows: In the formula, λ1, λ2, and λ3 are the weight factors of each loss function. Selecting appropriate weights between loss functions can improve training efficiency. Step 3: Pair the aperture coordinates (x) i ,y i and aperture diameter d i Initialization is performed to obtain the initial aperture arrangement; based on the gradient d method, the aperture arrangement is then determined. i and (x) i ,y i The parameters in the two-stage two-order restoration network are optimized by iterating through steps one and two until the joint loss decreases no longer significantly or the predetermined number of training iterations is reached. After training, a trained two-stage two-order restoration network is obtained. Based on the trained two-stage two-order restoration network, the optimized pupil arrangement of the optical sparse aperture imaging system under multiple constraints is predicted.

2. The method as described in claim 1, characterized in that: Steps one, two, and three are implemented using the PyTorch deep learning framework on a single GTX 4090 GPU. During training, the Adam optimizer is selected to adaptively adjust the learning step size to achieve the best training effect. The initial learning rate for the sub-aperture parameter is set to 0.0025, and the initial learning rate for the neural network parameter is set to 0.0001. The NWPU-RESISC45 remote sensing image dataset is used for both training and testing. This dataset contains 45 scenes and a total of 31,500 images. 1,200 remote sensing images containing 24 scenes are randomly selected from this dataset as the training set. The generalization test is performed on the entire NWPU-RESISC45 dataset. The input and output image size is 256×256 pixels. The batch size is 10 images. Traversing 1,200 images in the training set is considered as one training cycle. This process is repeated until the training iterations reach 1,000, thus completing the joint optimization design of steps one, two, and three.

3. The method as described in claim 1, characterized in that: In step three, the gradient descent method is used to analyze d. i and (x) i ,y i The method for optimizing the parameters in the two-stage, two-order restoration network is as follows: The parameter d of the sub-aperture i and (x) i ,y i The parameters v of the two-stage, two-order restoration network are continuously optimized, as are the parameters d of the sub-aperture. i and (x) i ,y i The parameters of the two-stage, two-order restoration network are optimized in the direction of minimizing the joint loss function, i.e.: Where x * ,y * ,d * ,v * This represents the sub-aperture parameters and image restoration neural network parameters after 1000 training iterations.

4. The method as described in claim 1 or 3, characterized in that: The process also includes step four: Based on the sub-aperture parameters obtained in step three, the sub-aperture arrangement of the OSA system is given, and the optical sparse aperture imaging process under this arrangement is simulated to obtain a blurred image; the image is then restored using the two-stage, two-order restoration network corresponding to this aperture arrangement to obtain a clear image; on the one hand, the sub-aperture arrangement of the OSA system is evaluated, and its MTF and the theoretical cutoff frequency of the optical system are calculated; on the other hand, the image restoration results are evaluated, with the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) among the quantitative evaluation indicators; the entire NWP is calculated. The average SSIM and PSNR of the restored images on the U-RESISC45 dataset were used, and imaging restoration tests were performed on chessboard images. Based on the evaluation results of the sub-aperture arrangement of the OSA system, the image restoration evaluation results, and the imaging restoration test, the pupil arrangement optimization results of the optical sparse aperture imaging system under multiple constraints were verified in multiple dimensions. According to the optimal pupil arrangement optimization results of the optical sparse aperture imaging system, the theoretical cutoff frequency of the optimized OSA system reaches the equivalent full aperture cutoff frequency, and the details of the restored effect on the chessboard image are very clear, and the imaging quality is significantly improved.

Images