Space camera on-orbit point spread function estimation method, storage medium and equipment

Abstract

The invention relates to a space camera on-orbit point spread function estimation method, a storage medium and equipment, and the method comprises the following steps: obtaining a blurred image collected by a camera, and obtaining a corresponding reference image based on the blurred image; taking the blurred image and the reference image as input of a trained PSF estimation model to obtain an estimated point spread function (PSF); the PSF estimation model comprises a first feature extraction network, a second feature extraction network and a PSF reconstruction network, the first feature extraction network takes a blurred image as input and is used for extracting a blurred image feature map, and the second feature extraction network takes a reference image as input and is used for extracting a reference image feature map; wiener filtering operation is carried out on the blurred image feature map and the reference image feature map to obtain a PSF feature map, and a PSF reconstruction network reconstructs and obtains the estimated PSF based on the PSF feature map. Compared with the prior art, the method has the advantages of high accuracy, simplicity in operation and the like.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and digital image processing, and in particular to an on-orbit point spread function estimation method, storage medium and device for a space camera. Background Art

[0002] The point spread function (PSF) is the response of the camera to a point light source and is the spatial expression of the transfer function of the imaging system. Through the point spread function, we can measure the imaging quality of the optical imaging system, and we can also use the point spread function to restore and enhance the image to improve the clarity of the image. PSF has important applications in Fourier optics, astronomical imaging, medical imaging, electron microscopy and other fields.

[0003] Generally, the methods for obtaining the point spread function mainly include the point pulse method, the pulse method, and the edge method. These methods all let the camera capture point light sources, slits, and edges to directly or indirectly obtain the point spread function of the camera. These measurement methods are usually performed in a laboratory environment. Although the camera point spread function has been measured in the laboratory before launch, due to the vibration of the camera structure during launch, gravity unloading in microgravity, changes in the thermal environment on orbit, and aging of materials after entering orbit, the point spread function of the camera will change, causing image degradation on orbit. Therefore, when the camera is working on orbit, it is necessary to re-model (measure / estimate) the camera point spread function.

[0004] At present, the technology of estimating the point spread function of space cameras on track has also made some progress. For example, the document "MTFAssessment of High Resolution Satellite Images using ISO 12233 Slanted-Edge Method" (Hwang H, Choi YW, Kwak S, et al. Image and Signal Processing for Remote Sensing XIV. SPIE, 2008, 7109: 34-42) uses a ground edge target to estimate the MTF of a space camera in both the vertical and along-track directions, and generates a point spread function for MTFC based on the MTF, but the error of fitting the point spread function from the MTF in the two perpendicular directions is large.

[0005] In addition to the edge method, there are also point spread function estimation methods based on ground point light sources, such as the document "Image Quality Evaluation Method for Optical Remote Sensing Satellite Based on an Array of Point Sources" (ZHENG Yujun, XU Weiwei, LI Xin, et al. Acta Photonica Sinica, 2023, 52 (4): 0428001), which uses ground reflectors or spotlights as point light sources. The image of the point light source captured by the camera can be used to accurately estimate the point spread function. The document "New approach to measure the on-orbit point spread function for spaceborne imagers" (Guo L, Wu Z, Zhang L, et al. Optical Engineering, 2013, 52 (3): 033602-033602.) proposes a point spread function estimation based on natural point targets, using point targets on a single background area and using blind convolution to extract the point spread function. The above methods are based on specific ground targets and use the inherent geometric features of the target to calculate the point spread function. To estimate the spatially varying point spread function of the entire camera field of view, a lot of manual effort is required.

[0006] Another document “Point spread function estimation for wide field small aperture telescopes with deep neural networks and calibration data” (Jia P, WuX, Li Z, et al. Monthly Notices of the Royal Astronomical Society, 2021, 505(4): 4717-4725.) proposes an estimation method for spatially varying point spread functions, which predicts the point spread function of the entire space based on the point spread function at a known position. However, this method is based on the condition that the point spread function varies uniformly in space, and cannot cope with the situation where the remote sensing camera sensor splices the point spread function image in the vignetting area. In the document “Blur estimation and restoration of remote sensing images using a reference image” (WANG T, LI S, LIX. Journal of Remote Sensing, 14(3): 460-467.), it is proposed to use a reference image to perform non-iterative PSF estimation by maximizing the joint probability density under a Bayesian framework. The above method provides the possibility of accurately estimating the point spread function, but the accurate estimation of the point spread function has high requirements on the noise level and the consistency of the scene and brightness between the reference image and the blurred image. In the field of remote sensing images, it is necessary to obtain two pairs of images with a short time interval (small surface changes), similar solar altitude angles, and image brightness, which also makes the restoration time and labor costs relatively large.

[0007] Therefore, it is very difficult to realize point spread function estimation on-orbit based on the existing methods, and the accuracy of point spread function estimation is not high, which affects the image restoration effect. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method, storage medium and device for estimating the on-orbit point spread function of a space camera based on deep learning convolutional neural network and Fourier transform with high accuracy and simple operation.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A method for estimating a point spread function of a space camera on track comprises the following steps:

[0011] Acquire a blurred image captured by a camera, and obtain a corresponding reference image based on the blurred image;

[0012] Using the blurred image and the reference image as inputs of a trained PSF estimation model to obtain an estimated point spread function PSF;

[0013] Among them, the PSF estimation model includes a first feature extraction network, a second feature extraction network and a PSF reconstruction network. The first feature extraction network takes a blurred image as input and is used to extract a blurred image feature map. The second feature extraction network takes a reference image as input and is used to extract a reference image feature map. Wiener filtering operation is performed on the blurred image feature map and the reference image feature map to obtain a PSF feature map. The PSF reconstruction network reconstructs the estimated point spread function PSF based on the PSF feature map.

[0014] Furthermore, the reference image is an image registered with the blurred image.

[0015] Furthermore, the first feature extraction network and the second feature extraction network have the same network structure, both including an encoder and a decoder.

[0016] Furthermore, the acquisition expression of the PSF feature map is:

[0017]

[0018] Among them, feat. PSF Represents the PSF feature map, FB = FFT (feat B ), FR = FFT (feat R ), feat. B 、feat R They represent the blurred image feature map and the reference image feature map respectively, i represents the i-th channel of the feature map, FFT and iFFT represent the fast Fourier transform and inverse fast Fourier transform respectively, is the complex conjugate of FB, and k is the ratio of the variance of the noise to the variance of the blurred signal.

[0019] Furthermore, the PSF reconstruction network is a four-layer cascade network.

[0020] Furthermore, when the PSF estimation model is trained, simulated data is used to simulate real data to construct a data set.

[0021] Furthermore, the images included in the data set are preferably flat area images.

[0022] Furthermore, when the PSF estimation model is trained, the L1 norm is used as the loss function.

[0023] The present invention also provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the on-orbit point spread function estimation method for a space camera as described above.

[0024] The present invention also provides an electronic device, comprising one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the on-orbit point spread function estimation method for a space camera as described above.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention takes a blurred image and a registered reference image as input, extracts effective matching features from the two images through supervised learning, obtains features of the point spread function by performing Wiener filtering operations in the feature space, and reconstructs the point spread function PSF through a network, thereby providing a high tolerance for the sources of the blurred image and the reference image, image changes caused by time intervals, and noise levels, and simplifying the process of manually selecting data, so that only a small amount of public remote sensing data is needed to accurately estimate and model the point spread function of the entire field of view of a space camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the process of the present invention;

[0028] Figure 2 A schematic diagram of the structure of the PSF estimation model constructed by the present invention;

[0029] Figure 3 Schematic diagram of a randomly generated PSF in an embodiment of the present invention

[0030] Figure 4 Schematic diagram of comparison between the PSF predicted by DeepPSF and Wiener filter and the ground truth in an embodiment of the present invention;

[0031] Figure 5 It is a schematic diagram of the restoration effect on a real blurred remote sensing image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0033] The imaging system of the camera is a linear system. The image acquired by the camera is the convolution result of the point spread function of the camera optical system and the ideal image, plus various random noises in the system, which can be expressed by the following formula:

[0034] g(x,y)=h(x,y)*f(x,y)+eta(x,y)

[0035] Where * is the convolution symbol, f(x, y) is the ideal image, g(x, y) is the degraded image, h(x, y) is the degradation function, also known as the point spread function PSF, and η(x, y) is random noise.

[0036] After Fourier transform, the above formula is expressed in the frequency domain as follows:

[0037] G=H·F+N

[0038] Wherein, G, H, F, and N are the frequency spectra of g(x, y), h(x, y), f(x, y), and η(x, y), respectively.

[0039] When the ideal image is known, the PSF can be calculated using the Wiener filter method:

[0040]

[0041] Among them, iFFT means inverse fast Fourier transform, is the complex conjugate of F, and k is the ratio of the variance of the noise η(x, y) to the variance of the blurred signal g(x, y).

[0042] The Wiener filter can only output an accurate PSF when the blurred image and the sharp image are exactly the same. However, we cannot obtain an ideal image from a natural scene, except for artificial targets such as calibration targets or point light sources. Instead, we can obtain a sharp, noisy, and slightly altered image that is close to the ideal image. This non-ideal image is called a reference image.

[0043] Based on the above formula, the present invention uses a noisy clear reference image to replace the ideal image f without noise, utilizes the learning ability of the neural network, uses a convolutional neural network to overcome the interference of noise, and extracts the features of the blurred image and the reference image. Performs Wiener filtering in the feature space and calculates the features of the PSF. Finally, the PSF is accurately reconstructed. Figure 1 As shown, the method for estimating the point spread function of a space camera on-orbit provided in this embodiment includes the following steps:

[0044] Step S1, obtaining a blurred image captured by a camera, and obtaining a corresponding reference image based on the blurred image;

[0045] Step S2: using the blurred image and the reference image as inputs of a trained PSF estimation model to estimate a point spread function PSF.

[0046] The above method is a point spread function estimation method that combines deep learning convolutional neural network with Fourier transform, denoted as DeepPSF (On-orbit Point Spread Function Modeling with DeeplearningMethod).

[0047] The structure of the PSF estimation model designed in this embodiment is as follows Figure 2 As shown, it includes two first feature extraction networks, a second feature extraction network and a PSF reconstruction network.

[0048] The PSF estimation model is a two-stream network. The two branches of the network (i.e., the first feature extraction network and the second feature extraction network) are respectively based on the blurred image image B and clear image R is the input, image B and image R is the registered image. These two branches are denoted as FEN B and FEN R (feature extraction network, FEN, feature extraction network), the image feature maps extracted by these two networks are expressed as:

[0049] feat B =FEN B (image B )

[0050] feat R =FEN R (image R )

[0051] After that, we perform Wiener filtering on the two feature maps to obtain the feature map feat of the point spread function PSF PSF , the Wiener filter calculation is performed channel by channel, that is,

[0052]

[0053] Where i represents the i-th channel of the feature map, FB = FFT (feat B ), FR = FFT (feat R ), where k is the ratio of the variance of the noise to the variance of the blurred signal.

[0054] Network FEN B and FEN RThey have the same structure, both of which are based on the FPN (Feature Pyramid Network) structure. FEN consists of an encoder and a decoder. The encoder consists of three stages of blocks, each of which includes a CNN layer and three residual layers (resnet).

[0055] PSF Reconstruction Network RN PSF It is a four-layer cascade network, represented by the ResBlock-CNN-ResBlock-CNN structure. The first layer ResBlock continues to process the feature map of PSF and extracts features with unchanged feature depth. The second layer is a CNN layer, which compresses the dimension of the input feature and reduces the feature depth to 1 / 2 of the original. The second ResBlock maintains the dimension of the feature map. The last layer compresses the dimension of the feature map to 1 and outputs the predicted PSF.

[0056] For remote sensing images, it is extremely difficult to obtain the real point spread function. Therefore, this embodiment uses simulated data to simulate real data to make a data set to realize the training of the estimation network. First, we select some areas on the high-definition remote sensing map, and select flat areas such as farmland and airports as preferred areas. The camera shooting angle and target height will affect the remote sensing image registration, so cities and areas with large height changes are not considered. We select two images of the same area at different times. The two images have the same image resolution and have been registered. Using this method, we obtained 18 pairs of images. The image size is about 2200*1200 pixels. After that, we split the image into 400*400 image blocks, and obtained a total of 902 pairs of image blocks. We randomly selected 800 pairs as training data sets and the remaining 102 pairs as validation data sets. We use random two-dimensional directional Gaussian kernels as point spread functions, and the size of each point spread function is 31×31. Each point spread function consists of 4 Gaussian kernels to enhance the richness and complexity of the PSF. Examples of PSF are as follows Figure 3 As shown. During the training process, we first randomly generated 100 PSFs as the PSF set using the above method. When extracting image pairs from the dataset, we randomly select a PSF from the PSF set, and then use the PSF to blur one image in the image pair as the blurred image image B , another image is not blurred as the reference image image R We add Gaussian noise with a standard deviation of 2% to each image, and randomly change the image color value in the range of [0.7, 1.2]. We randomly crop 320×320 sub-blocks, and the blurred image and the reference image are cropped at the same position. Finally, we use the image sub-block pairs and the corresponding point spread functions as training data.

[0057] In the training process of this embodiment, the L1 norm is used as the loss function, and the loss function is as follows:

[0058]

[0059] in, is the ground truth of PSF, x is the PSF predicted by the network, and N l It is batch_size.

[0060] The experiment of this embodiment is divided into two parts. The first part is to evaluate on the validation dataset. The second part is to evaluate the performance of image restoration using the predicted PSF for real blurry remote sensing.

[0061] (1) Evaluation on the validation dataset.

[0062] This embodiment uses the mean square error (MSE) and structural similarity (SSIM) to evaluate the accuracy of the predicted PSF. For evaluation, we added different degrees of Gaussian noise and used the obtained DeepPSF and Wiener filter (WF, Weiner) to predict PSF and evaluate its indicators. The results are shown in Table 1.

[0063] Table 1 Comparison of evaluation results of DeepPSF and Wiener filter

[0064]

[0065] We then compare the PSF predicted by DeepPSF and Wiener filter with the ground truth PSF, and the results are shown in Figure 4 shown. Figure 4 In the figure, scene (a) is part of an airport, scene (b) is part of a farmland, the first two columns are the reference image and the blurred image, the third column is the ground truth of the PSF, and the last three columns are the PSFs predicted by DeepPSF and WienerFilter when the noise level is 0%, 3%, and 10%, respectively.

[0066] As shown in Table 1, the MSE and SSIM indicators of DeepPSF are better than those of Wiener filtering at all three noise levels. As the noise level increases, the SSIM indicator of DeepPSF decreases, but it is still high overall, while the Wiener filtering method is no longer effective in comparison. Figure 4As shown, the DeepPSF proposed in the present invention overcomes the influence of time changes in the scene. When the noise level is 0, the SSIM of scene (a) and scene (b) reaches 0.917 and 0.919 respectively, and the predicted PSF is highly consistent with the ground truth PSF. As the noise increases, the SSIM drops to 0.862 and 0.905 in scene (a) and scene (b), respectively, and the shape of the predicted PSF remains unchanged. The evaluation results show that the network in DeepPSF has learned the features useful for predicting PSF through training, and plays a role in feature filtering in the feature extraction stage. Therefore, DeepPSF is more effective than WienerFilter and has stronger anti-interference ability.

[0067] (2) Test on real blurry remote sensing images.

[0068] This example applies DeepPSF to the task of restoring real remote sensing images. The blurred image comes from a push-broom space camera with a spatial resolution of 1.5 meters. After the camera entered orbit, the optical mechanical structure was abnormally deformed, resulting in an abnormal point spread function and severe image blur. The original blurred remote sensing image is as follows: Figure 5 shown.

[0069] In this embodiment, an L1 level image is acquired from the camera and the corresponding area is found on the HD map, thereby obtaining a reference image and highlighting it after registration. Figure 5 In the square box above. This embodiment uses DeepPSF to estimate PSF, such as Figure 5 The PSF of the push-scan camera is constant along the track. This embodiment uses DWDN as the image restoration algorithm and restores the image using the predicted PSF. The images before and after restoration are shown in Figure 5 In the restored image, Figure 5 As shown in the comparison between ① and ②, the docks, roads and other areas become clear, the ships on the river become sharp, and the image details can also be distinguished.

[0070] The restoration effect of this embodiment on the real RSI verifies that DeepPSF can accurately predict the PSF of the space camera.

[0071] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0072] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for estimating the point spread function of a space camera on track, characterized in that: The following steps are involved: Acquire a blurred image captured by a camera, and obtain a corresponding reference image based on the blurred image; Using the blurred image and the reference image as inputs of a trained PSF estimation model to obtain an estimated point spread function PSF; Among them, the PSF estimation model includes a first feature extraction network, a second feature extraction network and a PSF reconstruction network. The first feature extraction network takes a blurred image as input and is used to extract a blurred image feature map. The second feature extraction network takes a reference image as input and is used to extract a reference image feature map. Wiener filtering operation is performed on the blurred image feature map and the reference image feature map to obtain a PSF feature map. The PSF reconstruction network reconstructs the estimated point spread function PSF based on the PSF feature map.

2. The method for estimating the point spread function of a space camera on track according to claim 1, characterized in that: The reference image is an image registered with the blurred image.

3. The method for estimating the point spread function of a space camera on track according to claim 1, characterized in that: The first feature extraction network and the second feature extraction network have the same network structure, and both include an encoder and a decoder.

4. The method for estimating the point spread function of a space camera on track according to claim 1, characterized in that: The acquisition expression of the PSF feature map is: Among them, feat. PSF Represents the PSF feature map, FB = FFT (feat B ), FR = FFT (feat R ), feat. B 、feat R They represent the blurred image feature map and the reference image feature map respectively, i represents the i-th channel of the feature map, FFT and iFFT represent the fast Fourier transform and inverse fast Fourier transform respectively, is the complex conjugate of FB, and k is the ratio of the variance of the noise to the variance of the blurred signal.

5. The method for estimating the point spread function of a space camera on track according to claim 1, characterized in that: The PSF reconstruction network is a four-layer cascade network.

6. The method for estimating the point spread function of a space camera on track according to claim 1, characterized in that: When the PSF estimation model is trained, simulated data is used to simulate real data to construct a data set.

7. The method for estimating the point spread function of a space camera on track according to claim 6, characterized in that: The images contained in the dataset are preferably flat area images.

8. The method for estimating the point spread function of a space camera on track according to claim 1, characterized in that: When the PSF estimation model is trained, the L1 norm is used as the loss function.

9. A computer-readable storage medium, characterized in that: It comprises one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs comprise instructions for executing the method for estimating the point spread function of a space camera on track as claimed in any one of claims 1 to 8.

10. An electronic device, characterized in that: The method comprises one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the method for estimating the point spread function of a space camera on track as claimed in any one of claims 1 to 8.

Images