A fine optical image simulation method based on spatial array convolution

Abstract

The application discloses a fine optical image simulation method based on a space array convolution. The method comprises the following steps: 1) performing grid division on a field of view range of a space camera to be simulated to obtain a field of view array; and calculating a center field of view of each grid region in the field of view array; 2) according to optical system parameters of the space camera and a set interference atmospheric condition, calculating a point spread function of a center field of view corresponding to each image block to obtain a point spread function PSF array; 3) performing division on a clear image I0 and mapping each image block to a corresponding grid block in the field of view array; performing convolution calculation on each image block and a point spread function of the corresponding grid in a space domain to obtain a degraded simulation image block corresponding to each image block; and splicing the degraded simulation image blocks to obtain a degraded image I'; and 4) superimposing noise on the degraded image I' to obtain a simulation degraded image I'' simulating the space camera shooting the clear image I0.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image data simulation and processing, and particularly relates to a fine optical image simulation method based on spatial array convolution. BACKGROUND

[0002] Optical aberration is the deviation of the real imaging process of an optical system from the ideal imaging. It contains the defects in lens design and the influence of external environment, temperature and other interference factors, which will cause light diffusion and the light cannot be focused to form a clear image, generally showing blur and dispersion. In fact, although today's optical imaging system design is quite mature, but in the lens design process, it is inevitable to produce a certain amount of aberration. The space camera works in orbit for a long time, and the lens will deform in the space environment, and there are also atmospheric interference, lens defocus and other situations in the camera shooting process. Moreover, the degradation effects of these aberrations are not consistent in different field of view ranges of the whole image, such as field area, distortion, defocus, etc. which are usually distributed radially with the optical axis as the center, so it is not accurate to use one degradation function to represent the whole image for image degradation simulation. Image degradation is a complex physical process.

[0003] In order to improve the imaging effect of space camera, the upgrade of on-orbit hardware is too expensive, so the task of improving image quality is transferred from "hard means" to "soft means", and the correction of optical aberration is regarded as an image restoration problem. There are two technical routes for image restoration algorithm: one is the traditional model-driven method, which uses various natural image priors to optimize the blur kernel multiple times, and then deconvolves to get the target clear image; the other is the data-driven deep learning method, which needs a large amount of data set to learn the parameters of the network model. The traditional method is not robust enough when dealing with spatially varying blur kernels; the deep learning method is becoming more and more popular recently, but it faces the main problem of being unable to obtain real clear images, lacking of data sets, and the ordinary image degradation simulation is not aimed at a specific space camera structure, and does not contain accurate physical mechanism, but is simplified and equivalent to a simple Gaussian degradation kernel, resulting in a large difference between the characteristics of image simulation and real imaging results. Moreover, each optical system shows completely different characteristics, so it is necessary to design a fine optical image simulation method combining optical parameters from the physical process.

[0004] In existing technologies, the most common image simulation method is to convolve Gaussian blur kernels of different sizes with sharp images to simulate the aberration degradation process. This method is inaccurate for space camera optical systems because optical aberrations in space camera optical systems are not spatially invariant, especially when the camera's swath width is large, as the PSF (point spread function) varies spatially with the field of view (FOV). Dataset simulation methods using Gaussian blur kernels cannot simulate the spatially varying characteristics of real systems. Current technologies lack a method that can flexibly simulate space camera imaging; therefore, a sophisticated optical image simulation method is needed to provide a robust support platform for research on remote sensing image anti-interference technology, target recognition, and image processing algorithms. Summary of the Invention

[0005] To address the technical problems existing in the prior art, the present invention aims to provide a refined optical image simulation method based on spatial array convolution. This invention directly uses the optical system parameters and sharp images of a space camera as input, and outputs simulated images with varying degrees of aberration degradation. It can generate large simulation datasets to provide training data for subsequent tasks such as image restoration.

[0006] The technical solution of this invention is as follows:

[0007] A refined optical image simulation method based on spatial array convolution, comprising the following steps:

[0008] Step (1) assumes that the neighborhood degradation of natural images is similar, as shown in the appendix. Figure 1 As shown, the entire field of view of the space camera corresponds to the size of the sharp image, encompassing the entire sharp image. The actual image captured by the space camera is a degraded sharp image. The entire field of view of the space camera is divided into grids, and the central field of view of each grid region is calculated. Then, the segmented sharp image is spatially convolved with the point spread function (PSF) of its corresponding central field of view to simulate the image degradation process. A mathematical model is established for the degradation process of the space camera, and the resulting mathematical model for the simulated image degradation process is as follows:

[0009]

[0010] Where (i,j) are the center coordinates of the segmented degraded image I'; I0 is the sharp image corresponding to the degraded image I'; k(i,j) is the energy-normalized point spread function, representing the energy spread caused by system aberrations.

[0011] Step (2) Accurately calculate the point spread function. Based on the optical system parameters of the pre-simulated space camera and the interfering atmospheric conditions, calculate the point spread function (PSF) of the central field of view corresponding to each grid sub-region.

[0012] The setting parameters include the number of pupil sampling, the number of image plane sampling, wavelength, the number of grid division in horizontal X and vertical Y directions, and central field of view. The default wavelength is RGB, the refractive index of the wavelength is calculated, and the field of view angle corresponding to the grid center position is calculated. According to the obtained optical system parameters, the refractive index, and the grid center field of view, the image plane grid center coordinates are calculated, and the optical system wave aberration W0(i,j) corresponding to the field of view of each grid region center is calculated by ray tracing.

[0013] Then, the atmospheric disturbance phase W 大气 (i,j) is generated according to the Zernike polynomial.

[0014]

[0015] Wherein, is the Zernike polynomial, C n,m is the coefficient of the Zernike polynomial, N is the order of the Zernike polynomial, W 大气 (i,j) is the atmospheric disturbance phase.

[0016] The atmospheric disturbance phase W 大气 (i,j) and the optical system wave aberration W0(i,j) are superimposed to calculate the total wave aberration W(i,j) of the corresponding image region field of view:

[0017] W(i,j)=W 大气 (i,j)+W0(i,j)

[0018] Wherein, (i,j) represents the coordinates, W 大气 (i,j) is the atmospheric disturbance phase, W0(i,j) is the optical system wave aberration, and W(i,j) is the total wave aberration at the coordinates (i,j) position.

[0019] According to the total wave aberration of each image region, the point spread function PSF array corresponding to each image region is calculated:

[0020]

[0021] Wherein, j is the imaginary part, λ is the wavelength, z is the focal length, (x,y) is the image plane coordinate, (ξ,η) is the pupil plane coordinate, psf(x,y) is the point spread distribution function of each image block, FFT is the Fourier transform, A(ξ,η) is the pupil function, W(ξ,η) is the total wave aberration at the coordinates (ξ,η) position, and FFT is the Fourier transform.

[0022] Step (3) is the array convolution in the spatial domain, and the PSF calculated in step (2) is convolved with the image of the corresponding region according to step (1) in the spatial domain.

[0023] Firstly, according to the number of networks divided in X and Y directions respectively, the image is cropped into a uniform array. Then the image array is respectively convolved with the PSF point spread function of the corresponding central field of view in the spatial domain. In the convolution process, it is necessary to ensure that the pixel interval of the image is consistent with the pixel interval of the PSF, and it is also necessary to prevent the simulated degradation image obtained after convolution from having black edges, so the image should be interpolated in advance, and the nearest neighbor interpolation is used in the application. In addition, in order to ensure that the energy of the image is unchanged, the PSF matrix needs to be normalized in advance, and then block convolution is performed. Finally, the degraded simulated image blocks are spliced together to obtain a complete degraded image I' all (i,j).

[0024] Step (4) superimposes noise on the image obtained in step (3), which can be random white noise or salt and pepper noise, etc., and is determined according to the specific imaging scene.

[0025] I”(i,j)=I' all (i,j)+N(i,j)

[0026] Where (i,j) is the coordinate, I' all (i,j) is the complete degraded image obtained by splicing the block degradation, N(i,j) is the noise, and I”(i,j) is the final simulated image.

[0027] Compared with the prior art, the application has the following beneficial effects:

[0028] (1) Based on the method of fine optical image simulation, the problem of lack of clear image contrast under ideal conditions in the spatial camera remote sensing image restoration process can be solved, a large number of data sets can be created for the research of image restoration algorithm through simulation, and reference can be provided for the design of spatial cameras in the design stage through pre-image simulation, thereby promoting the optimization and improvement of the design.

[0029] (2) The simulated data set using the Gaussian blur kernel in the prior art is inaccurate, because the optical aberration in the optical system of the spatial camera has a spatial variation characteristic. The real PSF point spread function varies spatially with the change of the field of view FOV, and is not uniform and constant. The application adopts the method of calculating the PSF in a grid region, effectively retains the spatial variation characteristic of the image optical aberration, makes the simulated image closer to the real image, and thus improves the effect of image restoration based on deep learning.

[0030] (3) Compared with the commonly used frequency domain imaging simulation, the spatial domain imaging simulation is faster under the premise of ensuring the spatial variation characteristics of the image optical aberration. The reason is as follows: in order to ensure the spatial variation characteristics of the PSF, block convolution needs to be performed. The method of the application performs convolution in the spatial domain, each sub-region performs convolution calculation and then splices all the regions. If the calculation is performed in the frequency domain, the frequency domain characteristics of each sub-region are not complete, the point spread of each sub-region needs to be convolved with the frequency information of the whole picture and then deconvolved, otherwise the frequency domain information will be lost, which will cause the simulated image to have a fault at the joint of each sub-region, and if the point spread of each sub-region is convolved with the frequency information of the whole picture, the calculation amount will be extremely large and the calculation efficiency will be extremely low for the space camera with a resolution of billions of pixels, so the block convolution in the spatial domain is more efficient. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The imaging field is divided into regions.

[0032] Figure 2 The flowchart of the fine optical image simulation method based on spatial domain array convolution.

[0033] Figure 3 The structure of the spatial camera optical system is shown in a two-dimensional view.

[0034] Figure 4 The PSF visualization gray scale contrast diagram of the central field of view of the image under different defocus distances is shown.

[0035] (a) defocus-6mm, (b) defocus-3mm, (c) no defocus, (d) defocus 3mm, (e) defocus 6mm.

[0036] Figure 5 The PSF array grid diagram is shown.

[0037] Figure 6 The simulation before and after the image contrast diagram is shown.

[0038] (a) original clear image, (b) simulation image. DETAILED DESCRIPTION

[0039] The application discloses a fine optical image simulation method based on spatial domain array convolution, and will be described in detail below in combination with the drawings and specific embodiments:

[0040] The flowchart of the embodiment 1 is shown in the accompanying Figure 2 The detailed implementation process of the technical scheme of the embodiment 1 will be described in combination with the accompanying Figure 2

[0041] The image to be simulated is input, and in this example, a target image with a size of 2048*2048 is used, as shown in the accompanying​Figure 5 The left image shows.

[0042] Input the grid number of X, Y direction PSF X POINT, PSF Y POINT, and cut the image into uniform size image blocks according to the set grid number. In this case, PSF X POINT is 5, and PSF Y POINT is 5.

[0043] Input the optical system of the simulated space camera lens, the number of pupil samples, the number of image plane samples, the wavelength, the central field of view, the field of view height, and the field of view type. The parameters of the optical system are shown in Table 1, and the corresponding 2D view is shown in FIG. 1. The number of pupil samples is 64, the number of image plane samples is 64, the wavelength is RGB (R = 0.656um, G = 0.587um, B = 0.486um), the central field of view is (FIELD X, FIELD Y) = (0, 0), the field of view height FIELD HEIGHT is 166.7mm, the field of view type is object height, and the system has a circular aperture. Figure 3

[0044] Table 1 Optical system parameters (unit: mm)

[0045]

[0046] According to the grid number of X, Y direction PSF X POINT, PSF Y POINT, the field of view height FIELD HEIGHT, and the field of view type, the central field of view position of each grid image is calculated. Note that when the field of view type is angle, the reference point is the coordinate corresponding to the center of the image plane, which needs to be converted; when the field of view type is object height, the reference point is the coordinate corresponding to the center of the object plane.

[0047] According to the optical system parameters, the wavelength, the number of pupil samples, the number of image plane samples, and the central field of view position of each image block, the ray tracing is calculated for each ray, and the system wave aberration of each field of view is generated.

[0048] Couple the atmospheric additional aberration, and output the point spread function PSF matrix corresponding to the image block. When the different defocus distances of the optical system are set, the point spread function presents different diffusion degrees. Taking the central field of view of the image as an example, it is visualized as shown in FIG. 2. The point spread function PSF matrix is spliced according to the original position, and the PSF grid map is obtained by visualizing it, as shown in FIG. 3. Figure 4 Figure 5

[0049] ​​​According to the network number PSF_X_POINT and PSF_Y_POINT divided in X and Y directions respectively, the image is cropped into uniform small blocks, and then the image blocks are respectively convolved with the PSF point spread function of the corresponding central field of view in the spatial domain. In the convolution process, it is necessary to ensure that the pixel interval of the image is consistent with the pixel interval of the PSF, and it is also necessary to prevent the simulation degradation image obtained after convolution from having black edges, so the image should be interpolated in advance, and the nearest neighbor interpolation is used in this case. In addition, in order to ensure that the energy of the image is unchanged, the PSF matrix needs to be normalized in advance, and then the block convolution is performed. Finally, the image blocks after degradation simulation are spliced together, and a complete aberration degradation image can be obtained. According to the type and intensity of the noise set by the user, the noise is not considered in this case, and the final simulation result image is obtained, as shown in the right image of Fig. 1. Figure 6

[0050] As for the simulation speed, using the above parameters for spatial domain image simulation on a 2048*2048 image, the average simulation time of a single image is 2.132 seconds. While using the frequency simulation method, the average simulation time of a single image is 40.743 seconds. Therefore, the method has obvious improvement in simulation efficiency.

[0051] As for the effect of the spatial array convolution imaging fine optical image simulation method, 60 scene pictures are selected from DIV2K, and batch image simulation is performed on them using the above method to make a data set. The specific parameters are as follows: the variable is the thickness of the image surface (No. 13 surface), the range of the defocus distance is set to-1.5mm to 1.5mm, the interval is 0.05mm, and there are 61 defocus degrees. After the image restoration algorithm Restormer (CVPR2022) is trained on the simulation data set as above, the model is obtained. Then, the test is performed on the simulation image, and after the Restormer algorithm is trained on the simulation data set, the image signal-to-noise ratio after restoration is improved by an average of 37.4%, which can effectively restore the image structure detail content. This verifies the feasibility and effectiveness of the method of using spatial convolution imaging simulation to make a spatial camera simulation data set.

[0052] The part of the application not described in detail belongs to the known technology of those skilled in the art.

[0053] Although specific embodiments of the application are disclosed for illustrative purposes, the purpose is to help understand the content of the application and to implement it, those skilled in the art can understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the application and the appended claims. Therefore, the application should not be limited to the disclosed content of the best embodiment, and the scope of the application claimed is defined by the scope of the claims.​

Claims

1. A method for simulating fine optical images based on spatial array convolution, comprising the following steps: 1) The field of view of the space camera to be simulated is divided into a grid to obtain a field of view array; Calculate the central field of view for each grid region in the field of view array; 2) Based on the optical system parameters of the space camera and the set interfering atmospheric conditions, calculate the point spread function (PSF) of the central field of view for each image patch to obtain a PSF array. The method for obtaining the PSF array is as follows: 21) Set the optical system parameters, including the number of pupil samples, the number of image plane samples, the wavelength, the number of grids divided in the horizontal and vertical directions, and the central field of view; then calculate the refractive index of the wavelength and the field of view angle corresponding to the grid center position; based on the optical system parameters, refractive index, and the field of view angle corresponding to the grid center position, calculate the coordinates of the image plane grid center; finally, calculate the optical system wavelet aberration corresponding to the central field of view of each grid region through ray tracing. 22) Generate the atmospheric interference phase for each grid region; superimpose the atmospheric interference phase and optical system wavefront aberration for each grid region to obtain the total wavefront aberration of the corresponding image region's field of view; 23) Calculate the point spread function corresponding to the image patch based on the total wavelet aberration of each image region. Where j is the imaginary part, λ is the wavelength, z is the focal length of the space camera, (x,y) are the image plane coordinates of the space camera, (ξ,η) are the pupil plane coordinates of the space camera, FFT is the Fourier transform, A(ξ,η) is the pupil function of the space camera, and W(ξ,η) is the total wavefront aberration at the coordinate (ξ,η) position in the space camera. 3) Divide the clear image I0 and map each image block to the corresponding grid block of the field of view array; Each image patch is convolved with the point spread function of the corresponding grid in the spatial domain to obtain the degraded simulation image patch corresponding to each image patch; the degraded simulation image patches are then stitched together to obtain the degraded image I'. 4) Noise is superimposed on the degraded image I' to obtain a simulated degraded image I that simulates the clear image I0 taken by the space camera.

2. The method according to claim 1, characterized in that, Before performing convolution calculation in step 3), interpolation is performed on each image block to prevent black borders in the simulated image, and the point spread function (PSF) array is normalized before performing convolution calculation in the spatial domain.

3. The method according to claim 1 or 2, characterized in that, The atmospheric disturbance phase for each grid region is generated using Zernike polynomials.

4. The method according to claim 1 or 2, characterized in that, The noise superimposed on the degraded image I' is determined based on the imaging scene.

5. The method according to claim 4, characterized in that, The noise is random white noise or salt and pepper noise.

6. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing each step of the method of any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images