Variable pupil imaging system mismatched image super-resolution reconstruction method

Abstract

The present application relates to the technical field of image processing, and more particularly to a variable pupil imaging system mismatch image super-resolution reconstruction method, a variable pupil imaging system is controlled to shoot a target scene at different rotation angles, an initial latent image sequence is obtained, and an initial point spread function sequence corresponding to the initial latent image sequence is determined; the initial latent image sequence is converted to the frequency domain, the amplitude mean value is calculated, and the initial latent image sequence is updated based on the frequency domain amplitude mean value; the initial point spread function sequence is updated based on the mean value after the initial point spread function sequence is averaged in the same direction; the initial latent image sequence, the latent image sequence, and the point spread function sequence are alternately projected to obtain a new latent image sequence and a new point spread function sequence; the new latent image sequence and the new point spread function sequence are iterated to obtain a final reconstructed image. The present application effectively solves the problem that the traditional method cannot effectively register and reconstruct the variable pupil system image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for super-resolution reconstruction of mismatched images in a variable pupil imaging system. Background Technology

[0002] The variable pupil imaging system features a rectangular primary mirror with a high aspect ratio and a high distribution rate along the long axis of the primary mirror. By rotating the primary mirror, it captures and fuses high-resolution images of the observed target in different directions, achieving omnidirectional high-resolution imaging with the long axis of the primary mirror as the equivalent aperture. This is a novel large-aperture imaging technology.

[0003] Current image fusion methods for variable pupil imaging systems involve image acquisition, image registration, and fusion calculation. Due to factors such as platform jitter, misalignment between the primary mirror's mechanical rotation axis and the optical axis, and non-perpendicularity between the optical axis and the detector, misalignment of several to tens of pixels exists between different frames in the variable pupil system. Registration is necessary to achieve accurate fusion of high-resolution information. Existing registration methods mainly include region-based and feature-based methods.

[0004] Region-based methods achieve registration by comparing the similarity of image pixel grayscale values ​​within a certain range. However, due to the significant directional blurring in each frame of the variable pupil system, the similarity between different frames is low, making high-precision registration difficult.

[0005] Feature-based methods perform registration by extracting geometric features such as edges, curves, textures, and corners of ground objects, requiring that the two images have sufficiently obvious identical features. However, due to the low resolution of the minor axis of the aperture in variable pupil systems, feature loss is significant, making high-precision registration difficult to achieve with this method. Summary of the Invention

[0006] In view of this, the present invention aims to provide a super-resolution reconstruction method for mismatched images of a variable pupil imaging system, which uses alternating iterative projection as a framework and is based on the frequency domain correlation between each frame image to accurately reconstruct high-resolution images of the variable pupil system under the condition of inter-frame misalignment.

[0007] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0008] A method for super-resolution reconstruction of mismatched images in a variable pupil imaging system includes:

[0009] S1: Control the variable pupil imaging system to capture images of the target scene at different rotation angles, obtain the initial latent image sequence, and determine the initial point spread function sequence corresponding to the initial latent image sequence;

[0010] S2: After converting the initial latent image sequence obtained in step S1 to the frequency domain, calculate the amplitude mean, and then update the initial latent image sequence based on the frequency domain amplitude mean;

[0011] S3: After taking the average value of the initial point spread function sequence in the same direction, update the initial point spread function sequence based on the average value;

[0012] S4: Alternately project the initial latent image sequence of step S1, the latent image sequence updated in step S2, and the point spread function sequence updated in step S3 to obtain a new latent image sequence and a new point spread function sequence.

[0013] S5: Following steps S2 to S4, iterate over the new latent image sequence and the new point spread function sequence to obtain the final reconstructed image.

[0014] Furthermore, the initial latent images in the initial latent image sequence in step S1 are:

[0015] ;

[0016] Where I(x,y) represents the initial latent image, and I0(x,y) represents the true image of the target scene. This represents the imaging offset in the x and y directions, respectively, when the variable pupil imaging system rotates to capture images, within a Cartesian coordinate system with the rotation axis of the variable pupil imaging system as the z-axis.

[0017] Furthermore, imaging shift for:

[0018] ;

[0019] ;

[0020] Where f represents the focal length of the variable pupil imaging system, φ represents the angle between the rotation axis and the optical axis of the variable pupil imaging system, β represents the rotation angle of the variable pupil imaging system, (θ) x ,θ y ) represents the tilt angle of the variable pupil imaging system relative to the x and y directions when it rotates in a Cartesian coordinate system with the rotation axis of the variable pupil imaging system as the z-axis.

[0021] Furthermore, in step S2, the latent image sequence is updated using the following formula:

[0022] ;

[0023] in, This represents the m-th latent image in the latent image sequence during the i-th iteration. Let F represent the updated latent image, M represent the total number of latent images in the latent image sequence, and F represent the Fourier transform.

[0024] Furthermore, in step S3, the point spread function sequence is updated using the following formula:

[0025] ;

[0026] in, This represents the m-th point spread function in the point spread function sequence under the i-th iteration. R(β) represents the updated point spread function, M represents the total number of point spread functions in the point spread function sequence, and R(β) represents the total number of point spread functions in the sequence. m ) represents the rotation matrix, β m This represents the rotation angle of the variable pupil imaging system corresponding to the spread function at the m-th point.

[0027] Furthermore, in step S4, alternating projection is performed using the following formula:

[0028] ;

[0029] ;

[0030] in, This represents the m-th point spread function in the initial point spread function sequence. Let F represent the m-th point spread function in the sequence of point spread functions updated in the i-th iteration, and let F denote the Fourier transform. Indicates conjugate. This represents the m-th latent image in the new latent image sequence obtained in the i-th iteration. This represents the m-th latent image in the initial latent image sequence. This represents the m-th latent image in the latent image sequence updated in the i-th iteration. Let m be the point spread function in the new point spread function sequence obtained in the i-th iteration.

[0031] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0032] The present invention provides a method for super-resolution reconstruction of mismatched images in a variable pupil imaging system. By introducing PSF rotation consistency and frequency domain invariance of sequential image translation into the alternating projection method, the method achieves super-resolution reconstruction of mismatched images and solves the problem that traditional methods cannot effectively register and reconstruct images of variable pupil imaging systems. Attached Figure Description

[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0034] Figure 1A schematic flowchart of the mismatch image super-resolution reconstruction method of the variable pupil imaging system described in the embodiments of the present invention;

[0035] Figure 2 A flowchart illustrating the method for super-resolution reconstruction of mismatched images in a variable pupil imaging system according to an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of the shooting process of the variable pupil imaging system described in the embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram illustrating the offset of the variable pupil imaging system described in the embodiment of the present invention during the rotation and image acquisition process.

[0038] Explanation of reference numerals in the attached figures:

[0039] 1. Variable pupil imaging system; 2. Target scene; 3. High-resolution image; 4. Clear image in all directions; 5. Rotation axis; 6. Optical axis. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0041] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] like Figure 1 and Figure 2 As shown in the embodiments of the present invention, the method for super-resolution reconstruction of mismatched images in a variable pupil imaging system includes:

[0043] S1: Control the variable pupil imaging system to capture images of the target scene at different rotation angles, obtain an initial latent image sequence, and determine the initial point spread function sequence corresponding to the initial latent image sequence. The imaging process of the variable pupil imaging system is as follows: Figure 3 As shown, the detector in the variable pupil imaging system 1 has a rectangular aperture, with high resolution along the long axis and low resolution along the short side. Rotating the detector changes the aperture direction, acquiring high-resolution images 3 of the target scene 2 in different directions, and using the method provided by this invention to calculate and reconstruct a clear image 4 in all directions.

[0044] In some embodiments, the offset of the variable pupil imaging system during the rotation and image acquisition process is as follows: Figure 4As shown, a Cartesian coordinate system is established with the rotation axis 5 of the variable pupil imaging system as the z-axis. When the variable pupil imaging system rotates by an angle β, the offset of the image generated by the variable pupil imaging system relative to the original position of the target scene is:

[0045] ;

[0046] ;

[0047] Where (δx,δy) represents the offset of the variable pupil imaging system in the x and y directions, f represents the focal length of the variable pupil imaging system, and φ represents the angle between the rotation axis 5 and the optical axis 6 of the variable pupil imaging system.

[0048] Meanwhile, when the variable pupil imaging system rotates to capture images, it tilts as follows, resulting in the following additional image shift:

[0049] ;

[0050] ;

[0051] Where (dx,dy) represents the additional image shift in the x and y directions of the variable pupil imaging system, (θ) x ,θ y The angle () represents the tilt angle of the variable pupil imaging system relative to the x and y directions when it rotates.

[0052] Therefore, the imaging shift of the variable pupil imaging system during rotational shooting can be obtained as follows:

[0053] ;

[0054] ;

[0055] in, This indicates the imaging offset produced by the variable pupil imaging system in the x and y directions.

[0056] Furthermore, we can obtain that the initial latent image in the initial latent image sequence is:

[0057] ;

[0058] Where I(x,y) represents the initial latent image, and I0(x,y) represents the real image of the target scene. It should be noted that the planar coordinate system in the latent image corresponds to the xy plane in the Cartesian coordinate system established with the rotation axis 5 of the variable pupil imaging system as the z-axis.

[0059] The frequency domain relationship between the initial latent image I(x,y) and the true image I0(x,y) of the target scene is as follows:

[0060] ;

[0061] Here, F represents the Fourier transform, and (u,v) represents the coordinates in the frequency domain. For misaligned images actually captured by a variable pupil imaging system, their spectral amplitudes are the same, but their phases are different. Therefore, image super-resolution reconstruction can be achieved based on properties such as frequency domain translation invariance and PSF (point spread function) rotation invariance.

[0062] In this embodiment of the invention, since the sampling of the variable pupil imaging system causes a deviation in the PSF distribution, the images are grouped according to whether the rotation angle is orthogonal before subsequent processing.

[0063] The pupil function of an ideal variable pupil imaging system is:

[0064] ;

[0065] Where P represents the pupil function, rect represents the rectangular cutoff function, and a and b represent the length and width of the rectangular aperture, respectively;

[0066] The point spread function of the variable pupil imaging system is further obtained as follows:

[0067] ;

[0068] in, This represents the autocorrelation operation. The intensity distribution I(x,y) of the image acquired by the variable pupil imaging system is:

[0069] ;

[0070] Here, o(x,y) represents the intensity distribution of the target scene. High-resolution information from each frame can be extracted using computational methods such as deconvolution, enabling image super-resolution reconstruction. When aberrations exist in the variable pupil imaging system, the distribution of the PSF will change, but it still exhibits a pattern of variation with the rotation angle β within a small field of view.

[0071] S2: After converting the initial latent image sequence obtained in step S1 to the frequency domain, calculate the amplitude mean, and then update the initial latent image sequence based on the frequency domain amplitude mean.

[0072] In some embodiments, the latent image sequence is updated using the following formula:

[0073] ;

[0074] in, This represents the m-th latent image in the latent image sequence during the i-th iteration. This represents the updated latent image, and M represents the total number of latent images in the latent image sequence.

[0075] S3: After aligning the initial point spread function sequence from step S1 with the same direction and taking the mean, update the initial point spread function sequence based on the mean.

[0076] In some embodiments, the point spread function sequence is updated using the following formula:

[0077] ;

[0078] in, This represents the m-th point spread function in the point spread function sequence under the i-th iteration. R(β) represents the updated point spread function, M represents the total number of point spread functions in the point spread function sequence, and R(β) represents the total number of point spread functions in the sequence. m ) represents the rotation matrix, β m This represents the rotation angle of the variable pupil imaging system corresponding to the spread function at the m-th point.

[0079] S4: Alternately project the initial latent image sequence from step S1, the latent image sequence updated in step S2, and the point spread function sequence updated in step S3 to obtain a new latent image sequence and a new point spread function sequence.

[0080] In some embodiments, alternating projection is performed using the following formula:

[0081] ;

[0082] ;

[0083] in, This represents the m-th point spread function in the initial point spread function sequence. Indicates conjugate. This represents the m-th latent image in the new latent image sequence obtained in the i-th iteration. This represents the m-th latent image in the initial latent image sequence. Let m be the point spread function in the new point spread function sequence obtained in the i-th iteration.

[0084] S5: Following steps S2-S4, iterate through the new latent image sequence and the new point spread function sequence to obtain the final reconstructed image. It should be noted that after the iteration in step S5, the resulting image is still a latent image sequence. However, the difference between the M images in this sequence lies in the change in the position of the target scene, while the sharpness of the M images tends to be consistent. Therefore, only one image that clearly displays the target scene needs to be selected as the reconstructed image.

[0085] In this embodiment of the invention, the cutoff condition for the iteration operation is that the iteration count i reaches the preset maximum iteration count i. maxIn some other embodiments, the output reconstructed image can also be used to calculate metrics, and when the metrics meet the requirements, the final reconstructed image is obtained.

[0086] In some other embodiments, the alternating projection operation in step S4 can be repeated multiple times during each iteration.

[0087] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0088] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for super-resolution reconstruction of mismatched images in a variable pupil imaging system, characterized in that, include: S1: Control the variable pupil imaging system to capture images of the target scene at different rotation angles, obtain the initial latent image sequence, and determine the initial point spread function sequence corresponding to the initial latent image sequence; S2: After converting the initial latent image sequence obtained in step S1 to the frequency domain, calculate the amplitude mean, and then update the initial latent image sequence based on the frequency domain amplitude mean; S3: After taking the average value of the initial point spread function sequence in the same direction, update the initial point spread function sequence based on the average value; S4: Alternately project the initial latent image sequence of step S1, the latent image sequence updated in step S2, and the point spread function sequence updated in step S3 to obtain a new latent image sequence and a new point spread function sequence. S5: Following steps S2 to S4, iterate over the new latent image sequence and the new point spread function sequence to obtain the final reconstructed image.

2. The method for super-resolution reconstruction of mismatched images in a variable pupil imaging system according to claim 1, characterized in that, The initial latent images in the initial latent image sequence in step S1 are: ； Where I(x,y) represents the initial latent image, and I0(x,y) represents the true image of the target scene. This represents the imaging offset in the x and y directions, respectively, when the variable pupil imaging system rotates to capture images, within a Cartesian coordinate system with the rotation axis of the variable pupil imaging system as the z-axis.

3. The method for super-resolution reconstruction of mismatched images in a variable pupil imaging system according to claim 2, characterized in that, Imaging offset for: ； ； Where f represents the focal length of the variable pupil imaging system, φ represents the angle between the rotation axis and the optical axis of the variable pupil imaging system, β represents the rotation angle of the variable pupil imaging system, (θ) x ,θ y ) represents the tilt angle of the variable pupil imaging system relative to the x and y directions when it rotates in a Cartesian coordinate system with the rotation axis of the variable pupil imaging system as the z-axis.

4. The method for super-resolution reconstruction of mismatched images in a variable pupil imaging system according to claim 1, characterized in that, In step S2, the latent image sequence is updated using the following formula: ； in, This represents the m-th latent image in the latent image sequence during the i-th iteration. Let F represent the updated latent image, M represent the total number of latent images in the latent image sequence, and F represent the Fourier transform.

5. The method for super-resolution reconstruction of mismatched images in a variable pupil imaging system according to claim 1, characterized in that, In step S3, the point spread function sequence is updated using the following formula: ； in, This represents the m-th point spread function in the point spread function sequence under the i-th iteration. R(β) represents the updated point spread function, M represents the total number of point spread functions in the point spread function sequence, and R(β) represents the total number of point spread functions in the sequence. m ) represents the rotation matrix, β m This represents the rotation angle of the variable pupil imaging system corresponding to the spread function at the m-th point.

6. The method for super-resolution reconstruction of mismatched images in a variable pupil imaging system according to claim 1, characterized in that, In step S4, alternating projection is performed using the following formula: ； ； in, This represents the m-th point spread function in the initial point spread function sequence. Let F represent the m-th point spread function in the sequence of point spread functions updated in the i-th iteration, and let F denote the Fourier transform. Indicates conjugate. This represents the m-th latent image in the new latent image sequence obtained in the i-th iteration. This represents the m-th latent image in the initial latent image sequence. This represents the m-th latent image in the latent image sequence updated in the i-th iteration. Let m be the point spread function in the new point spread function sequence obtained in the i-th iteration.

Images