An Extended Object Hartmann Wavefront Detection Method Based on Preprocessing of Reference Images

Abstract

The present invention provides an extended object Hartmann wavefront detection method based on reference image preprocessing. By modifying the ratio of the template occupying the central sub-aperture and using relevant image processing (edge extraction), the stability and accuracy of the extended object related Hartmann wavefront detection can be improved. The present invention combines an optimization algorithm and image processing, reducing the wavefront reconstruction residual in the extended object Hartmann detection. This image preprocessing method has a simple and stable process, is easy to implement, and provides technical support for the better application of the extended object wavefront detection system in wavefront detection fields such as astronomical observation, high-quality laser beam transmission, target recognition, and target imaging.

Description

Technical Field

[0001] The present invention belongs to the fields of optical systems and image processing, and particularly relates to an extended target Hartmann wavefront detection method based on reference image preprocessing. Background Art

[0002] With the technological development in the application fields of adaptive optics (AO) systems such as high-beam-quality laser transmission, target recognition, and target imaging, under certain conditions, the observed targets cannot be simply defined as point targets. To distinguish them from point targets, these targets are generally called extended targets. Regarding the definition of extended targets, there is no unified view in the academic community at present. Two commonly recognized views are as follows: one is to define by the number of pixels of the imaging target, and a target occupying more than 2 / 3 of the field of view area is called an extended target; the other is through signal analysis, and an image with more than one spatially distributed measurement value obtained in each frame is called an extended target. Compared with astronomical observations, the objects of action in the fields of high-beam-quality laser transmission, target recognition, and target imaging are mostly moving non-cooperative targets, that is, the extended state such as the target attitude and distance changes in real time, resulting in problems such as a decrease in detection accuracy and poor stability in wavefront detection based on the relevant Hartmann.

[0003] Regarding the extended object Hartmann wavefront sensing, researchers have conducted a large number of studies, and the main achievements are as follows: In 1998, the NSO low-order solar adaptive system was successfully applied, marking the birth of an extended object wavefront detector based on the related Hartmann (Rao C, Jiang W, Ling N, et al. Correlation tracking algorithms for low-contrast extended object [C] / / Adaptive Optics Systems and Technology II. International Society for Optics and Photonics, 2002, 4494: 245-251); In 2003, L.A. Poyneer et al. conducted a systematic study on the correlation algorithm based on FFT and parabolic interpolation, and proposed the idea of applying the correlation algorithm to the wavefront sensing of extended objects other than the sun. When the relative offset between sub-images is 0, starting from the calculation of the variance of image noise, the interpolation error formula of the correlation function was deduced and a concise result was obtained (Poyneer LA. Scene-based Shack-Hartmann wave-front sensing: analysis and simulation [J]. Applied Optics, 2003, 42(29): 5807-5815); In 2005, the team conducted a distorted wavefront sensing experiment on an extended object with a horizontal length of 100 meters at a height of 1 to 2 meters above the asphalt road under the hot daytime weather in California. This experiment verified the feasibility of using the related Hartmann wavefront detector for wavefront detection of horizontal extended objects (Poyneer L A, Palmer DW, LaFortune K N, et al. Experimental results for correlation-based wavefront sensing [C] / / Advanced Wavefront Control: Methods, Devices, and Applications III. International Society for Optics and Photonics, 2005, 5894: 58940N); In 2014, Shen Tingting et al. from the Chengdu Institute of Optics and Electronics, Chinese Academy of Sciences, for the first time adopted the cross-correlation factor algorithm to complete real-time wavefront processing in a 37-element low-order solar AO system, and obtained correct wavefront slope results in the laboratory (Shen Tingting. Research on real-time wavefront processing technology of solar adaptive optics based on cross-correlation factor algorithm [D].Graduate School of the Chinese Academy of Sciences (Institute of Optoelectronics), 2015); In 2016, M Rais et al. proposed a new method for calculating the sub-aperture offset based on the global optical flow equation. This method is more accurate and stable, less sensitive to noise and has lower variability than all current state-of-the-art methods, and can perform more accurate wavefront reconstruction (Rais M, Morel JM, Thiebaut C, et al. Improving the accuracy of a Shack-Hartmann wavefront sensor on extended scenes [C] / / Journal of Physics: Conference Series. IOP Publishing, 2016, 756(1): 012002.); In 2019, Wang Yawen et al. from the Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences proposed the gamma transformation related algorithm and the gradient cross-correlation algorithm, effectively solving the problems of low wavefront detection accuracy for low-contrast extended targets and low detection accuracy for targets at the edge of the feature-adjacent field of view, and verified the experiments using an extended light source with a wavelength band of 400 - 1700 nm and a Hartmann with 200 effective sub-apertures (pixel number 28×28) (Wang Yawen. Research on the related Hartmann wavefront detection method based on extended targets [D]. University of Chinese Academy of Sciences (Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences), 2019).

[0004] Analyzing the reported wavefront detection research on related Hartmann extended targets, there is no relevant report at home and abroad on how to improve the detection accuracy from the perspectives of reference image preprocessing and algorithm optimization. The present invention provides a method for detecting the wavefront of an extended target by a Hartmann based on reference image preprocessing. By modifying the proportion of the template occupying the central sub-aperture and using related image processing (edge extraction), the stability and accuracy of the wavefront detection of the related Hartmann for extended targets can be improved. The present invention combines the optimized algorithm and image processing, reducing the wavefront reconstruction residual in the Hartmann detection of extended targets. This image preprocessing method has a simple, stable and easy-to-implement process, providing technical support for the better application of the extended target wavefront detection system in wavefront detection fields such as astronomical observation, high-quality laser beam transmission, target recognition, and target imaging. Summary of the Invention

[0005] The technical problem to be solved by the present invention: When detecting the wavefront of an extended target by a Hartmann, the detection accuracy decreases and the stability is poor due to the real-time changes in the states of the extended target such as attitude and distance. In view of this problem, a method for preprocessing the reference image using prior knowledge is proposed to ensure higher accuracy and stability in wavefront detection under the condition of real-time changes in the target extension.

[0006] The technical solution adopted by the present invention to solve its technical problems is as follows: An extended object Hartmann wavefront detection method based on reference image preprocessing, the method comprising the following steps:

[0007] Step 1) Obtain a sub-aperture image array from a Hartmann sensor with a sub-aperture unit of (2P - 1)×(2P - 1) or 2P×2P. Take the sub-aperture at the center of the Hartmann sub-aperture array (the corresponding number of rows and columns of this sub-aperture are both P) as the selection position of the reference image I. The pixel size of the reference image I is M×M, and the single-pixel gray value is represented by I r (x, y), where (x, y) is the two-dimensional coordinate value of the pixel. The size of a single-aperture image is N×N, and the single-pixel gray value is represented by denoted as, is the offset of the central pixel point of a single sub-aperture image from the central pixel point of the reference image I in both the horizontal and vertical directions, (N≥M);

[0008] Step 2) Denote the ratio of the starting position coordinate values of the reference image I in the horizontal and vertical directions within the central sub-aperture to the pixel size of the central sub-aperture in the horizontal and vertical directions as TemplateL, and the ratio of the pixel size in the horizontal and vertical directions to the pixel size of the central sub-aperture in the horizontal and vertical directions as TemplateR. The actual values of TemplateL and TemplateR are determined according to the number of pixels occupied by the extended object in the Hartmann sub-aperture and the background situation of the sub-aperture image;

[0009] Step 3) Convert the pixel gray value data format of the reference image I to the double type, and then perform normalization processing, that is, divide the single-pixel gray value by the maximum gray value of the entire image to make the gray value range between [0, 1]. Then, the gray values in the range [τ, υ] are converted to the range [0, 1] through a mapping relationship to obtain the converted image J of the reference image. The mapping relationship formula is as follows

[0010]

[0011] where t is the normalized pixel gray value before conversion, and s is the pixel gray value after conversion. τ is the lower limit of the pixel value in the conversion interval, υ is the upper limit of the pixel value in the conversion interval, 0<τ<υ<1. After conversion, the pixel value τ corresponds to 0, υ corresponds to 1, the pixel values less than τ are set to 0, and the pixel values higher than υ are set to 1. Then, use an edge extraction operator to extract the image edge of the reference image I, and also convert the gray value data type to the double type to obtain the edge image K. Superimpose the image J and the image K to form a new image H;

[0012] Step 4) Using image H as the reference image, calculate the normalized correlation coefficient matrix with the size of (N + M - 1) × (N + M - 1) by using the NCC algorithm. The calculation formula of the correlation coefficient is as follows. The maximum value in the matrix elements is the sub-aperture offset value required for wavefront reconstruction, where is the average gray value of the reference image pixels, is the average gray value of the single sub-aperture image pixels, is the real-time pixel gray value of the single sub-aperture image, I r (x, y) is the real-time pixel gray value of the reference image,

[0013]

[0014] The present invention has the following advantages compared with the prior art:

[0015] 1. The present invention realizes that when detecting the wavefront of an extended target, it can effectively improve the calculation accuracy and stability of the peak position of the correlation function of the NCC algorithm. The RMS (Root Mean Square) and PV value (the difference between the peak and valley: Peak to Valley) of the wavefront reconstruction residual are improved, and the wavefront reconstruction accuracy is effectively improved;

[0016] 2. The present invention combines image processing and algorithm optimization, and effectively improves the wavefront reconstruction accuracy without increasing the system complexity, providing a technical option for the application of image processing in wavefront detection, such as using the method of feature extraction to develop in the direction of establishing a deep learning model;

[0017] 3. The present invention reduces the requirements for the hardware of the wavefront detection optical system to a certain extent and improves the environmental adaptability of the system to extended targets by adjusting the reference image template parameters and using the edge extraction operator to process the reference image. Description of the Drawings

[0018] Figure 1 is the operation process of the image preprocessing method in the present invention for detecting an extended target;

[0019] Figure 2 is to detect an extended target, namely a six-rotor UAV with a diameter of about 600 mm;

[0020] Figure 3 is the position comparison of the reference images (images of the same six-wing UAV at different distances) in the sub-aperture before and after parameter adjustment (the large virtual frame from 0.6 km to 2 km is before adjustment, and the small virtual frame is after adjustment; the large virtual frame from 5 km to 10 km is after adjustment, and the small virtual frame is before adjustment); (a) 0.6 km, (b) 1.0 km, (c) 1.5 km, (d) 2.0 km, (e) 5.0 km, (f) 10.0 km;

[0021] Figure 4 It is a schematic diagram of the implementation process of the Laplacian operator for the edge extraction algorithm;

[0022] Figure 5 They are the original image (hexarotor UAV) and the reference images processed by 5 operators;

[0023] Figure 6 They are the comparisons of the RMS value and PV value of a certain restoration residual after using different operators. Specific implementation manners

[0024] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation manners.

[0025] For a certain extended target wavefront detection system, the number of sub-aperture units is 13×13, and the size of a single-aperture image is 64×64.

[0026] As Figure 1 shown, the specific implementation method of an extended target Hartmann wavefront detection method based on reference image preprocessing of the present invention is as follows:

[0027] Step 1) Obtain a sub-aperture image array from a Hartmann sensor with 13×13 sub-aperture units. Take the sub-aperture at the center of the Hartmann sub-aperture array (the row number and column number of this sub-aperture are both 6) as the selection position of the reference image I. The size of a single-aperture image is 64×64, and the single-pixel gray value is represented by denoted as, is the offset of the center pixel point of a single sub-aperture image and the center pixel point of the reference image I in the horizontal and vertical directions. The pixel size of the reference image I is M×M, and the single-pixel gray value is represented by I r (x, y).

[0028] Step 2) Denote the ratio of the starting position coordinate value of the reference image I in the horizontal and vertical directions within the central sub-aperture to the pixel size of the central sub-aperture in the horizontal and vertical directions as TemplateL, and denote the ratio of the pixel size in the horizontal and vertical directions to the pixel size of the central sub-aperture in the horizontal and vertical directions as TemplateR. The values of TemplateL and TemplateR are determined according to the number of pixels occupied by the extended target in the Hartmann sub-aperture and the background situation of the sub-aperture image; for example, select a hexarotor UAV with a diameter of about 600 mm (as Figure 2 shown), study the change of the relevant Hartmann wavefront detection accuracy with the change of the target distance, and its optimal parameter adjustment range is shown in Table 1, and the reference image selection area is as Figure 3 shown.

[0029] Table 1

[0030]

[0031] Step 3) After converting the pixel grayscale value data format of the reference image I to the double type, perform normalization, that is, divide the single-pixel grayscale value by the maximum grayscale value of the entire image, so that the grayscale value range is between [0, 1]. Then, the grayscale values in the range of [0.1, 0.9] are converted to the range of [0, 1] through a mapping relationship. The mapping relationship formula is as follows:

[0032]

[0033] After conversion, the pixel value 0.1 corresponds to 0, 0.9 corresponds to 1, the pixel values less than 0.1 are set to 0, and the pixel values higher than 0.9 are set to 1, obtaining the converted image J of the reference image. Then, use an edge extraction operator to extract the image edge of the reference image I, and also convert the grayscale value data type to the double type, obtaining the edge image K. The reference image after being processed by 5 operators and the original reference image are as Figure 5 shown. Superimpose image J and image K to form a new image H. The entire processing process is as Figure 4 shown.

[0034] Step 4) Using image H as the reference image, calculate the normalized correlation coefficient matrix with the size of (N + M - 1) × (N + M - 1) by using the NCC algorithm. The correlation coefficient calculation formula is as follows, where is the average grayscale value of the reference image pixels, and is the average grayscale value of the single-aperture image pixels. The maximum value in the elements of the correlation coefficient matrix is the required centroid offset value of the sub-aperture,

[0035]

[0036] Step 5) Obtain the centroid offset of the relevant Hartmann sub-aperture, use the modal method to restore the wavefront, and calculate the restoration residual between the restored wavefront and the wavefront to be corrected. The comparison of the RMS value and PV value of a certain restoration residual after using different operators is as Figure 6 shown.

[0037] In this implementation scheme, for a variable extent target, the change in the extent will affect the wavefront restoration accuracy of the relevant Hartmann. The change in the extent causes the number of pixels of the target in the sub-aperture image to change. When the number of pixels decreases from 55 × 47 to 6 × 5, the average values of the restoration residuals RMS and PV in the simulation are reduced by 54.1% and 51.3% respectively. Through the optimization of the reference image parameters, the average values of the restoration residuals RMS and PV under different extents are reduced by 8.5% and 5.7% respectively. Further use the edge extraction operator (Canny operator) to process the reference image, so that the average values of the restoration residuals RMS and PV under different extents are reduced by 7.8% and 6.2% respectively.

[0038] The wavefront restoration algorithm described above may also be the regional method, the direct slope method, etc.

[0039] This image preprocessing method may also establish a deep learning model to effectively estimate the extended target and pre-match the template.

[0040] The content not detailedly described in the specification of the present invention belongs to the prior art well-known to those skilled in the art.

Claims

1. An extended object Hartmann wavefront detection method based on reference image preprocessing, characterized in that: The implementation steps of this method are as follows: Step 1) Obtain a sub-aperture image array from a Hartmann sensor with a sub-aperture unit of (2P - 1)×(2P - 1) or 2P×2P. Select the sub-aperture at the center of the Hartmann sub-aperture array as the position for the reference image I. The corresponding number of rows and columns of this sub-aperture is P. The pixel size of the reference image I is M×M, and the single-pixel gray value is represented by , where (x, y) is the two-dimensional coordinate value of the pixel. The size of a single-aperture image is N×N, and the single-pixel gray value is represented by . is the offset of the center pixel of the single sub-aperture image from the center pixel of the reference image I in both the horizontal and vertical directions, and N≥M; In step 2), the ratio of the starting position coordinate values of the reference image I in the horizontal and vertical directions within the central sub-aperture to the pixel sizes of the central sub-aperture in the horizontal and vertical directions is denoted as TemplateL, and the ratio of the pixel sizes in the horizontal and vertical directions to the pixel sizes of the central sub-aperture in the horizontal and vertical directions is denoted as TemplateR. The actual values of TemplateL and TemplateR are determined according to the number of pixels occupied by the extended target in the Hartmann sub-aperture and the background situation of the sub-aperture image; Step 3) Convert the pixel grayscale value data format of the reference image I to double type and perform normalization processing to make the grayscale value range between [0, 1]; then the grayscale values in the range of are converted to the range of [0, 1] through the mapping relationship, where is the lower limit of the pixel value in the conversion interval, is the upper limit of the pixel value in the conversion interval, , after conversion, the pixel value corresponds to 0, corresponds to 1, the pixel values less than are set to 0, and the pixel values higher than are set to 1 to obtain the converted image J of the reference image. Then, use the edge extraction operator to extract the image edge of the reference image I, and also convert the grayscale value data type to double type to obtain the edge image K. Superimpose the image J and the image K to form a new image H; Step 4) Using image H as the reference image, calculate the normalized correlation coefficient matrix with the size of (N+M-1)×(N+M-1) by using the normalized correlation algorithm NCC. The correlation coefficient calculation formula is shown in Equation (1), where is the average gray value of the reference image pixels, is the average gray value of the single-aperture image pixels, is the real-time pixel gray value of the single-aperture image, is the real-time pixel gray value of the reference image. The maximum value in the elements of the correlation coefficient matrix is the required offset value: （1）。 2. The extended object Hartmann wavefront detection method based on reference image preprocessing according to claim 1, characterized in that: The normalization processing method of the reference image I described in step 3 is to divide the single-pixel gray value by the maximum gray value of the entire image.

3. The extended object Hartmann wavefront detection method based on reference image preprocessing according to claim 1, characterized in that: The gray value mapping relationship described in step 3 is shown in Equation (2). （2） Among them, t is the normalized pixel gray value before conversion, and s is the pixel gray value after conversion, where is the lower limit of the pixel value in the conversion interval, is the upper limit of the pixel value in the conversion interval, .

4. The extended object Hartmann wavefront detection method based on reference image preprocessing according to claim 1, characterized in that: The edge extraction operators described in step 3 include first-order operators and second-order operators. The first-order operators are the Sobel operator, Roberts operator, and Prewitt operator, and the second-order operators are the Canny operator and Laplacian operator. Among them, The Sobel operator realizes image edge extraction by calculating the first-order gradient approximation value of the luminance function; The Roberts operator calculates the gradient by using the oblique deviation difference method. Its direction is fixed and perpendicular to the edge. The edge strength is the calculation result of the gradient. The gradient expression can be approximated as: （3） Represents a certain pixel point in the image, is the gradient operator in the x direction, is the gradient operator in the y direction; The Prewitt operator uses templates in the up, down, left, and right four directions to perform neighborhood convolution with the image. The edge has two directions, horizontal and vertical, and is detected using the horizontal and vertical templates respectively; The processing steps of the Canny operator are divided into three steps: the first step is to remove image noise using a Gaussian smoothing filter; the second step is to calculate the gradient magnitude and direction of each pixel point in the image using the Sobel operator; the third step is to apply non-maximum suppression and double-threshold detection to determine the true and potential edges of the image, and finally complete edge extraction by suppressing isolated weak edges; The function of the Laplacian operator is to enhance the regions with sudden gray value changes in the image and weaken the regions with slow gray value changes.

5. The extended object Hartmann wavefront detection method based on reference image preprocessing according to claim 4, characterized in that: Before using the Laplacian operator, the image needs to be smoothed first.

Images