An automatic focusing method based on pixel-level aberration detection

Abstract

The invention discloses an autofocus method based on pixel-level aberration detection, which belongs to the field of image processing technology. The method mainly uses Fraunhofer diffraction to calculate the pupil function and obtain the amplitude point spread function, then Fourier transforms the amplitude point spread function to obtain the amplitude transfer function of the coherent imaging system, and finally squares the amplitude transfer function to obtain the intensity point spread function of the incoherent imaging system. Drawing on the relationship between the pupil function and the point spread function in wavefront coding, a point spread function model at each pixel position in the space-variable system is constructed, thereby converting the estimation problem of the aberration in the autofocus into the estimation problem of the polynomial coefficients of the point spread function at each pixel position.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an automatic focusing method based on pixel-level aberration detection. Background Art

[0002] The number of photosensitive cells in digital sensors has increased rapidly with the improvement of light perception capabilities. However, due to inevitable aberrations such as assembly errors and incorrect focus settings, it is difficult for the light emitted from the object point to converge on the camera's photosensitive cells, which will form a diffuse light spot and cause image blur. The point spread function (PSF) describes the distribution of light emitted by a single object point on the image plane after passing through the lens. The PSF covers both the structural information of the lens and the depth information of the scene. Estimating the PSF of an imaging system can be applied to 3D microscopy, autofocus, image deblurring, adaptive optics and astronomical imaging, and monocular depth estimation. Summary of the invention

[0003] In view of the defects of the prior art, the present invention provides an auto-focusing method based on pixel-level aberration detection, which is used to solve the problem of the increasing gap between the imaging model and the actual situation in the prior art.

[0004] To achieve the above-mentioned object and other related objects, the present invention provides an automatic focusing method based on pixel-level aberration detection, comprising:

[0005] The pixel arrangement of the image samples is randomly shuffled using a histogram-invariant spatial confusion method to obtain a fully focused image. For any given image I, the row and column indices of the image are randomly shuffled and recombined, and the pixel positions are arranged according to the recombined indices, while keeping the RGB channels unchanged;

[0006] After setting the aberration range, polynomial fitting is performed on the aberration using, but not limited to, Zernike polynomials and Seidel aberrations;

[0007] The point spread function of each pixel position is calculated based on the corresponding relationship between the aberration and the point spread function, that is, the point spread function of each pixel position is calculated based on the corresponding relationship between the aberration and the point spread function, that is, the point spread function is obtained by performing Fourier transformation on the pupil function and then taking the square of the modulus;

[0008] According to the linear invariance of the optical system to the intensity of the light wave field and the superposition theorem, the image of the object can be expressed as the weighted sum of two-dimensional pulse functions using the object field;

[0009] The blurred image obtained by simulation is used as input, and the polynomial coefficients of each pixel position are used as labels to train a neural network. The output feature map of the neural network is consistent with the input, and each position of the feature map records the defocus amount of the pixel at the corresponding position.

[0010] The defocus amount of each pixel position is output by the neural network, and the movement amount of the lens assembly is obtained using methods including but not limited to PID control and lens parameter conversion.

[0011] As described above, an autofocus method based on pixel-level aberration detection of the present invention has the following beneficial effects: the present invention can automatically generate fully focused and defocused image pairs with spatially varying defocus amount annotations in large quantities, and at the same time, aberrations including but not limited to distortion, spherical aberration and coma are added to the simulation to improve the quantity and annotation quality of fully focused images and defocused image pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flowchart for modeling point spread functions;

[0013] Figure 2 This is a system workflow diagram provided for Example 1 of the present invention. DETAILED DESCRIPTION

[0014] In order to make the technical solutions and advantages of the present invention more clearly understood, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application.

[0015] The present invention provides an automatic focusing method based on pixel-level aberration detection, which specifically comprises the following steps:

[0016] Step S10, obtain three-channel images of any natural scene to form a sample set of images to be processed, and use a histogram-invariant spatial obfuscation method to process the images to be processed to produce a sample set of aberration-free images. To obtain image samples, you can use a camera to take pictures or download public data sets to obtain image samples with rich styles, and use a histogram-invariant spatial obfuscation method to randomly shuffle the pixel arrangement of the image samples to obtain a fully focused image. For any given image, randomly shuffle the row and column indexes of the image and recombine them, and arrange the pixel positions according to the recombined indexes, while keeping the RGB channels unchanged;

[0017] Step S20, using random numbers to generate polynomial coefficients of the aberration at each pixel position, wherein the polynomial coefficients include but are not limited to Zernike polynomials and Seidel aberrations. After setting the aberration range, polynomial fitting is performed on the aberration using including but not limited to Zernike polynomials and Seidel aberrations;

[0018] Step S30: Figure 1 As shown in the figure, the pupil function is calculated by Fraunhofer diffraction and the amplitude point spread function is obtained. Then, the amplitude point spread function is Fourier transformed to obtain the amplitude transfer function of the coherent imaging system. Finally, the amplitude transfer function is modulo squared to obtain the intensity point spread function of the incoherent imaging system. Drawing on the relationship between the pupil function and the point spread function in wavefront coding, the PSF model at each pixel position in the space-variant system is constructed, thereby converting the aberration estimation problem into the estimation problem of the PSF polynomial coefficients at each pixel position.

[0019] Step S40, representing the image of the object point as a weighted sum of two-dimensional impulse functions, multiplying the light intensity value of each pixel by the corresponding point spread function according to the aberration-free image and the intensity point spread function of each pixel position, and finally superimposing them in sequence according to the pixel position to obtain an image of the scene;

[0020] Step S50: Using the blurred image as input and the polynomial coefficients of each pixel position as labels to train a neural network, such as Figure 2 As shown in the figure, given a single image, the neural network-based model PSFNet is used to estimate the Zernike coefficient map or Seidel coefficient map of the PSF of each pixel in the input image. The number of output channels of the CNN model is equal to the order of the aberration. During the model training process, the input image is converted into a fully focused image using the histogram-invariant spatial aliasing sampling method. During the model inference test process, the image of the scene to be focused is directly input without converting it into a fully focused image.

[0021] Step S60: The image of the desired focus scene is transmitted to the neural network, the channel corresponding to the defocus aberration in the output feature map is extracted, and then the value of the corresponding position of the focus area is taken out to convert the movement amount of the lens assembly.

[0022] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An automatic focusing method based on pixel-level aberration detection, characterized in that: include: 1) Obtaining three-channel images of any natural scene to form a sample set of images to be processed, and using a histogram-invariant spatial confusion method to process the images to be processed to produce a sample set of images without aberration; 2) using random numbers to generate polynomial coefficients of the aberration at each pixel position, wherein the polynomial coefficients include but are not limited to Zernike polynomials and Seidel aberrations; 3) The pupil function is calculated using Fraunhofer diffraction to obtain the amplitude point spread function, which is then Fourier transformed to obtain the amplitude transfer function of the coherent imaging system. Finally, the amplitude transfer function is modulo squared to obtain the intensity point spread function of the incoherent imaging system. 4) The image of the object point is represented as a weighted sum of two-dimensional impulse functions. According to the aberration-free image and the intensity point spread function of each pixel position, the blurred image is calculated by sliding addition of the product of each pixel value of the aberration-free image and the corresponding point spread function; 5) Using the blurred image as input and the polynomial coefficients of each pixel position as labels to train a neural network; 6) The image of the desired focus scene is passed into the neural network, the numerical value of the position corresponding to the focus area in the output feature map is extracted, and the movement amount of the lens assembly is calculated.

2. The automatic focusing method based on pixel-level aberration detection according to claim 1, characterized in that: The pixel arrangement of the image samples is randomly shuffled using a histogram-invariant spatial confusion method to obtain a fully focused image. For any given image I, the row and column indices of the image are randomly shuffled and recombined, and the pixel positions are arranged according to the recombined indices while keeping the RGB channels unchanged.

3. The automatic focusing method based on pixel-level aberration detection according to claim 1, characterized in that: After the aberration range is set, polynomial fitting is performed on the aberration using methods including but not limited to Zernike polynomials and Seidel aberrations.

4. The automatic focusing method based on pixel-level aberration detection according to claim 1, characterized in that: The point spread function of each pixel position is calculated based on the corresponding relationship between the aberration and the point spread function. That is, the point spread function is obtained by performing Fourier transformation on the pupil function and then taking the square of the modulus.

5. The automatic focusing method based on pixel-level aberration detection according to claim 1, characterized in that: The image of the scene is represented by the object field as a weighted sum of two-dimensional impulse functions, that is, the light intensity value of each pixel is multiplied by the corresponding point spread function, and finally the image of the scene is obtained by superimposing them in sequence according to the pixel position.

6. The automatic focusing method based on pixel-level aberration detection according to claim 1, characterized in that: The blurred image obtained by simulation is used as input, and the polynomial coefficients of each pixel position are used as labels to train a neural network. The output feature map of the neural network is consistent with the input, and each position of the feature map records the defocus amount of the pixel at the corresponding position.

7. The automatic focusing method based on pixel-level aberration detection according to claim 1, characterized in that: The neural network outputs the defocus amount at each pixel position, and the defocus amount is converted into the movement amount of the lens assembly using methods including but not limited to PID control and lens parameter conversion.

Images