Simple lens point spread function estimation method

Abstract

The application discloses a simple lens point spread function estimation method, comprising the following steps: a point spread function generated according to the characteristics of a single lens is applied to a clear image to obtain a simulated single lens blurred image; the single lens blurred image is subjected to Fourier transform into a frequency domain, serving as an input of an end-to-end neural network to generate an estimated point spread function; the estimated point spread function and the generated point spread function are calculated to obtain a loss function, and the parameters of the end-to-end neural network are updated with training iteration; and a final estimated point spread function is obtained. The application makes accurate parameterized modeling and generation of the single lens point spread function, so that the generated point spread function is more in line with the characteristics of the single lens, a series of image processing technical features are provided, and finally, a more accurate point spread function is obtained by using a generative adversarial neural network model.

Description

A simple method for estimating the spread function of a lens point Technical Field

[0001] This invention relates to the field of digital image processing, specifically to a point spread function (PSF) estimation method applied to simple lens imaging. Background Technology

[0002] In recent years, with the continuous development of computational photography technology and optical design, simple lens computational imaging technology has gradually become a new research direction. Simple lens imaging refers to the phenomenon that light forms a real image on the focal plane through a single lens. Compared with traditional complex lens imaging, simple lens imaging can significantly reduce the economic cost of the imaging system, but the imaging effect is affected by the fact that the light path of the simple lens cannot converge to a single point on the focal plane, resulting in a blurry image. The point spread function is a function representing the image formed on the focal plane by light emitted from a point source through a lens. Estimating the point spread function of the simple lens at different imaging block positions based on the blurred image obtained by the lens is one of the preliminary steps for image restoration of the blurred image.

[0003] Regarding single-lens computational imaging, different methods have been proposed to estimate the PSF (Power Sequence Filter) of a single-lens optical imaging system. Patent ZL.201410064041.7 proposes a fast PSF calibration method for single-lens imaging. This method calculates the average of N PSF values ​​as a PSF template for a certain type of single lens, and uses this PSF template as the initial value for the PSF correction process to quickly estimate the single-lens PSF. Patent ZL.201510222290.9 proposes a symmetry-based fast PSF calibration method for single-lens computational imaging. This method utilizes the spatial symmetry of the single-lens PSF, using the estimated PSF of an image patch as the initial value for estimating the PSF of its symmetrical image patch, thereby reducing the iterative process required for PSF estimation and reducing the PSF estimation time. Patent ZL.201510379305.2 proposes a single-lens computational imaging PSF estimation method based on sparse representation. This method represents the sharp image in the objective function as the product of an overcomplete dictionary and sparse coefficients, and constrains the sparse coefficients. Then, through an iterative optimization algorithm, the blur kernel, the overcomplete dictionary, and the sparse coefficients are estimated alternately to estimate the PSF of the single lens.

[0004] With the continuous development of single-lens computational imaging technology and the increasing demands for image quality, existing methods, while capable of estimating the PSF (Power Spread Function) of a single lens, no longer meet the practical requirements of single-lens computational imaging. Simple lens imaging generally exhibits characteristics of small distortion in the central region and large distortion in the peripheral region. Therefore, a single-point spread function cannot be used to model the entire simple lens image; instead, the image must be divided into blocks and modeled individually based on the distance from the imaging region to the image center. Summary of the Invention

[0005] To overcome the problems in the prior art, this invention aims to provide a simple lens point spread function estimation method to solve the problem that existing PSF estimation methods are not accurate enough.

[0006] The objective of this invention is achieved through the following technical solution: a method for optimizing the attacker's strategy in complex network games based on intuitionistic fuzzy logic, the method comprising:

[0007] A simple method for estimating the spread function of a lens point includes the following steps:

[0008] Step 1: Apply the point spread function generated based on the characteristics of a single lens to the sharp image to obtain a simulated blurred image of a single lens;

[0009] Step 2: The single-lens blurred image is Fourier transformed to the frequency domain and used as the input to the end-to-end neural network to generate the estimated point spread function.

[0010] Step 3: Calculate the loss function using the estimated point spread function and the generated point spread function, and update the end-to-end neural network parameters with each training iteration.

[0011] Step 4: Obtain the final estimated point spread function.

[0012] Specifically, the point spread function generation process described in step 1 includes the following steps:

[0013] Parametric modeling and generation of the point spread function of a single lens;

[0014] Let n be the number of angles of the generated star-shaped polygon, and D be the outer diameter. o ∈R n The inner diameter is D i ∈R n The deviation angle is A r ∈R n Let the k-th dimensions of the outer diameter and inner diameter be respectively... and Initialize the outer diameter using a uniformly distributed random distribution: The formula for initializing the inner diameter is: Where r ~ U(0.1,0.3) is randomly generated. This represents the (l-1)th dimension of the outer diameter, and the deviation angle is initialized using a uniform distribution.

[0015] Calculate the vertex coordinates {P} of the star-shaped polygon based on the parameters. k =(x k ,y k x |k=1,…,2n},x k Let y represent the x-coordinate of the k-th vertex. k Let A0 ∈ R be the ordinate of the k-th vertex, and let A0 ∈ R be the polar coordinate angle of the vertex of the star-shaped polygon. 2n+1 ,and Let A ∈ R be the polar coordinate angle of the 0th vertex, also known as the initial polar coordinate angle. 2n+1 ,and A k This represents the k-th dimension of the polar coordinate angle A. This represents the difference between the k-th dimension of the polar coordinate angle A and the initial polar coordinate angle, and is obtained through random generation. When k is an even number, the formula for calculating the vertex coordinates of the generated star-shaped polygon is as follows:

[0016]

[0017] When k is odd, the calculation formula is as follows:

[0018]

[0019] Filter out the star-shaped polygon S P The set of points inside G = {(i,j)|(i,j)∈S} p}, where i represents the x-coordinate of a point inside the star-shaped polygon, and j represents the y-coordinate of a point inside the star-shaped polygon;

[0020] Model the energy at the point: E(i,j)=αexp(-β‖i 2 +j 2 ‖), where α and β are undetermined constants, ‖·‖ represents calculating the absolute value, and the point energy should satisfy the constraint that the total energy sum is 1, i.e., ∏ (i,j)∈G E(i,j) = 1;

[0021] Therefore, we have: α=1 / (∏) (i,j)∈G exp(-β‖i 2 +j 2 The undetermined constant β characterizes the degree of energy dispersion of the point spread function;

[0022] After the undetermined constants are determined, the energy distribution of the point spread function within the star-shaped polygon is calculated, and the generated point spread function {E(i,j)|i,j=1,…,m} is obtained, where m∈N + Let be the size of the point spread function.

[0023] Preferably, the generated point spread function is the convolution of the point spread function energy distribution and the smooth convolution kernel κ: PSF = E * κ.

[0024] Preferably, smooth convolution kernel

[0025] The undetermined coefficients β ~ U (0.004, 0.008).

[0026] Specifically, during the process of applying the point spread function (PSF) generated based on the characteristics of a single lens to the sharp image, the sharp image block y is convolved with the generated PSF to obtain the simulated single-lens blurred image block x, i.e.: x = y * PSF; during the process of the single-lens blurred image undergoing Fourier transform to the frequency domain, the blurred image block x is first converted to a frequency domain image through a two-dimensional discrete Fourier transform, and the zero-frequency point is moved to the center of the spectrum: X = fftshift(fft2(x)), where fft2(·) represents the two-dimensional discrete Fourier transform, and fftshift(·) represents moving the zero-frequency point to the center of the spectrum, and then the power spectral density is calculated: The functions real(·) and imag(·) represent extracting the real and imaginary parts of the frequency domain image, respectively.

[0027] Specifically, the input of the end-to-end neural network is the power spectral density of a certain block of frequency domain blurred image, and the output is the point spread function estimated by a certain block of a single lens.

[0028] The end-to-end neural network is trained using a generative adversarial approach, where the generator employs a U-Net structure, denoted as G. θ (·), where θ is the set of model parameters, and the loss function for generator training is:

[0029] L G =λ‖G θ (S)-PSF‖1+PSF·(G θ (S)-PSF) 2 +‖D φ (G θ (S))-1‖2

[0030] Where the first term is the fidelity term, λ is the coefficient, and G θ(S) represents a U-Net with parameter set θ, and ||·||1 represents the computation of the 1-norm; the second sub-item is the shape prior, highlighting the influence of the ground truth PSF on the model output, where PSF represents the generated point spread function; the third sub-item is the truth term, which is determined by the discriminator D. φ (·) is the difference between the squared difference between the discriminant result and 1, which is the difference between the result generated by the generator and the true value PSF. ‖·‖2 represents the calculation of the 2-norm; where φ is the set of model parameters of the discriminant.

[0031] Discriminator D φ (·) Using the ResNet-34 residual classification network, the training loss function is:

[0032] L D =||D φ (PSF)-1‖2+‖D φ (G θ (S))-0‖2

[0033] The first term causes the discriminator to tend to recognize the true PSF as real, while the second term causes the discriminator to tend to recognize the generator G. θ The generated PSF (·) is false.

[0034] Preferably, λ = 100.

[0035] Specifically, the end-to-end neural network model is trained using stochastic gradient descent with a batch size of 8, the Adam operator is used for optimization, and the initial learning rate is 1e. -4 Every 20,000 iterations, the learning rate is reduced to 1 / 10 of its current value, until the learning rate decreases to 1e. -7 The total number of training iterations is 200,000. During each iteration of optimization, the generator loss function is calculated first, and the generator parameters θ are updated by calculating the gradient through the optimization operator. Then, the discriminator loss function is calculated, and the generator parameters φ are updated by calculating the gradient through the optimization operator.

[0036] Furthermore, the segmented single-lens image is converted into a grayscale image, and then the power spectral density of the grayscale image is input into the trained generator G. θ (·), the output is the estimated point spread function of each sub-block of the single-lens image. Attached Figure Description

[0037] Figure 1 shows a flowchart of an embodiment of the present invention;

[0038] Figure 2 shows a single-lens structure diagram according to an embodiment of the present invention;

[0039] Figure 3 shows a block division diagram of single-lens imaging according to an embodiment of the present invention;

[0040] Figure 4 shows a schematic diagram of the point spread function estimation process in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0042] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] As shown in Figure 1, a simple method for estimating the spread function of a lens point includes the following steps:

[0044] Step 1: Apply the point spread function generated based on the characteristics of a single lens to the sharp image to obtain a simulated blurred image of a single lens;

[0045] Step 2: The single-lens blurred image is Fourier transformed to the frequency domain and used as the input to the end-to-end neural network to generate the estimated point spread function.

[0046] Step 3: Calculate the loss function using the estimated point spread function and the generated point spread function, and update the end-to-end neural network parameters with each training iteration.

[0047] Step 4: Obtain the final estimated point spread function.

[0048] As shown in Figure 2, simple lens imaging can significantly reduce the economic cost of the imaging system, but the imaging effect is affected by the fact that the light paths of a simple lens cannot converge to a single point on the focal plane, resulting in a blurry image. Estimating the point spread function of the simple lens at different imaging block positions based on the blurred image obtained through the lens is one of the preliminary steps for image restoration of the blurred image.

[0049] The image segmentation method in this embodiment is shown in Figure 3. First, the 4:3 aspect ratio single-lens imaging is divided into 12×9 blocks, and then blur kernel estimation is performed block by block.

[0050] The single-lens point spread function estimation process proposed in this embodiment is shown in Figure 4. The point spread function generated based on the characteristics of a single lens is applied to a sharp image to obtain a simulated blurred image. This image is then Fourier transformed to the frequency domain, and its real part is extracted and used as input to an end-to-end neural network to generate the estimated point spread function. The estimated point spread function and the generated point spread function are used to calculate the loss function, which drives the updating of the neural network parameters during training iterations.

[0051] Point spread function (PSF) generation. Due to manufacturing precision issues, the actual measured results of the block PSF of a simple lens exhibit an irregular "quadrilateral star" shape. Therefore, it is necessary to perform parametric modeling and generation of the PSF for a single lens. Let n be the number of corners of the generated star-shaped polygon, and D be the outer diameter. o ∈R n The inner diameter is D i ∈R n The deviation angle is A r ∈R n Let the k-th dimensions of the outer diameter and inner diameter be respectively... and The outer diameter is initialized randomly using a uniform distribution.

[0052]

[0053] The initial calculation of the inner diameter is as shown in formula (2).

[0054]

[0055] Where r ~ U(0.1,0.3) is randomly generated. The deviation angle A i Similar to the outer diameter, the uniform distribution initialization is adopted using formula (1).

[0056] Calculate the vertex coordinates {P} of the star-shaped polygon based on the parameters. k =(x k ,y k Let A0 ∈ R be the polar coordinate angle of the vertex of the star-shaped polygon, where k = 1, ..., 2n. 2n+1 ,and

[0057]

[0058] Let the polar coordinate angles of the vertices of the generated star-shaped polygon be A∈R. 2n+1 and

[0059]

[0060] The vertex coordinates of the generated star-shaped polygon are calculated using formula (5) when k is even.

[0061]

[0062] When k is odd, use formula (6) to calculate.

[0063]

[0064] Generate a point spread function {E(i,j)|i,j=1,…,m.} based on the vertex coordinates of the star-shaped polygon, where m∈N. + Let be the size of the point spread function. First, select points within the star-shaped polygon S. P The set of points inside G = {(i,j)|(i,j)∈S} p Secondly, model the energy at the points:

[0065] E(i,j)=αexp(-β‖i 2 +j 2 (7)

[0066] Where α and β are undetermined constants. The potential energy should satisfy the constraint that the total energy sum is 1, i.e.

[0067] ∏ (i,j)∈G E(i,j)=1 (8)

[0068] Substituting formula (7) into (8) yields...

[0069] α=1 / (π) (i,j)∈G exp(-β‖i 2 +j 2 ‖)) (9)

[0070] The undetermined constant β characterizes the degree of dispersion of the point spread function energy, and is usually taken as β ~ U(0.004, 0.008). Once the constant β is determined, the constant α can be calculated according to formula (8), and then the point spread function energy distribution within the star-shaped polygon can be calculated according to formula (7).

[0071] We utilize a smooth point spread function (PSF) generated by convolution. The PSF generated in the previous steps may have discontinuous edges; therefore, we construct a smooth convolution kernel κ. The final PSF is the convolution of the energy distribution E and the smooth convolution kernel κ.

[0072] PSF=E*κ (10)

[0073] Where the convolution kernel is taken

[0074]

[0075] Input data processing for the neural network model. First, the sharp image block y is convolved with the point spread function (PSF) generated in step S3 to obtain the blurred image block x for simulated single-lens imaging, i.e.

[0076] x=y*PSF (12)

[0077] Secondly, the power spectral density of a blurred image formed by a single lens in the frequency domain is correlated with the shape of the point spread function of the single lens. Therefore, the blurred image is divided into blocks x, which are first converted into frequency domain images through a two-dimensional discrete Fourier transform (fft2), and the zero-frequency point is moved to the center of the spectrum (fftshift).

[0078] X = fftshift(fft2(x)) (13)

[0079] Then calculate its power spectral density.

[0080]

[0081] The functions real(·) and imag(·) represent extracting the real and imaginary parts of the frequency domain image, respectively.

[0082] S5. Structure and Training Method of the Neural Network Model. The input to the neural network model is the power spectral density S of a certain block of blurred image in the frequency domain; the output is the estimated spread function of a certain block point of a single lens. The model is trained in a generative adversarial manner, and the generative neural network model adopts the U-Net structure, denoted as G. θ (·), where θ is the set of model parameters. The loss function for training the generator U-Net model is...

[0083] L G =λ‖G θ (S)-PSF‖1+PSF·(G θ (S)-PSF) 2 +‖D φ (G θ (S))-1‖2(15)

[0084] The first term is the fidelity term, with a coefficient λ = 100; the second term is the shape prior, highlighting the influence of the true PSF on the model output; the third term is the realism term, determined by the discriminator D. φ (·) The squared difference between the discrimination result and 1 measures the difference in accuracy between the generator network's generated result and the ground truth PSF; where φ is the set of model parameters for the discriminator. Discriminator D φ (·) This is implemented using the ResNet-34 residual classification network, whose training loss function is:

[0085] L D =||D φ (PSF)-1‖2+‖D φ (G θ (S))-0‖2(16)

[0086] The first term tends to make the model recognize the true PSF as true, and the second term tends to make the model recognize the generator G. θThe generated PSF (·) is false.

[0087] Model training and parameter optimization employed stochastic gradient descent with a batch size of 8. The Adam operator was used for optimization, and the initial learning rate was 1e. -4 Every 20,000 iterations, the learning rate is reduced to 1 / 10 of its current value, until the learning rate decreases to 1e. -7 The total number of training iterations is 200,000. During each iteration of optimization, the generator loss function is first calculated using formula (15), and the generator parameters θ are updated by calculating the gradient using the optimization operator. Then, the discriminator loss function is calculated using formula (16), and the generator parameters φ are updated by calculating the gradient using the optimization operator.

[0088] Point spread function estimation for a single lens. The segmented single-lens image is converted to a grayscale image, and then its power spectral density is input into a neural network G. θ (·), the output of the neural network is the estimated point spread function of each sub-block of the single-lens imaging image.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A simple method for estimating the spread function of a lens point, characterized in that, Includes the following steps: Step 1: The point spread function generated based on the characteristics of a single lens is applied to the sharp image to obtain a simulated blurred image of a single lens. Step 2: The blurred image of the single lens is Fourier transformed to the frequency domain and used as input to an end-to-end neural network to generate an estimated point spread function. Step 3: The estimated point spread function and the generated point spread function are used to calculate a loss function, and the parameters of the end-to-end neural network are updated with each training iteration. Step 4: The final estimated point spread function is obtained. The generation process of the point spread function in Step 1 includes the following steps: parametric modeling and generation of the single lens point spread function; assuming the number of angles of the generated star-shaped polygon is... The outer diameter is The inner diameter is The deviation angle is Record the first diameter of the outer diameter and the second diameter of the inner diameter. The dimensions are respectively and Initialize the outer diameter using a uniformly distributed random distribution: The formula for initializing the inner diameter is: ,in, Randomly generated, The outer diameter is represented by the first Each dimension, with the deviation angle initialized using a uniform distribution; the vertex coordinates of the star-shaped polygon are calculated based on the parameters. ， Indicates the first The x-coordinates of the vertices, Indicates the first The ordinate of each vertex is used to denote the polar coordinate angle of the vertex of the star-shaped polygon. ,and ， Indicates the first The polar coordinates of each vertex, also known as the initial polar coordinates, are denoted as the polar coordinates of the vertices of the generated star-shaped polygon. ,and ， Represents polar coordinate angle The One dimension, Represents polar coordinate angle The The difference between each dimension and the initial polar coordinate angle is obtained through random generation. When the vertex coordinates are even, the formula for calculating the vertex coordinates of the generated star-shaped polygon is as follows: when The formula for calculating odd numbers is as follows: Filtering out star-shaped polygons internal set of points ， Represents the x-coordinate of a point inside the star-shaped polygon. Represents the ordinate of the interior of the star-shaped polygon; models the energy at the points: ,in, For undetermined constants, This indicates that the absolute value of the energy at a point should satisfy the constraint that the total energy sum is 1. Therefore, we can conclude that: Undetermined constants The degree of energy dispersion of the point spread function is characterized; after the undetermined constant is determined, the energy distribution of the point spread function within the star-shaped polygon is calculated to obtain the generated point spread function; the generated point spread function is the point spread function energy distribution. With smooth convolution kernel Convolution: 。 2. The simple lens point spread function estimation method according to claim 1, characterized in that, Smooth convolution kernel 。 3. The simple lens point spread function estimation method according to claim 1, characterized in that, Undetermined coefficients 。 4. The simple lens point spread function estimation method according to claim 1, characterized in that, The point spread function, generated based on the characteristics of a single lens, is applied to a sharp image during the process of image segmentation. With the generated point spread function Perform convolution to obtain simulated single-lens blurred image blocks. ,Right now: ； During the process of transforming a blurred image from a single lens to the frequency domain using Fourier transform, the blurred image is divided into blocks. First, convert it to a frequency domain image using a two-dimensional discrete Fourier transform, and then move the zero-frequency point to the center of the spectrum: ， Represents the two-dimensional discrete Fourier transform. This means moving the zero-frequency point to the center of the spectrum and then calculating the power spectral density: , where the function and These represent the extracted real and imaginary parts of the frequency domain image, respectively.

5. A simple lens point spread function estimation method according to claim 4, characterized in that, The end-to-end neural network takes the power spectral density of a certain block of frequency-domain blurred image as input and outputs the point spread function estimated for a certain block by a single lens. The end-to-end neural network is trained using a generative adversarial approach, where the generator employs a U-Net structure, denoted as... ,in Given the set of model parameters, the loss function for generator training is: The first item is the fidelity item. For coefficients, The parameter set is U-Net, This indicates the calculation of the 1-norm; the second sub-item is the shape prior, highlighting the true value. Impact on model output This represents the generated point spread function; the third term is the truth value term, determined by the discriminator. The generator that produces the squared difference between the result and 1 is the same as the true value. Differences in authenticity This represents the calculation of the 2-norm; where The set of model parameters for the discriminator; the discriminator The ResNet-34 residual classification network is used, and the training loss function is: The first term causes the discriminator to tend to recognize the true PSF as real, while the second term causes the discriminator to tend to recognize the generator. The generated PSF is false.

6. The simple lens point spread function estimation method according to claim 5, characterized in that, 。 7. A simple lens point spread function estimation method according to claim 5, characterized in that, The end-to-end neural network model is trained using stochastic gradient descent with a batch size of 8. The Adam operator is used for optimization, and the initial learning rate is [missing information]. Every 20,000 iterations, the learning rate is reduced to 1 / 10 of its current value, until the learning rate is reduced to... ； The total number of training iterations is 200,000. During each iteration of optimization, the generator loss function is first calculated, and the generator parameters are updated by calculating gradients through optimization operators. Next, calculate the discriminator loss function and update the generator parameters by calculating gradients through optimization operators. 。 8. A simple lens point spread function estimation method according to claim 1, characterized in that, The segmented single-lens image is converted into a grayscale image, and then the power spectral density of the grayscale image is input into the trained generator. The output is the estimated point spread function of each sub-block of the single-lens image.

Images