Camera point spread function calibration method and device, electronic equipment and storage medium

Abstract

The invention relates to a camera point spread function calibration method and device, electronic equipment and a storage medium, and the method comprises the steps: employing a camera to shoot an actual image in a pre-constructed shooting environment, so as to obtain a corresponding captured image; rendering the original pattern to the captured image, and performing color balance on the original pattern and the actual image to obtain sample data meeting a preset matching condition; and constructing an optimization problem of the point spread function by using the sample data, and solving the optimization problem to obtain the point spread function meeting a preset accurate condition. Therefore, the technical problems that in the related technology, laser collimator detection is high in cost and strict in standard for the measurement environment, the small hole image analysis and measurement process is tedious, and the simplified process can only be applied to a specific scene or specific equipment and is difficult to apply and popularize are solved.

Description

Technical Field

[0001] This application relates to the field of digital image processing technology, and particularly relates to a method, device, electronic device, and storage medium for calibrating the point spread function of a camera. Background Art

[0002] In related technologies, in the field of microscopic imaging with higher precision requirements, a laser collimator or small-hole image analysis can be used to detect the point spread function. However, the former usually requires expensive measurement instruments and a harsh measurement environment, while the latter requires a cumbersome establishment of measurement standards and measurement processes. In some technologies, in order to simplify the measurement process of the point spread function, lens optical design or specific parameterization techniques are usually used to reduce the amount of modeling of the point spread function. However, such designs are usually only applicable to specific cameras. In addition, some technologies use degraded image data or multi-modal sensor measurements to predict or optimize the point spread function response at a certain field of view. However, such methods are usually applicable to situations where the scene illumination intensity changes little, and also require a complex layout of the sensor system, with poor scalability.

[0003] In summary, in related technologies, the cost is high, the standards for the measurement environment are harsh, the measurement process is cumbersome, and the simplified process can only be applied to specific scenarios or specific devices, making it difficult to promote and apply, and there is an urgent need for improvement. Summary of the Invention

[0004] This application provides a method, device, electronic device, and storage medium for calibrating the point spread function of a camera to solve the technical problems in related technologies, such as the high cost of laser collimator detection, the harsh standards for the measurement environment, the cumbersome measurement process of small-hole image analysis, and the simplified process can only be applied to specific scenarios or specific devices, making it difficult to promote and apply.

[0005] The first aspect of the embodiments of this application provides a method for calibrating the point spread function of a camera, including the following steps: using the camera to capture an actual image in a pre-constructed shooting environment to obtain a corresponding captured image; rendering the original pattern onto the captured image, and performing color balance on the original pattern and the actual image to obtain sample data that meets the preset matching conditions; using the sample data to construct an optimization problem for the point spread function, and solving the optimization problem to obtain a point spread function that meets the preset accuracy conditions.

[0006] Optionally, in an embodiment of this application, before using the camera to capture an actual image in a pre-constructed shooting environment, it further includes: obtaining the display screen parameters in the shooting environment; using the field of view angle, focal length of the camera, and the display screen parameters to determine whether the shooting environment meets the preset capture conditions; if the shooting environment meets the preset capture conditions, then using the camera to capture the actual image.

[0007] Optionally, in an embodiment of the present application, before using the camera to capture an actual image in a pre-constructed shooting environment, it further includes: using the camera to capture images of a target plane calibration board in multiple shooting scenarios; extracting feature point information of the images; using the feature point information to estimate the internal parameters and distortion coefficients of the camera; using the internal parameters and distortion coefficients of the camera to perform camera pose calibration so as to capture the actual image with the camera after pose calibration.

[0008] Optionally, in an embodiment of the present application, the rendering the original pattern onto the captured image and performing color balance on the original pattern and the actual image to obtain sample data that meets a preset matching condition includes: constructing a corresponding perspective camera parameter model based on the camera internal parameters; using the spatial pose of the shooting environment and the perspective camera parameter model to render the original pattern onto the captured image to obtain a rendered pattern after rendering; using the distortion coefficient to perform distortion adjustment on the rendered pattern so that the rendered pattern after distortion adjustment meets a preset camera distortion shooting condition; performing color balance on the rendered pattern after distortion adjustment and the actual pattern to obtain the sample data.

[0009] Optionally, in an embodiment of the present application, the constructing an optimization problem of the point spread function using the sample data and solving the optimization problem to obtain a point spread function that meets a preset accuracy condition includes: obtaining an initial solution of the point spread function by optical tracing; inputting the initial solution of the point spread function into the optimization problem and performing iterative calculation on the optimization problem until the point spread function that meets the preset accuracy condition is obtained.

[0010] An embodiment of the second aspect of the present application provides a camera point spread function calibration device, including: a first capture module, configured to use a camera to capture an actual image in a pre-constructed shooting environment to obtain a corresponding captured image; a processing module, configured to render an original pattern onto the captured image and perform color balance on the original pattern and the actual image to obtain sample data that meets a preset matching condition; a calibration module, configured to construct an optimization problem of the point spread function using the sample data and solve the optimization problem to obtain a point spread function that meets a preset accuracy condition.

[0011] Optionally, in an embodiment of the present application, it further includes: an acquisition module, configured to acquire display screen parameters in the shooting environment; a judgment module, configured to use the field of view angle, focal length of the camera, and the display screen parameters to judge whether the shooting environment meets a preset capture condition; a control module, configured to use the camera to capture the actual image when the shooting environment meets the preset capture condition.

[0012] Optionally, in an embodiment of the present application, it further includes: a second capture module, configured to capture images of the target plane calibration board in multiple shooting scenarios by using the camera; an extraction module, configured to extract feature point information of the images; an estimation module, configured to estimate the internal parameters and distortion coefficients of the camera by using the feature point information; and a calibration module, configured to perform camera pose calibration by using the internal parameters and distortion coefficients of the camera, so as to capture the actual images by using the camera with calibrated pose.

[0013] Optionally, in an embodiment of the present application, the processing module includes: a construction unit, configured to construct a corresponding perspective camera parameter model based on the internal parameters of the camera; a rendering unit, configured to render the original pattern onto the captured image by using the spatial pose of the shooting environment and the perspective camera parameter model to obtain a rendered pattern after rendering; an adjustment unit, configured to perform distortion adjustment on the rendered pattern by using the distortion coefficients, so that the rendered pattern after distortion adjustment meets the preset camera distortion shooting conditions; and a balancing unit, configured to perform color balancing on the rendered pattern after distortion adjustment and the actual pattern to obtain the sample data.

[0014] Optionally, in an embodiment of the present application, the calibration module includes: a first calculation unit, configured to obtain an initial solution of the point spread function by using optical tracing; and a second calculation unit, configured to input the initial solution of the point spread function into the optimization problem and perform iterative calculation on the optimization problem until the point spread function meeting the preset precise conditions is obtained.

[0015] An embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the camera point spread function calibration method as described in the above embodiment.

[0016] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the camera point spread function calibration method as described in the above embodiment.

[0017] An embodiment of the fifth aspect of the present application provides a computer program product, including a computer program, where when the computer program is executed, it is used to implement the camera point spread function calibration method as described above.

[0018] Embodiments of the present application can capture an actual image using a camera in a pre-constructed shooting environment to obtain a corresponding captured image, render an original pattern onto the captured image, and perform color balancing on the original pattern and the actual image to obtain sample data that meets preset matching conditions. Furthermore, an optimization problem of the point spread function is constructed and the optimization problem is solved to obtain a point spread function that meets preset accuracy conditions, so as to achieve point spread function calibration, optimize the measurement process while ensuring accuracy, and do not require expensive equipment. When applied, there are no special requirements for the scene or equipment, which is convenient for popularization and application. Thus, the technical problems in the related art are solved, such as the high cost of laser collimator detection, the strict standards for the measurement environment, the cumbersome measurement process of small hole image analysis, and the simplified process can only be applied to specific scenes or specific equipment and is difficult to be popularized and applied.

[0019] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the drawings, in which:

[0021] Figure 1 is a flowchart of a method for calibrating a camera point spread function according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a method for calibrating a camera point spread function according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of the principle of a method for calibrating a camera point spread function according to an embodiment of the present application;

[0024] Figure 4 is a schematic structural diagram of a device for calibrating a camera point spread function according to an embodiment of the present application;

[0025] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0027] The method, device, electronic device, and storage medium for calibrating the camera point spread function according to the embodiments of the present application will be described below with reference to the accompanying drawings. In view of the technical problems in the related art mentioned in the above background art, such as the high cost of laser collimator detection, the strict standards for the measurement environment, the cumbersome measurement process of small hole image analysis, and the simplified process can only be applied to specific scenarios or specific devices and is difficult to be popularized and applied, the present application provides a method for calibrating the camera point spread function. In this method, the camera can be used to capture an actual image in a pre-constructed shooting environment to obtain a corresponding captured image, the original pattern is rendered onto the captured image, and color balance is performed on the original pattern and the actual image to obtain sample data that meets the preset matching conditions. Then, an optimization problem of the point spread function is constructed and the optimization problem is solved to obtain a point spread function that meets the preset accuracy conditions, so as to achieve the calibration of the point spread function, optimize the measurement process while ensuring accuracy, without expensive equipment, and there are no special requirements for the scenario or device during application, which is convenient for popularization and application. Thus, the technical problems in the related art, such as the high cost of laser collimator detection, the strict standards for the measurement environment, the cumbersome measurement process of small hole image analysis, and the simplified process can only be applied to specific scenarios or specific devices and is difficult to be popularized and applied, are solved.

[0028] Specifically, Figure 1 is a flowchart of a method for calibrating the camera point spread function according to the embodiments of the present application.

[0029] As Figure 1 shown, the method for calibrating the camera point spread function includes the following steps:

[0030] In step S101, the camera is used to capture an actual image in a pre-constructed shooting environment to obtain a corresponding captured image.

[0031] It can be understood that the camera system can be regarded as a linear system, and its unit impulse response is also called the point spread function; according to the signal and system theory: in a linear camera imaging system, the image of an object can be calculated by representing the field on the object plane as a weighted sum of two-dimensional impulse functions. The point spread function is essentially the image of the impulse function and is caused by the diffraction of light and lens aberration. More specifically, the point spread function can represent the tracing and rendering result of a unit point light source on a certain plane in the object space on the image plane. An accurate point spread function can make the image blur model more accurate. When an accurate point spread function is obtained, an image deblurring algorithm with a known blur kernel (the point spread function in the current scenario) can be used for image deblurring.

[0032] Based on the above principle, the method for calibrating the camera point spread function according to the embodiments of the present application can first perform image capture to construct an optimization problem and calculate an accurate solution of the point spread function in the subsequent process.

[0033] Among them, before image capture, an embodiment of the present application requires a shooting environment. Specifically, the construction of the shooting environment will be elaborated below.

[0034] Optionally, in an embodiment of the present application, before using a camera to capture an actual image in a pre-constructed shooting environment, it further includes: obtaining the display screen parameters in the shooting environment; using the field of view angle, focal length, and display screen parameters of the camera to determine whether the shooting environment meets the preset capture conditions; if the shooting environment meets the preset capture conditions, then using the camera to capture the actual image.

[0035] In the actual execution process, in the shooting environment of the embodiment of the present application, a display screen and a camera are erected to display the original pattern on the display screen, and the camera is used to capture the actual image to obtain the captured image.

[0036] The embodiment of the present application can give the measurement distance {z i} of the point spread function. It is necessary to calculate the size and shooting distance of the corresponding display screen according to the field of view angle fov and focal length of the camera. The width of the display screen The length of the display screen At the same time, it is also necessary to ensure that under the capture of the unit pixel size on the display screen, it cannot be smaller than the unit pixel size of the camera, and it is best that the unit pixel size captured on the display screen is an integer multiple of the unit pixel size of the camera, otherwise undersampling textures will appear on the captured pattern. Define the length and width of the sensor as w s , h s , and the number of pixels is r s ×c s , the focal length is f, and the number of pixels of the display screen is R×C. Then it is necessary to ensure

[0037] Based on the above shooting environment, the embodiment of the present application can generate a certain number of coded aperture patterns according to the coded aperture principle, that is, the original patterns for registration, and display them on the display screen. Among them, according to the definition of the coded aperture pattern, the embodiment of the present application can sample and generate several coded aperture patterns, including but not limited to the frequency, width, repetition times of the coding appearance, and the rank of the corresponding defined source code.

[0038] Among them, the coded aperture is a very important technology in the fields of optical imaging and signal processing. It improves the sensitivity, signal-to-noise ratio, and resolution of the imaging system by specifically encoding light, can provide higher-quality images under restricted conditions, and provides more information for backend reconstruction. It is mainly used to improve the acquisition efficiency and quality of images, and has a wide range of applications especially under low-light or other restricted imaging conditions. Common application scenarios include astronomical imaging, low-light imaging, computational photography, and compressive sensing reconstruction, etc. Since the coded aperture can encode the positions with different aberrations in the image into unique patterns, researchers can use these patterns to deblur the blurred areas during post-processing. In addition, the coded aperture makes the acquired data contain richer frequency information. By decoding this information, super-resolution imaging can be achieved, significantly improving the image details.

[0039] MURA (Modified Uniformly Redundant Array) is a very classic mask in coded imaging, which can effectively reduce the sidelobe effect, thereby improving the contrast and resolution of the image. The MURA mask is usually generated by the MURA sequence, and the sequence is the quadratic residue sequence modulo a prime number. It can be expressed as:

[0040]

[0041] Therefore, the mask can be expressed as the two-dimensional extension of the MURA sequence:

[0042]

[0043] During the experiment, the embodiments of the present application can choose to generate a large-scale MURA pattern to increase the number of constraints of the optimization problem. Given a fixed prime number L, a single-frame MURA pattern will generate a mask matrix of size L×L. According to the physical size of the screen, resolution, camera field of view angle, and the shooting distance of the pattern, the pattern can be upsampled at a sampling rate of m for pixels, and then the template can be extended periodically with a period of s in the two-dimensional space, and boundary filling can be performed according to the number of physical pixels of the screen, and the filling amount is p. Then a template of size (p + smL)×(p + smL) can be obtained. For the prime number sequence {L i |L i =4m i +1, m i ∈Z *}, interval sampling can be performed, and any number of templates can be generated. In addition, corresponding calculations need to be performed according to the actual chip pixel size, measurement distance, and display screen size.

[0044] Generating a registration pattern using the coded aperture principle adds high-frequency signals compared to other registration patterns and contains richer restored signals in the frequency domain, enabling better convergence of the optimization problem.

[0045] Optionally, in an embodiment of the present application, before using a camera to capture an actual image in a pre-constructed shooting environment, it further includes: using the camera to capture images of a target plane calibration board in multiple shooting scenarios; extracting feature point information of the images; estimating the internal parameters and distortion coefficients of the camera using the feature point information; and calibrating the camera pose using the internal parameters and distortion coefficients of the camera to capture the actual image using the camera with calibrated pose.

[0046] Before shooting, the embodiment of the present application also needs to calibrate the camera and obtain the internal parameters of the camera to facilitate the construction of the optimization problem subsequently.

[0047] The embodiment of the present application can set up a display screen and a camera according to the sampled shooting environment, i.e., the scene distribution; prepare a pose calibration pattern, usually including a checkerboard, a dot grid, etc.; among them, it includes calibrating the internal camera representation parameters, distortion parameters, and the pose distribution of the object relative to the camera. Among them, the internal parameters of the camera are calibrated. The process of calibrating the internal camera representation parameters is as follows: The calibration pictures need to be taken with a calibration board at different positions, different angles, and different poses. At least 3 pictures are required, and 10 - 20 pictures are preferably used. The calibration board needs to be a checkerboard pattern composed of black and white rectangles, and the production accuracy requirements are relatively high; for each calibration picture, corner point information is extracted, and the function for finding checkerboard corner points needs to be used to extract the corner points. Here, the corner points specifically refer to the internal corner points on the calibration board, and these corner points do not touch the edge of the calibration board; for each calibration picture, sub-pixel corner point information is further extracted. Among them, in order to improve the calibration accuracy, pixel information needs to be further extracted on the basis of the initially extracted corner point information to reduce the camera calibration deviation; after obtaining the image coordinates of the internal corner points of the checkerboard calibration picture, the PnP algorithm (Point-n-Perspective) can be used to calculate the internal parameters of the camera, including the magnification in the horizontal and vertical directions of the camera, and the offsets in the horizontal and vertical directions. The distortion parameters describe the radial and tangential deformation distances from the center to the edge, and are represented by a linear polynomial, with the independent variable being the distance from the current pixel to the center pixel. After that, the capture module measures the pose of the display screen to the camera in all scenarios, including the 3-degree-of-freedom rotation amount and the 3-degree-of-freedom displacement amount. The PnP problem converts the problem of solving the rotation amount and displacement amount into a system of equations constrained by multiple cosine theorems, and finally can be solved by means of mathematical modeling or constrained sampling to obtain a stable solution value.

[0048] To achieve the alignment of the scene and the camera coordinates, it is necessary to solve the camera coordinate p c to the coordinates p of the scene w Convert the homography matrix The mathematical problem can be expressed as:

[0049] αp c = K[R|t]p w ,

[0050] where K is the pinhole camera intrinsic parameter model of the camera, which can be expressed as:

[0051]

[0052] where f x and f y are the focal length magnification factors in the x and y directions, and c x and c y are the pixel offsets in the x and y directions; R and t represent the 3D rotation matrix and 3D translation vector from the scene coordinate system to the camera coordinate system; α is the scaling coefficient of the homogeneous coordinates; and p w and p c are both represented in homogeneous coordinates.

[0053] To obtain the intrinsic matrix K, a common method is to use Zhang Zhengyou calibration method. It mainly uses a planar calibration board with a known geometric structure (usually a checkerboard pattern) to complete the calibration of the internal and external parameters of the camera. This method takes multiple images of the calibration board at different angles and positions, uses the feature point information extracted from the images, and estimates the internal parameters and distortion coefficients of the camera through least squares optimization. In addition, it can also obtain the parameters of the radial distortion of the image, and its physical meaning is the high-dimensional polynomial coefficients of the pixel coordinate mapping between the distorted image and the undistorted image, which can be used for camera undistortion.

[0054] To obtain R and t, the Perspective-n-Point (PnP) algorithm can be used for solution. First, in the embodiments of the present application, a checkerboard pattern matching the size of the physical display device can be formulated, and the corner points of which can obtain their scene world coordinates p w through pre-defined formats; secondly, in the embodiments of the present application, the feature points can be detected and the rotation amount and displacement amount of the physical device relative to the camera can be solved, and most operations can be performed to obtain the checkerboard calibration pattern projected on the physical screen as well. After shooting, the corner points of the checkerboard can be obtained by using the feature point extraction algorithm to obtain the image coordinates p c of the key feature points; the PnP problem transforms the problem of solving R and t into a system of equations constrained by multiple cosine theorems, and finally can be solved by means of mathematical modeling or constrained sampling to obtain stable solution values.

[0055] In step S102, the original pattern is rendered onto the captured image, and color balancing is performed on the original pattern and the actual image to obtain sample data that meets the preset matching conditions.

[0056] Embodiments of the present application can utilize internal camera representation parameters and the pose distribution of an object (calibration board) relative to the camera, render the original pattern onto the captured image, perform distortion adjustment to make the rendered pattern conform to the distorted shooting of the camera, and then perform color balancing to achieve gamut matching between the original pattern and the actual image, so that the actual image and the original pattern are matched in both the reference color and white balance. Furthermore, based on the two matched patterns, a set of sample data is formed.

[0057] Among them, the use of the rendering registration scheme can solve the registration problem in image restoration, complete the registration of the captured pattern and the original pattern at the pixel level, and eliminate the spatial error solved in the optimization problem.

[0058] Optionally, in an embodiment of the present application, rendering the original pattern onto the captured image and performing color balancing on the original pattern and the actual image to obtain sample data that meets preset matching conditions includes: constructing a corresponding perspective camera parameter model based on the camera internal parameters; using the spatial pose of the shooting environment and the perspective camera parameter model to render the original pattern onto the captured image to obtain the rendered pattern after rendering; using the distortion coefficient to perform distortion adjustment on the rendered pattern so that the rendered pattern after distortion adjustment meets the preset camera distortion shooting conditions; performing color balancing on the rendered pattern after distortion adjustment and the actual pattern to obtain sample data.

[0059] In a rendering system based on the OpenGL rendering framework, embodiments of the present application can construct a perspective camera parameter model using the internal parameters of the camera; place the position of the object (calibration board) using the pose transformation from the display screen to the camera, and use the perspective parameter model to perform refined rendering using vertex rendering, geometric rendering, and patch super-resolution techniques to generate the original pattern with an original 0-1 distribution, which only contains black and white colors and has extremely high-frequency signal change regions, while there are many influencing factors for the chromaticity and brightness of the captured pattern. After that, embodiments of the present application can perform color matching on the captured original pattern to make the colors in the original pattern match the actual image obtained under the shooting environment, so that the point spread function can be solved more accurately during subsequent solution, and prevent abnormal solution values due to color and brightness errors.

[0060] After obtaining the pose transfer matrix, the embodiments of the present application can use the rendering engine to obtain the true template that has not been convolved with the point spread function through simulation. Before optimization, there are very large differences in brightness and chromaticity between the actual image (original pattern) and the captured pattern. The reasons are as follows: The actual image is a 0-1 mask image, only containing black and white colors and having extremely high-frequency signal change regions, while the captured pattern is obtained by camera shooting. There are many influencing factors for the chromaticity and brightness of the image obtained by camera shooting. Besides the reflection color of the imaging object itself, the emission spectrum, color accuracy, color gamut of the physical display screen, the diffraction and mixing differences of different wavelength color lights by the camera microlens group, the spectral response characteristics of the camera CMOS (Complementary Metal-Oxide-Semiconductor) imaging plane, and the compression and mapping of colors by the ISP (Image Signal Processor) will all have a great impact on the imaging color. The color evaluation criteria of the display screen for colors include color accuracy, color gamut, and color depth. Among them, color accuracy represents the accuracy of the color presentation of the display screen. Usually, the color correction matrix is calculated by the difference between the captured image of the standard color color card under a certain light source, and it is used to correct the deviation of the original display screen from color accuracy; the color gamut of the display screen refers to the color display range it can present, and is implemented by its internal color control circuit according to specific color gamut definition standards (such as sRGB (Standard Red Green Blue), Adobe RGB (Adobe Red Green Blue), DCI-P3 (Digital Cinema Initiatives Protocol 3), and NTSC (National Television System Committee color space)); color depth is characterized by the color storage bit depth to represent its quantization accuracy. In addition, the spectral response characteristics of the camera will also affect the perception of visual colors. Common measurement indicators include quantum efficiency and relative spectral response. The former represents the absorption efficiency of the semiconductor material for optical signals with wavelength, and the latter represents the number of photons per unit energy at different wavelengths. Due to the differences in the spectral response characteristics of the camera, the colors captured by the camera will also change. Under the influence of the above multiple factors, the captured pattern will show a change deviating from the black and white color gamut in the hybrid system. Since other errors are generated between the captured pattern and the actual image under the influence of factors other than the point spread function, subsequent optimization of the point spread function will also be impossible to solve. Therefore, the embodiments of the present application need to perform color balance matching on the pattern.

[0061] When preparing for color matching, the embodiments of the present application need to use a white pattern w and a black pattern b and project them onto the display screen. Since the original patterns are all 0-1 distribution images, the color balance of the display screen part can be achieved by using w and b. While matching colors, the vignetting phenomenon of the camera can also be removed. Assume the original true image y o are all sampled from [0,1], and the color balance calculation of the actual image can be calculated as:

[0062] y = b + y o (w - b).

[0063] In step S103, an optimization problem of the point spread function is constructed using sample data, and the optimization problem is solved to obtain a point spread function that meets the preset accuracy conditions.

[0064] As a possible implementation manner, the embodiments of the present application can construct an optimization problem of the point spread function using multiple sets of sample data, and solve the exact solution of the point spread function by solving the optimization problem.

[0065] The point spread function is solved by a method that combines the optimization problem and the measurement problem, and at the same time incorporates the advantages of the measurement scheme using microscopic imaging and the simulation scheme using optical tracing methods, and more accurately measures the camera point spread function model.

[0066] Optionally, in an embodiment of the present application, constructing an optimization problem of the point spread function using sample data and solving the optimization problem to obtain a point spread function that meets the preset accuracy conditions includes: obtaining an initial solution of the point spread function using optical tracing; inputting the initial solution of the point spread function into the optimization problem, and performing iterative calculation on the optimization problem until a point spread function that meets the preset accuracy conditions is obtained.

[0067] The embodiments of the present application can perform optical tracing through a camera engineering file parameterized by Zemax: that is, by sampling spectral rays, the rays are traced along the trajectory according to the method of geometric optics. After the rays pass through the camera system, the image formed on the imaging plane is sampled and processed to become the value of the preliminary point spread function; then, according to the premise that the convolution of the point spread function and the original image is the captured image, an optimization problem is constructed: the captured pattern is input as the captured image, the original pattern is output as the original image, and the point spread function is used as the parameter to be optimized, and iterative optimization is performed. Specifically, the gradient descent method will be used to solve the problem to obtain the final point spread function.

[0068] Among them, in order to optimize the point spread function of each field of view, the embodiment of the present application takes the similarity between the captured image and the region of interest of the selected field of view of the original pattern as an optimization problem. This problem can be described in the following form; where x and y are the degraded signal and the true value signal respectively, and k is the point spread function.

[0069]

[0070] First of all, the embodiment of the present application should sample the camera field of view; according to the above formula, since the camera imaging system is a rotationally symmetric system, it is only necessary to sample the field of view in one dimension. The corresponding region of interest can be sampled on the captured mode image according to the sampled field of view. In order to accelerate the forward process, the space-variant convolution will be implemented as a block convolution. Considering that the final point spread function convolution kernel should have certain non-negativity, sparsity and continuity, the embodiment of the present application can represent the optimization problem as:

[0071]

[0072] In addition, during the development process, the embodiment of the present application should pay attention to the influence of parameter initialization on the optimization problem. Obviously, this problem requires that the convolution kernel k to be solved is a parameter with physical meaning, and the optimization from an initial solution space to a globally optimal point with strict constraints will be extremely difficult. Therefore, random initialization poses a great challenge to the optimization efficiency of the problem. Therefore, the parameters will be initialized to the initial solution of the point spread function. For cameras with an open white-box design system, the embodiment of the present application will use its industrial design value for initialization; for black-box camera systems, the embodiment of the present application will use the measurement value of the high-contrast point spread function at the photosensitive plane measured by the darkroom point light source measurement method as the initial value of the point spread function.

[0073] The initial value of the point spread function is generated by the optical tracing method, and the approximate point spread function can be solved by programming under different spectral, spatial sampling and lens group settings, providing a perfect initial value for the solution of the optimization problem and greatly accelerating the convergence of the problem.

[0074] Combined Figure 2 and Figure 3 As shown, the working principle of the camera point spread function calibration method of the embodiment of the present application is elaborated in detail with a specific embodiment.

[0075] The embodiments of the present application can use the MURA mode to calibrate the point spread function structure. The embodiments of the present application design a set of calibration optimization processes for the screen mode; in the process, the embodiments of the present application omit the homography matching matrix when the camera is aligned with the three-dimensional scene and the distortion estimation of the lens, so the geometric alignment error is greatly reduced, and pixel-level geometric correspondence is provided. Since the screen itself can project pattern images of a large order of magnitude, compared with expensive custom image modes with high calibration, this solution provides more constraints for the solution of the point spread function, so the constraint problem of the single template is converted into an online iterative optimization problem. In addition, since each screen itself has unique spectral characteristics, the embodiments of the present application need to perform white balance on the captured images to eliminate their radiation deviation. Finally, the embodiments of the present application use the stochastic gradient descent scheme to solve the point spread function of each field of view.

[0076] Specifically, as Figure 2 and Figure 3 shown, the embodiments of the present application may include the following steps:

[0077] Step S201: Generate a certain number of coded aperture patterns according to the coded aperture principle, sample and set up the layout of the display screen and the camera to ensure that the coded aperture patterns are captured under an accurate spatial layout.

[0078] The embodiments of the present application can set up the display screen and the camera according to the sampled scene distribution, and generate a certain number of coded aperture patterns, that is, the original patterns, according to the coded aperture principle, and capture the actual images through the set-up camera to obtain the captured images.

[0079] Step S202: Capture the pose calibration pattern, and obtain the internal camera representation parameters and the pose distribution of the object relative to the camera, including the distortion parameters, the camera internal parameters, the rotation amount and the displacement amount.

[0080] Prepare pose calibration patterns, usually including checkerboards, dot grids, etc.; among them, it is carried out by including the internal camera representation parameters, distortion parameters of the camera calibration, and the pose distribution of the object relative to the camera. Among them, the internal parameters of the camera are calibrated. The process of calibrating the internal camera representation parameters is as follows: The calibration pictures need to be taken with a calibration board at different positions, different angles, and different poses. At least 3 pictures are required, and 10 - 20 pictures are appropriate. The calibration board needs to be a checkerboard pattern composed of black and white rectangles, and the production accuracy requirements are relatively high; for each calibration picture, corner point information is extracted. The function of finding checkerboard corner points needs to be used to extract the corner points. Here, the corner points specifically refer to the internal corner points on the calibration board, and these corner points do not touch the edge of the calibration board; for each calibration picture, sub-pixel corner point information is further extracted. Among them, in order to improve the calibration accuracy, pixel information needs to be further extracted on the basis of the initially extracted corner point information to reduce the camera calibration deviation; after obtaining the image coordinates of the internal corner points of the checkerboard calibration picture, the PnP algorithm (Point-n-Perspective) can be used to calculate the internal parameters of the camera, including the magnification in the horizontal and vertical directions of the camera, and the offsets in the horizontal and vertical directions. The distortion parameters describe the radial and tangential deformation distances from the center to the edge, and are represented by a linear polynomial, with the independent variable being the distance from the current pixel to the center pixel. After that, the capture module measures the pose of the display screen to the camera in all scenarios, including the 3-degree-of-freedom rotation amount and the 3-degree-of-freedom displacement amount. The PnP problem converts the problem of solving the rotation amount and displacement amount into a system of equations constrained by multiple cosine theorems, and finally can be solved by means of mathematical modeling or constrained sampling to obtain a stable solution value.

[0081] Step S203: Use the internal camera representation parameters and the pose distribution of the object relative to the camera to render the original pattern onto the captured image, perform distortion adjustment to make the rendered pattern conform to the camera distortion shooting, and then perform color balance to achieve color matching between the original pattern and the actual image.

[0082] In the rendering system based on the OpenGL rendering framework, the embodiment of the present application can construct a perspective camera parameter model using the internal parameters of the camera; use the pose transformation from the display screen to the camera to place the position where the object (calibration board) is located, and use vertex rendering, geometry rendering, and patch super-resolution technology to perform refined rendering to generate the original pattern with an original 0-1 distribution, which only contains black and white colors and has extremely high-frequency signal change regions, while there are many influencing factors for the chromaticity and brightness of the captured pattern. After that, the embodiment of the present application can perform color matching on the captured original pattern to make the colors in the original pattern match the shooting environment of the actual captured pattern, that is, the original pattern and the actual image match, so that the point spread function can be solved more accurately during subsequent solution, and prevent the solution value from being abnormal due to color and brightness errors.

[0083] Step S204: Use optical tracing to generate an initial solution of the point spread function, then construct an optimization problem using the original pattern and the captured pattern, and use the gradient descent method to solve the exact solution of the point spread function.

[0084] In summary, the embodiments of the present application generate a high-frequency signal pattern for calibrating the point spread function according to the principle of the coded aperture, and arrange the on-site scene as the scene after precise measurement according to the measurement requirement situation. Capture the pose calibration pattern, and obtain the internal camera representation parameters and the pose distribution of the object relative to the camera, including distortion parameters, camera internal parameters, rotation amount, and displacement amount. Then, use the OpenGL-based rendering method to render the pattern, and use the color balance pattern for color matching. Finally, use the optimization problem solving method to solve the point spread function of the commercial camera under specific measurement conditions, greatly improving the stability and accuracy of the measurement. At the same time, it prevents the time-consuming and laborious microscopic measurement method, greatly improving the measurement efficiency. The point spread function can be used in downstream image restoration tasks and performs excellently.

[0085] According to the camera point spread function calibration method proposed by the embodiments of the present application, a camera can be used to capture an actual image in a pre-constructed shooting environment to obtain a corresponding captured image, render the original pattern onto the captured image, and perform color balance on the original pattern and the actual image to obtain sample data that meets the preset matching conditions. Then, an optimization problem of the point spread function is constructed, and the optimization problem is solved to obtain a point spread function that meets the preset precise conditions, so as to achieve point spread function calibration. While ensuring accuracy, the measurement process is optimized, and no expensive equipment is required. When applied, there are no special requirements for the scene or equipment, which is convenient for popularization and application. Thus, it solves the technical problems in the related art, such as the high cost of laser collimator detection, the strict standards for the measurement environment, the cumbersome measurement process of small-hole image analysis, and the simplified process can only be applied to specific scenes or specific equipment and is difficult to popularize and apply.

[0086] Next, refer to the drawings to describe a camera point spread function calibration device according to an embodiment of the present application.

[0087] Figure 4 It is a block diagram of a camera point spread function calibration device according to an embodiment of the present application.

[0088] As Figure 4 shown, the camera point spread function calibration device 10 includes: a first capture module 100, a processing module 200, and a calibration module 300.

[0089] Specifically, the first capture module 100 is configured to use a camera to capture an actual image in a pre-constructed shooting environment to obtain a corresponding captured image.

[0090] The processing module 200 is configured to render the original pattern onto the captured image and perform color balancing on the original pattern and the actual image to obtain sample data that meets the preset matching conditions.

[0091] The calibration module 300 is configured to construct an optimization problem of the point spread function using the sample data and solve the optimization problem to obtain a point spread function that meets the preset accuracy conditions.

[0092] Optionally, in an embodiment of the present application, the camera point spread function calibration device 10 further includes: an acquisition module, a judgment module, and a control module.

[0093] Among them, the acquisition module is configured to acquire the display screen parameters in the shooting environment.

[0094] The judgment module is configured to judge whether the shooting environment meets the preset capture conditions by using the field of view angle, focal length, and display screen parameters of the camera.

[0095] The control module is configured to use the camera to shoot the actual image when the shooting environment meets the preset capture conditions.

[0096] Optionally, in an embodiment of the present application, the camera point spread function calibration device 10 further includes: a second capture module, an extraction module, an estimation module, and a calibration module.

[0097] Among them, the second capture module is configured to use the camera to capture images of the target plane calibration board in multiple shooting scenarios.

[0098] The extraction module is configured to extract the feature point information of the image.

[0099] The estimation module is configured to estimate the internal parameters and distortion coefficients of the camera by using the feature point information.

[0100] The calibration module is configured to perform camera pose calibration by using the internal parameters and distortion coefficients of the camera, so as to use the camera after pose calibration to shoot the actual image.

[0101] Optionally, in an embodiment of the present application, the processing module 200 includes: a construction unit, a rendering unit, an adjustment unit, and a balancing unit.

[0102] Among them, the construction unit is configured to construct a corresponding perspective camera parameter model based on the camera internal parameters.

[0103] The rendering unit is configured to render the original pattern onto the captured image by using the spatial pose of the shooting environment and the perspective camera parameter model to obtain the rendered pattern after rendering.

[0104] The adjustment unit is configured to perform distortion adjustment on the rendered pattern by using the distortion coefficient, so that the rendered pattern after distortion adjustment meets the preset camera distortion shooting conditions.

[0105] A balancing unit for color balancing the rendered pattern and the actual pattern after distortion adjustment to obtain sample data.

[0106] Optionally, in an embodiment of the present application, the calibration module 300 includes: a first calculation unit and a second calculation unit.

[0107] The first calculation unit is used to obtain an initial solution of the point spread function by using optical tracing.

[0108] The second calculation unit is used to input the initial solution of the point spread function into the optimization problem and perform iterative calculations on the optimization problem until a point spread function that meets the preset accuracy condition is obtained.

[0109] It should be noted that the foregoing explanation of the embodiments of the camera point spread function calibration method also applies to the camera point spread function calibration device of this embodiment, and will not be repeated here.

[0110] The camera point spread function calibration device proposed according to the embodiments of the present application can use the camera to capture an actual image in a pre-constructed shooting environment to obtain a corresponding captured image, render the original pattern onto the captured image, and perform color balancing on the original pattern and the actual image to obtain sample data that meets the preset matching conditions. Furthermore, an optimization problem of the point spread function is constructed, and the optimization problem is solved to obtain a point spread function that meets the preset accuracy condition, so as to realize the calibration of the point spread function. While ensuring accuracy, the measurement process is optimized, and expensive equipment is not required. When applied, there are no special requirements for the scene or equipment, which is convenient for popularization and application. Thus, the technical problems in the related art are solved, such as the high cost of laser collimator detection, the strict standards for the measurement environment, the cumbersome measurement process of small hole image analysis, and the simplified process can only be applied to specific scenes or specific equipment and is difficult to be popularized and applied.

[0111] Figure 5 The structural schematic diagram of the electronic device provided by the embodiment of the present application. The electronic device may include:

[0112] A memory 501, a processor 502, and a computer program stored on the memory 501 and executable on the processor 502.

[0113] When the processor 502 executes the program, it implements the camera point spread function calibration method provided in the above embodiment.

[0114] Furthermore, the electronic device further includes:

[0115] A communication interface 503 for communication between the memory 501 and the processor 502.

[0116] A memory 501 for storing a computer program that can run on a processor 502.

[0117] The memory 501 may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

[0118] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 can be interconnected via a bus to complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity in representation, Figure 5 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0119] Optionally, in a specific implementation, if the memory 501, the processor 502, and the communication interface 503 are integrated on a single chip, the memory 501, the processor 502, and the communication interface 503 can complete communication with each other through an internal interface.

[0120] The processor 502 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0121] This embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above camera point spread function calibration method is implemented.

[0122] The embodiments of the present application also provide a computer program product, including a computer program, and when the computer program is executed by a processor, the camera point spread function calibration method provided by the embodiments of the present invention is implemented.

[0123] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0124] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0125] Any process or method description shown in the flowchart or described in other ways herein can be understood to represent a module, segment, or portion of code including one or N executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of this application belong.

[0126] The logic and / or steps represented in the flowchart or otherwise described herein can, for example, be considered as a definable list of executable instructions for implementing logical functions, and can be embodied specifically in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion (electronic device) having one or N wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0127] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0128] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0129] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, may exist separately as individual physical units, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0130] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A camera point spread function calibration method, characterized in that: The following steps are involved: Using a camera to shoot an actual image in a pre-built shooting environment to obtain a corresponding captured image; Rendering the original pattern onto the captured image, and performing color balancing on the original pattern and the actual image to obtain sample data that meets a preset matching condition; The sample data is used to construct an optimization problem of a point spread function, and the optimization problem is solved to obtain a point spread function that meets preset precise conditions.

2. The method according to claim 1, characterized in that Before using the camera to capture actual images in a pre-built capture environment, it also includes: Obtaining display screen parameters under the shooting environment; Using the field of view angle, focal length and display screen parameters of the camera, determining whether the shooting environment meets the preset capture conditions; If the shooting environment meets the preset capturing condition, the actual image is shot by using the camera.

3. The method according to claim 1, characterized in that Before using the camera to capture actual images in a pre-built capture environment, it also includes: Using the camera to capture images of the target plane calibration plate in various shooting scenarios; Extracting feature point information of the image; estimating the intrinsic parameters and distortion coefficients of the camera using the feature point information; The camera posture calibration is performed using the intrinsic parameters and distortion coefficients of the camera, so as to capture the actual image using the posture-calibrated camera.

4. The method according to claim 3, characterized in that: The rendering of the original pattern onto the captured image and color balancing the original pattern and the actual image to obtain sample data satisfying a preset matching condition includes: Constructing a corresponding perspective camera parameter model based on the camera intrinsic parameters; Rendering the original pattern onto the captured image using the spatial pose of the shooting environment and the perspective camera parameter model to obtain a rendered pattern; Using the distortion coefficient to perform distortion adjustment on the rendering pattern, so that the distortion-adjusted rendering pattern meets a preset camera distortion shooting condition; Color balancing is performed on the distortion-adjusted rendering pattern and the actual pattern to obtain the sample data.

5. The method according to claim 1, characterized in that The step of constructing an optimization problem of a point spread function using the sample data and solving the optimization problem to obtain a point spread function that satisfies a preset precise condition includes: The initial solution of the point spread function is obtained by optical tracing; The initial solution of the point spread function is input into the optimization problem, and the optimization problem is iteratively calculated until the point spread function of the preset precise condition is obtained.

6. A camera point spread function calibration device, characterized in that: include: A capture module, used to capture an actual image using a camera in a pre-built shooting environment to obtain a corresponding captured image; A processing module, used for rendering the original pattern onto the captured image, and performing color balance on the original pattern and the actual image to obtain sample data that meets a preset matching condition; The calibration module is used to construct an optimization problem of a point spread function using the sample data, and solve the optimization problem to obtain a point spread function that meets preset precise conditions.

7. The device according to claim 6, characterized in that Also includes: An acquisition module, used to acquire display screen parameters under the shooting environment; A judgment module, used to judge whether the shooting environment meets the preset capture conditions by using the field of view angle, focal length and display screen parameters of the camera; A control module is used to capture the actual image using the camera when the shooting environment meets the preset capture condition.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the camera point spread function calibration method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the camera point spread function calibration method as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, it is used to implement the camera point spread function calibration method according to any one of claims 1 to 5.

Images