Optimization method, device, light field camera and medium for light field camera calibration

Abstract

The present application is applicable to the field of light field camera technology, and in particular to an optimization method, device, light field camera and medium for light field camera calibration. The method uses a combination of linear transformation and radial basis transformation to accurately describe the reprojection error, and combines the regularization term formed by the constraint of the parameters during the transformation to construct an energy function, and performs variational optimization with the minimum of the energy function as the goal to obtain accurate transformation parameters, and then uses the transformation parameters to calculate the correction coordinates, and uses the correction coordinates to optimize the initial calibration internal and external parameters to obtain the calibration internal and external parameters after distortion correction, thereby achieving distortion correction, making the calibration internal and external parameters more accurate, and improving the accuracy of the camera when used.

Description

Technical Field

[0001] The present application is applicable to the field of light field camera technology, and in particular, relates to an optimization method, device, light field camera and medium for light field camera calibration. Background Art

[0002] At present, the calibration technology of light field cameras is of great value in modern industrial inspection. Light field cameras can record the spatial position and direction information of light, thereby realizing three-dimensional reconstruction of the scene. This makes light field cameras widely used in industrial inspection. For example, in the inspection of printed circuit boards (PCBs), it can obtain a three-dimensional image of the PCB through a single imaging, and then accurately measure the length of PCB pins and detect surface defects. However, the high-precision application of light field cameras is inseparable from efficient and accurate calibration methods.

[0003] Existing research focuses more on improving calibration accuracy, while ignoring the impact of calibration speed on industrial inspection efficiency. In actual industrial environments, the intrinsic and extrinsic parameters of light field cameras inevitably change due to operations such as zooming, refocusing, and mechanical vibration, which requires frequent recalibration. The calibration process is time-consuming and directly affects the efficiency of industrial inspection. It can be seen that the current optimization methods for light field camera calibration have defects such as redundancy and inefficiency in high-dimensional data calculation, instability in solving high-dimensional nonlinear equations, and the impact of complex lens distortion on calibration accuracy.

[0004] Therefore, how to correct the distortion caused by complex lenses to improve the accuracy of calibration parameters has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the embodiments of the present application provide an optimization method, device, light field camera and medium for light field camera calibration to solve the problem of how to correct the distortion caused by complex lenses to improve the accuracy of calibration parameters.

[0006] In a first aspect, an embodiment of the present application provides an optimization method for light field camera calibration, and the optimization method for light field camera calibration includes:

[0007] Acquire a light field image of a calibration plate scene acquired by a light field camera, determine calibration internal and external parameters for calibration of the light field camera based on the light field image, extract feature points from the light field image, select a preset number of feature points as control points, and determine the image coordinates of each control point;

[0008] Taking any pixel point in the light field image as a target point, constructing an image coordinate difference between the target point and each control point based on a radial basis function, performing a linear transformation on the image coordinate of the target point using a linear transformation function to obtain a linear transformation coordinate, and performing a weighted summation on all image coordinate differences to obtain a summation result;

[0009] Constructing the distorted coordinates of the target point according to the summation result and the linear transformation coordinates, determining the reprojection error according to the image coordinates of the target point and the distorted coordinates, constraining the weight corresponding to each image coordinate difference using constraint conditions to obtain a regularization term, and taking the sum of the reprojection error and the regularization term as an energy function;

[0010] Iteratively optimizing the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the condition of the minimum energy function is satisfied, thereby obtaining optimized linear parameters and optimized weights, and calculating the distorted coordinates of the feature points according to the optimized linear parameters and the optimized weights to obtain corrected coordinates;

[0011] The image coordinates of the feature points are projected using the calibration internal and external parameters to obtain calibration coordinates, and the calibration internal and external parameters are optimized with the goal of minimizing the difference between the correction coordinates and the calibration coordinates to obtain optimized calibration internal and external parameters.

[0012] In a second aspect, an embodiment of the present application provides an optimization device for light field camera calibration, the optimization device for light field camera calibration comprising:

[0013] A feature extraction module is used to obtain a light field image in a calibration plate scene acquired by a light field camera, determine calibration internal and external parameters for calibration of the light field camera based on the light field image, extract feature points from the light field image, select a preset number of feature points as control points, and determine the image coordinates of each control point;

[0014] A coordinate transformation module is used to take any pixel point in the light field image as a target point, obtain the image coordinate difference between the target point and each control point based on a radial basis function, use a linear transformation function to linearly transform the image coordinates of the target point to obtain linear transformation coordinates, and perform weighted summation on all image coordinate differences to obtain a summation result;

[0015] An energy function module is used to construct the distorted coordinates of the target point according to the summation result and the linear transformation coordinates, determine the reprojection error according to the image coordinates of the target point and the distorted coordinates, use constraint conditions to constrain the weight corresponding to each image coordinate difference to obtain a regularization term, and use the sum of the reprojection error and the regularization term as an energy function;

[0016] The correction coordinate calculation module is used to iteratively optimize the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the condition of the minimum energy function is met, thereby obtaining the optimized linear parameters and the optimized weights, and calculating the distorted coordinates of the feature points according to the optimized linear parameters and the optimized weights to obtain the correction coordinates;

[0017] The calibration parameter optimization module is used to use the calibration internal and external parameters to project the image coordinates of the feature points to obtain calibration coordinates, and optimize the calibration internal and external parameters with the goal of minimizing the difference between the correction coordinates and the calibration coordinates to obtain optimized calibration internal and external parameters.

[0018] In a third aspect, an embodiment of the present application provides a light field camera, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the optimization method for light field camera calibration as described in the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the optimization method for light field camera calibration as described in the first aspect is implemented.

[0020] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application adopts a combination of linear transformation and radial basis transformation to accurately describe the reprojection error, and constructs an energy function in combination with a regularization term formed by constraints on parameters during transformation, performs variational optimization with the goal of minimizing the energy function, and obtains accurate transformation parameters, thereby using the transformation parameters to calculate the correction coordinates, and using the correction coordinates to optimize the initially obtained calibration internal and external parameters to obtain the calibration internal and external parameters after distortion correction, thereby achieving distortion correction, making the calibration internal and external parameters more accurate, and improving the accuracy of the camera when used. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 This is a schematic diagram of an application environment of an optimization method for light field camera calibration provided in Example 1 of the present application;

[0023] Figure 2It is a flowchart of an optimization method for light field camera calibration provided in Embodiment 2 of the present application;

[0024] Figure 3 It is a flowchart of an optimization method for light field camera calibration provided in Embodiment 3 of the present application;

[0025] Figure 4 It is a schematic diagram of the principle of an optimization method for light field camera calibration provided in Example 3 of the present application;

[0026] Figure 5 It is a flowchart of an optimization method for light field camera calibration provided in Embodiment 4 of the present application;

[0027] Figure 6 It is a structural schematic diagram of an optimization device for light field camera calibration provided in Example 5 of the present application;

[0028] Figure 7 It is a structural schematic diagram of a light field camera provided in Example 6 of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0030] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0031] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0033] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0034] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0035] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0036] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0037] The optimization method for light field camera calibration provided in the first embodiment of the present application can be applied in the following aspects: Figure 1 In the application environment, the light field camera is set in the corresponding PCB detection environment, which can effectively capture images of the PCB on the platform, and the computer is connected to the light field camera to obtain data collected by the light field camera. Among them, the computer can include but is not limited to a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, a cloud light field camera, a personal digital assistant (personal digital assistant, PDA), etc., and can also be implemented by an independent server. In addition, the optimization method for light field camera calibration can be executed in the processor of the light field camera itself, and of course, the camera calibration can also be performed through a computer connected thereto.

[0038] See also Figure 2 , is a flow chart of an optimization method for light field camera calibration provided in the second embodiment of the present application. The optimization method for light field camera calibration is applied to a light field camera or a computer. Figure 2 As shown, the optimization method for light field camera calibration may include the following steps:

[0039] Step S201, obtain a light field image based on a calibration plate scene captured by a light field camera, determine the calibration internal and external parameters used for light field camera calibration based on the light field image, extract feature points from the light field image, select a preset number of feature points as control points, and determine the image coordinates of each control point.

[0040] In this embodiment, the light field camera includes an image sensor, a microlens array and a main lens. The image sensor is placed at twice the focal length of the microlens array. The microlens array is placed at the focal plane of the camera. Through refraction by the main lens and focusing by the microlens, it is finally recorded by the acquisition unit on the image sensor.

[0041] The calibration plate is imaged using a light field camera to obtain a light field image, which includes initial light field data, such as light, light intensity, and light angle. Calibration of the light field image can obtain initial calibration internal and external parameters, on which basis, the calibration internal and external parameters are optimized.

[0042] Feature points are extracted from the light field image. The feature points can be detected using the corner point detection method to obtain corner points. The corner points are used as feature points. An appropriate number of control points are selected from the feature points as the basis for optimization. The coordinates of each control point can be expressed as .

[0043] Step S202, taking any pixel point in the light field image as a target point, constructing an image coordinate difference between the target point and each control point based on a radial basis function, performing a linear transformation on the image coordinates of the target point using a linear transformation function to obtain a linear transformation coordinate, performing a weighted summation on all image coordinate differences to obtain a summation result.

[0044] In this embodiment, any pixel point in the light field image is transformed as a target point. is the undistorted image coordinate, and the distorted image coordinate is , calculated by the distortion model as follows:

[0045] ;

[0046] in, is the control point, is the radial basis function, and is the weight coefficient to be optimized, is the linear transformation parameter.

[0047] Step S203, construct the distorted coordinates of the target point based on the summation result and the linear transformation coordinates, determine the reprojection error based on the image coordinates and the distorted coordinates of the target point, use constraints to constrain the weights corresponding to each image coordinate difference, obtain the regularization term, and use the sum of the reprojection error and the regularization term as the energy function.

[0048] Among them, the distorted coordinates of the target point are the sum of the linear transformation coordinates after linear transformation through the linear transformation parameters and the summation result. The coordinates of the target point before and after distortion are reprojected and the error is calculated.

[0049] The goal of distortion correction is described by introducing an energy function E as follows:

[0050] ;

[0051] in, is the reprojection error, , is a regularization term, combined with topological constraints, ;

[0052] in, is the topological constraint coefficient, which is used to control the relationship between different control points and ensure the smoothness and geometric consistency of the transformation.

[0053] Step S204, iteratively optimize the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the condition of minimum energy function is met, and obtain optimized linear parameters and optimized weights. According to the optimized linear parameters and optimized weights, calculate the distorted coordinates of the feature points to obtain corrected coordinates.

[0054] In this embodiment, the finite element method is combined with the gradient descent optimization algorithm to perform parameter optimization. The finite element method can effectively handle the deformation problem of complex geometric shapes. By discretizing the image plane into a finite number of elements, the local influence of the transformation is accurately calculated. The optimized variational method finds the optimal nonlinear deformation parameters by minimizing the energy function.

[0055] Optionally, the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference are iteratively optimized until the condition of the minimum energy function is satisfied, and the optimized linear parameters and optimized weights are obtained, including:

[0056] The pixels of the light field image are discretized using the finite element method to obtain discrete pixel points, and the partial derivatives of the discrete pixel points with respect to the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference are calculated to obtain the partial derivatives of each linear parameter and weight;

[0057] The partial derivatives of each linear parameter and weight are used as elements of the Jacobian matrix, and the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference are iteratively optimized using the gradient descent method until the reprojection error converges to a preset threshold or reaches the maximum number of iterations, thus obtaining the optimized linear parameters and optimized weights.

[0058] Among them, construct an optimized variational model and define the transformation function and , and initialize the weight coefficients and , and the linear transformation parameters and .

[0059] The image plane is discretized using the finite element method, the transformation effect within each element is calculated, and the Jacobian matrix J is constructed, whose elements are the partial derivatives of the reprojection error with respect to the optimization parameters, as follows:

[0060] .

[0061] The specific partial derivatives are calculated as follows:

[0062] ;

[0063] in, is the Kroneckerdelta function, which is 1 when j=k and 0 otherwise. By calculating these partial derivatives, the Jacobian matrix can be obtained, and the gradient descent method can be used to update the parameters. represents the coordinates on the direction plane of the light field camera, that is, the coordinates of the plane where the microlens array is located. These coordinates describe the direction information of each light ray. Specifically, and are the spatial sampling points of a light ray on the direction plane. They reflect the distribution of the light field data in the direction dimension. is the index of the high-frequency feature subview selected after the light field data is analyzed in the frequency domain, corresponding to , represents the directional coordinates containing important geometric details and parallax information. Through frequency domain analysis, the directional dimension of the light field data is converted into frequency domain form, from which the part with the richest information, namely the high-frequency part, is screened out.

[0064] Using the gradient descent algorithm, update the optimization parameters as follows:

[0065] ;

[0066] Among them, c represents all optimization parameters, including , is the learning rate, is the gradient of the reprojection error. Through iterative optimization, the parameters are continuously updated until the reprojection error converges to the preset threshold or reaches the maximum number of iterations.

[0067] Step S205, using the calibration internal and external parameters, projecting the image coordinates of the feature points to obtain calibration coordinates, optimizing the calibration internal and external parameters with the goal of minimizing the difference between the correction coordinates and the calibration coordinates to obtain optimized calibration internal and external parameters.

[0068] In this embodiment, the optimized nonlinear deformation parameters are obtained. After completing the nonlinear deformation parameter optimization based on the optimal variation method and topological constraints, we have obtained a more accurate correction result. , this result accurately reflects the ideal projection position of the feature point in the image coordinate system. In order to integrate this high-precision correction result into the standard camera model (intrinsic parameter K, external parameter R, T and distortion parameter ), the optimization problem can be reconstructed as follows after the optimization is completed:

[0069] ;

[0070] In the above objective function, is based on the calibration of internal and external parameters (given ) is the pixel coordinate obtained by projecting the feature point, and The ideal correction coordinates are obtained from the optimization of nonlinear deformation parameters. By solving this optimization problem, the advanced complex distortion correction results are "fed back" to the standard model, thereby updating and improving the accuracy of the standard distortion parameters involved in the camera inside and outside.

[0071] The embodiment of the present application adopts a combination of linear transformation and radial basis transformation to accurately describe the reprojection error, and constructs an energy function in combination with a regularization term formed by constraints on parameters during transformation, performs variational optimization with the goal of minimizing the energy function, and obtains accurate transformation parameters, which are then used to calculate the correction coordinates. The correction coordinates are used to optimize the initially obtained calibration internal and external parameters to obtain the calibration internal and external parameters after distortion correction, thereby achieving distortion correction, making the calibration internal and external parameters more accurate, and improving the accuracy of the camera when used.

[0072] See also Figure 3 , is a flow chart of an optimization method for light field camera calibration provided in the third embodiment of the present application. Figure 3 As shown, the step S201 of determining the calibration internal and external parameters for calibrating the light field camera according to the light field image may include the following steps:

[0073] Step S301, obtaining initial light field data corresponding to the light field image, performing frequency domain conversion on the directional plane coordinates in the initial light field data to obtain frequency domain light field data, and filtering the frequency domain light field image on the condition that the horizontal frequency domain value or the vertical frequency domain value is greater than the frequency domain threshold to obtain light field component data.

[0074] In this embodiment, the light field camera should be aligned with the calibration board for image acquisition during calibration, rather than the PCB, so as to obtain initial light field data in the calibration board scene. The initial light field data can include 7 dimensions. Of course, simplified light field data can be described using 4 dimensions, that is, 4D light field data. The 4D light field data is represented as ,in, is the directional plane coordinate of the light field, representing the index of the pixel corresponding to each microlens, and the range is , R represents the boundary value of the sampling range of the microlens array on the direction plane, which determines the number of discrete sampling in the light direction. is the spatial position plane coordinate of the light field, representing the position index of the microlens, and the range is , and Represents the resolution of the sensor in the horizontal and vertical directions respectively.

[0075] The above initial light field data includes the direction information and spatial position information of the light. The direction dimension of the light field data is converted from the spatiotemporal domain to the frequency domain. Through the two-dimensional discrete Fourier transform, the information of the direction dimension can be transferred to the frequency domain for processing. The frequency domain transformation formula is as follows:

[0076] ;

[0077] in, is the light field data in the frequency domain, and is the frequency coordinate, representing the coordinate of the directional dimension in the frequency domain. Through this transformation, the light field data is converted from the spatiotemporal domain to the frequency domain, reducing the computational complexity in the spatiotemporal domain.

[0078] In the frequency domain, the low-frequency part usually represents the overall information of the light field, while the high-frequency part contains more detailed geometric structure and disparity information. In order to more effectively extract features with important geometric information, a frequency domain threshold is set , only keep the frequency domain satisfying or The high-frequency component (i.e., light field component data) of the image is selected, wherein the screening process of the high-frequency component is as follows: .

[0079] like Figure 4As shown, it is a schematic diagram of the principle of an optimization method for light field camera calibration provided in Example 3 of the present application. It can be seen that the light passes through the main lens and the microlens array to reach the plane where the image sensor is located (i.e., the image plane), and each acquisition unit in the sensor can record complete image information.

[0080] Step S302 : using the images captured by the acquisition unit of the light field camera as sub-views, and pairing all sub-views according to the light field component data to obtain at least one matching pair of sub-views.

[0081] In this embodiment, each acquisition unit of the image sensor of the light field camera acquires a corresponding sub-view. After the frequency domain filtering is performed, each sub-view corresponds to the filtered light field component data. The light field component data is used as the image information of the sub-view, and all the sub-views are paired to find at least one pair of matching sub-views. Among them, the matching of two sub-views in the sub-view pair indicates that the image information recorded by the two is relatively close, which can further improve the probability of successful matching of subsequent matching point pairs.

[0082] After filtering the high-frequency components, adjacent sub-views are selected for pairing. The sub-view pairs corresponding to the high-frequency components usually contain more geometric details and disparity information, which can help improve the effectiveness of subsequent camera parameter estimation.

[0083] Optionally, all sub-views are paired according to the light field component data to obtain at least one pair of matching sub-views, including:

[0084] Determine image feature information corresponding to all sub-views according to the light field component data;

[0085] For any subview, obtain an adjacent subview of the subview, perform feature point matching between the image feature information of the subview and the image feature information of each adjacent subview, and determine the adjacent subview that matches the subview as a matching subview;

[0086] The subview and the matching subview are regarded as a subview pair, and all subviews are traversed to obtain at least one matching subview pair.

[0087] When matching subview pairs, the subview is paired with its adjacent subviews, and the two most matching subviews are found from the adjacent subviews as a subview pair. Adjacent subview matching can reduce the amount of matching data and improve matching efficiency.

[0088] In addition, the matching may be performed by extracting features from the sub-views and then performing similarity calculation on the features, thereby obtaining two sub-views with the highest feature similarity.

[0089] Step S303: for any sub-view pair, pairing is performed according to the corner point positions of the calibration plate collected in the sub-view pair to generate matching point pairs, and all sub-view pairs are traversed to obtain all matching point pairs.

[0090] In this embodiment, for any pair of sub-views, the positions of the same point represented by the two sub-views should be the same, so the positions of the corner points of the collected calibration plate are paired to obtain a pair of matching points.

[0091] Among them, each sub-view pair can obtain at least one matching point pair. If there are N intersection points in the calibration plate, then one sub-view should obtain N matching point pairs. All sub-view pairs need to calculate matching point pairs, thereby forming multiple matching point pairs for subsequent parameter calculation.

[0092] The feature points in each pair of subviews are detected and matched. The corner point positions of the calibration plate (e.g., checkerboard) are extracted from each pair of subviews through the corner point detection algorithm and marked according to the horizontal and vertical coordinates of the image plane. and Then, through the feature matching method, we ensure that the corner points in each pair of subviews correspond to each other to form matching point pairs. .

[0093] Optionally, pairing is performed according to the corner point positions of the calibration plate collected in the subview pair to generate matching point pairs, including:

[0094] Detecting corner points of the calibration plate image contained in the first subview and the second subview of the subview pair, obtaining a first group of corner points corresponding to the first subview and a first corner point position corresponding to each corner point, and a second group of corner points corresponding to the second subview and a second corner point position corresponding to each corner point;

[0095] Each corner point of the first group of corner points is matched one by one with each corner point of the second group of corner points, and the matched corner points are determined to be a matching point pair, wherein the matching point pair includes a first corner point position of a corresponding corner point in the first group of corner points and a second corner point position of a corresponding corner point in the second group of corner points.

[0096] Among them, for the subviews, it is necessary to use a corner detection algorithm to detect the corner points in the image and determine the corner point position of each corner point, and then match a corner point in one subview with all the corner points in another subview one by one to obtain paired corner points.

[0097] In the above matching, the judgment can be made according to the distance between the corner points. For example, the distance between two corner points in the calibration plate is S, and the distance between the matched corner points in the two subviews is required to be less than S / 2, so that the corner point matching result can be obtained quickly and accurately.

[0098] Step S304, construct the epipolar constraint equations of the corresponding matching point pairs by combining the basic matrices formed by each matching point pair and the internal and external parameters, and calculate the initial internal and external parameters with the goal of minimizing all the epipolar constraint equations. According to the initial internal and external parameters, determine the calibration internal and external parameters for light field camera calibration.

[0099] In this embodiment, after obtaining the matching point pairs, an epipolar constraint equation is constructed to solve the camera parameters. For each pair of subviews, it is assumed that their matching point pairs satisfy the epipolar constraint, as follows:

[0100] ;

[0101] Among them, F is the basis matrix, which describes the geometric relationship between the two subviews. The estimation process of the basis matrix is ​​performed by minimizing the epipolar error of the matching point pairs. Therefore, the objective function is as follows:

[0102] .

[0103] The basic matrix estimate of each pair of sub-views can be obtained through the least squares method, thus providing the necessary constraints for the estimation of the camera's internal and external parameters.

[0104] Furthermore, there is a clear mathematical relationship between the basic matrix F and the camera's internal and external parameters. Let K be the camera's internal parameter matrix, R be the camera's rotation matrix, and T be the camera's translation vector. The relationship between the basic matrix and the internal and external parameters is as follows:

[0105] ;

[0106] in, is the antisymmetric matrix of the translation vector T, .

[0107] By combining the estimates of multiple fundamental matrices, a joint optimization problem can be constructed to optimize the intrinsic and extrinsic parameters of the camera. The objective function of the joint optimization is as follows:

[0108] ;

[0109] The above objective function can be solved by the Levenberg-Marquardt algorithm, and the initial value parameters are estimated as the initial internal and external parameters of the light field camera.

[0110] Optionally, based on the initial internal and external parameters, calibration internal and external parameters for light field camera calibration are determined, including:

[0111] The initial internal and external parameters are initially optimized to obtain the internal and external parameters after initial optimization, and the internal and external parameters after initial optimization are determined as the calibration internal and external parameters.

[0112] In this embodiment, after obtaining the initial internal and external parameters, optimized internal and external parameters are obtained through operations such as distortion optimization. The optimized internal and external parameters are used to calibrate the light field camera. The calibrated light field camera can be used for subsequent PCB image acquisition work to obtain accurate images, which is helpful for subsequent PCB inspection work.

[0113] The embodiment of the present application obtains the initial light field data in the calibration plate scene based on the light field camera, performs frequency domain conversion on the directional plane coordinates in the initial light field data, obtains frequency domain light field data, screens the frequency domain light field image under the condition that the horizontal frequency domain value or the vertical frequency domain value is greater than the frequency domain threshold, obtains light field component data, uses the image collected by the acquisition unit of the light field camera as the subview, pairs all subviews according to the light field component data, obtains at least one pair of matching subview pairs, and for any subview pair, pairs according to the corner point position of the calibration plate collected in the subview pair, generates matching point pairs, traverses all subview pairs, obtains all matching point pairs, and constructs the epipolar constraint equations corresponding to the matching point pairs by combining each matching point pair with the basic matrix formed by the internal and external parameters, calculates the initial internal and external parameters with the goal of minimizing all the epipolar constraint equations, and determines the initial optimized internal and external parameters as the calibration internal and external parameters. The frequency domain analysis effectively reduces the computational redundancy in the light field data processing, and adopts the high-frequency feature extraction and the establishment of the epipolar constraint equation to accelerate the initial value estimation of the camera calibration and improve the efficiency of the camera calibration.

[0114] See also Figure 5 , is a flow chart of an optimization method for light field camera calibration provided in the fourth embodiment of the present application. Figure 5 As shown, the initial optimization of the initial internal and external parameters to obtain the internal and external parameters after the initial optimization, and determining the internal and external parameters after the initial optimization as the calibration internal and external parameters, may include the following steps:

[0115] Step S501, obtaining the 3D world coordinates of any point in the calibration plate in the world coordinate system, and using initial internal and external parameters, converting the 3D world coordinates into 3D camera coordinates in the camera coordinate system.

[0116] In this embodiment, the calibration plate is set Points is the representation in the world coordinate system (i.e., three-dimensional world coordinates), where They are the calibration plate The three-dimensional coordinates of points, and 1 is the homogeneous coordinate, through the external reference (where R is the rotation matrix and T is the translation vector) The points are transformed from the world coordinate system to the camera coordinate system as follows:

[0117] ;

[0118] in, is the coordinate in the camera coordinate system (i.e., three-dimensional camera coordinate).

[0119] The above-mentioned initial internal and external parameters are the parameters obtained through the above-mentioned step S204, wherein the initial internal and external parameters include the internal parameter K and the external parameters R and T. The initial three-dimensional camera coordinates can be obtained by using the external parameters to transform the coordinate system.

[0120] Step S502, obtaining a coefficient matrix of a light field camera.

[0121] Among them, the coefficient matrix represents the transformation relationship between the three-dimensional position of any point in the calibration plate in the world coordinate system and the measured value of the light field ray. The light field ray is the light between the main lens and the microlens array. The above 4D light field data represents the light between the microlens array and the image sensor (i.e., the image plane). The coefficient matrix can be used to express the three-dimensional position in the form of light field rays.

[0122] Step S503, obtaining a calibration matrix, setting the nonlinear parameters in the calibration matrix to fixed values, and obtaining an approximate calibration matrix.

[0123] The calibration matrix represents the transformation relationship between the parameters of the light field rays in the light field camera and the light on the image plane.

[0124] Parameterization of light field rays With the rays on the image plane The relationship between Description, as follows:

[0125] ;

[0126] Among them, the calibration matrix as follows:

[0127] ;

[0128] in, is the parameter of the calibration matrix, and the nonlinear parameter is , for the calibration matrix The nonlinear parameters in Approximate or fix to obtain an approximate calibration matrix.

[0129] In step S504, under the first-order distortion coefficient, the approximate calibration matrix, the external parameters to be optimized and the first-order distortion coefficient are formed into a parameter vector to be solved, the result of multiplying the coefficient matrix and the parameter vector to be solved is used to construct a linear equation with the three-dimensional camera coordinates, and the linear equation is solved using the singular value decomposition method to obtain a preliminary parameter estimation result of the parameter vector to be solved.

[0130] In this embodiment, considering that the camera lens may have radial distortion, the undistorted image coordinates are defined as and the distorted image coordinates The relationship between them is as follows:

[0131] ;

[0132] Among them, the distortion coefficient Describes radial distortion, is the sum of the squares of the image coordinates.

[0133] On this basis, a complete optimization objective function is constructed, combining light field ray mapping and distortion model. The objective function is as follows:

[0134] ;

[0135] in, is the number of points in the calibration plate, P is the number of light field rays, and They are the light field rays in The number of sampling points in the direction determines the sampling density and angular resolution in the direction dimension. For the The light field rays at the position, For the The three-dimensional position of the chessboard points in the camera coordinate system, the reprojection error Measures the distance between the light field ray and the point on the calibration plate.

[0136] The optimization process is divided into three levels, and each level gradually introduces more nonlinear terms to improve the accuracy and efficiency of optimization. In this embodiment, the primary level is involved, and the high-order distortion terms are first ignored. and , simplifying the optimization problem into a linear form. By simplifying the distortion model, only the first-order distortion coefficient is considered , ignoring higher-order terms , , the approximate linear distortion formula is as follows:

[0137] ;

[0138] Then, construct the linear equations as follows: ;

[0139] Where A is the coefficient matrix, X is the parameter vector to be determined, and contains the approximate calibration matrix Parameters, external parameters R, T and distortion parameters , B is the measurement value vector, represented by the ray parameters or pixel coordinates actually observed in the light field camera (i.e., the three-dimensional camera coordinates mentioned above).

[0140] In this embodiment, the linear equations are quickly solved by singular value decomposition method to obtain preliminary parameter estimation results. .

[0141] The preliminary parameter estimation result is used to obtain the optimized internal and external parameters. The preliminary parameter estimation result includes external parameters. The optimized internal and external parameters can be obtained by combining the external parameters with the internal parameters in the above-mentioned initial internal and external parameters.

[0142] Step S505 , predicting the measured value of the light field ray reaching the three-dimensional camera coordinates according to the calibration matrix and the first-order distortion coefficient in the preliminary parameter estimation result in combination with the second-order distortion coefficient, to obtain a first coordinate prediction value.

[0143] Step S506, with the goal of minimizing the reprojection error between the first coordinate prediction value and the three-dimensional camera coordinate, the Gauss-Newton iterative algorithm is used to iteratively update the preliminary parameter estimation result to obtain an updated parameter estimation result.

[0144] In this embodiment, the optimization of the intermediate layer is involved, and the second-order distortion coefficient is introduced , the further optimization objective function is as follows:

[0145] ;

[0146] in, is a low-order nonlinear model, .

[0147] The second-order distortion coefficient The distortion model is introduced as follows: ;

[0148] Then, the distortion model is substituted into the ray reprojection error and the nonlinear objective function is constructed as follows:

[0149] ;

[0150] In the formula, is the predicted value of the first coordinate, is the measured value of the light field ray.

[0151] Use the Gauss-Newton algorithm to iteratively update the parameters. The update formula is as follows:

[0152] ;

[0153] Where J is the Jacobian matrix, which represents the first-order partial derivative of the residual vector r with respect to the parameter vector X.

[0154] The elements of the Jacobian matrix are as follows: ;

[0155] in, is the error between the model prediction value and the actual observation value, and r is the residual vector, which represents the deviation between the light field ray and the three-dimensional position of the calibration plate.

[0156] Step S507, predicting the measured value of the light field ray reaching the three-dimensional camera coordinates according to the calibration matrix, the first-order distortion coefficient, the second-order distortion coefficient and the third-order distortion coefficient in the updated parameter estimation result to obtain a second coordinate prediction value.

[0157] Step S508, with the goal of minimizing the reprojection error between the second coordinate prediction value and the three-dimensional camera coordinate, and with the objective function and the bias vector of the parameter as the gradient, the updated parameter estimation result is optimized in the trust region to obtain the final parameter estimation result, and the final parameter estimation result is used to obtain the optimized internal and external parameters.

[0158] In this embodiment, the third-order distortion coefficient is introduced in the optimization of the high-level layer. As well as other high-order nonlinear terms, the optimized complete light field camera model is as follows:

[0159] ;

[0160] The optimization objective function at this point includes all high-order distortion terms, and the ultimate goal is to minimize all reprojection errors. Restore the complete distortion model, including the third-order distortion coefficients , as shown below:

[0161] ;

[0162] The trust region method is used to optimize the algorithm. The results of the first two layers of optimization are used as the initial value to further optimize , R, T, and , in trust region optimization, the gradient of the objective function It represents the bias vector of the objective function to the parameter, which is calculated as follows:

[0163] ;

[0164] Finally, we get accurate light field camera parameters .

[0165] Step S509 , extracting the external parameters in the final parameter estimation result, extracting the internal parameters in the initial internal and external parameters, combining the external parameters with the internal parameters, and obtaining the optimized internal and external parameters of the light field camera.

[0166] Among them, the optimized external parameters are obtained by performing optimization several times, and the optimized internal and external parameters are obtained by combining them with the internal parameters in the initial internal and external parameters.

[0167] The embodiment of the present application takes into account the influence of distortion. In order to optimize the distortion and obtain optimized parameters, an approximate linear equation method is used to reduce the complexity of the optimization model and improve the efficiency of distortion optimization. After the first-order distortion coefficient optimizes the parameters, the second-order distortion coefficient is introduced into the distortion model, and the light field rays of the points of the calibration plate are transformed and predicted to obtain predicted values, and the reprojection error is calculated with the three-dimensional camera coordinates obtained by the above calculation, so as to complete the optimization of parameters on the basis of the first-order and second-order distortions, and obtain more accurate parameter optimization results. With the help of a layered multi-optimization method, the optimization process can avoid falling into a local minimum, and the stability of solving complex high-dimensional nonlinear equations is improved.

[0168] Corresponding to the optimization method for light field camera calibration in the above embodiment, Figure 6 The structural block diagram of the optimization device for light field camera calibration provided in the fifth embodiment of the present application is shown. For the convenience of description, only the part related to the embodiment of the present application is shown.

[0169] See also Figure 6 , the optimization device for light field camera calibration comprises:

[0170] The feature extraction module 61 is used to obtain a light field image in a calibration plate scene acquired by a light field camera, determine the calibration internal and external parameters used for light field camera calibration according to the light field image, extract feature points from the light field image, select a preset number of feature points as control points, and determine the image coordinates of each control point;

[0171] A coordinate transformation module 62 is used to take any pixel point in the light field image as a target point, obtain the image coordinate difference between the target point and each control point based on the radial basis function, use a linear transformation function to linearly transform the image coordinate of the target point to obtain a linear transformation coordinate, and perform a weighted summation on all image coordinate differences to obtain a summation result;

[0172] An energy function module 63 is used to construct the distorted coordinates of the target point according to the summation result and the linear transformation coordinates, determine the reprojection error according to the image coordinates and the distorted coordinates of the target point, use the constraint condition to constrain the weight corresponding to each image coordinate difference, obtain the regularization term, and use the sum of the reprojection error and the regularization term as the energy function;

[0173] The correction coordinate calculation module 64 is used to iteratively optimize the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the condition of the minimum energy function is satisfied, thereby obtaining the optimized linear parameters and the optimized weights, and calculating the distorted coordinates of the feature points according to the optimized linear parameters and the optimized weights to obtain the correction coordinates;

[0174] The calibration parameter optimization module 65 is used to project the image coordinates of the feature points using the calibration internal and external parameters to obtain calibration coordinates, and optimize the calibration internal and external parameters with the goal of minimizing the difference between the correction coordinates and the calibration coordinates to obtain optimized calibration internal and external parameters.

[0175] Optionally, the correction coordinate calculation module 64 includes:

[0176] A partial derivative calculation unit is used to discretize the pixels of the light field image using a finite element method to obtain discrete pixel points, and to calculate partial derivatives of the discrete pixel points with respect to the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference to obtain partial derivatives of each linear parameter and weight;

[0177] The iterative optimization unit is used to use the partial derivatives of each linear parameter and weight as the elements of the Jacobian matrix, and use the gradient descent method to iteratively optimize the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the reprojection error converges to a preset threshold or reaches the maximum number of iterations, thereby obtaining the optimized linear parameters and optimized weights.

[0178] Optionally, the feature extraction module 61 includes:

[0179] A frequency domain screening unit is used to obtain initial light field data in a calibration plate scene based on light field camera acquisition, perform frequency domain conversion on the directional plane coordinates in the initial light field data to obtain frequency domain light field data, and screen the frequency domain light field image under the condition that the horizontal frequency domain value or the vertical frequency domain value is greater than the frequency domain threshold value to obtain light field component data;

[0180] A subview pair determination unit, configured to use the image captured by the acquisition unit of the light field camera as a subview, and pair all subviews according to the light field component data to obtain at least one matching subview pair;

[0181] A matching point pair determination unit is used to perform pairing for any subview pair according to the corner point positions of the calibration plate collected in the subview pair, generate matching point pairs, and traverse all subview pairs to obtain all matching point pairs;

[0182] The light field camera calibration unit is used to construct the epipolar constraint equations of the corresponding matching point pairs by combining the basic matrices formed by each matching point pair with the internal and external parameters, and calculate the initial internal and external parameters with the goal of minimizing all the epipolar constraint equations. Based on the initial internal and external parameters, the calibration internal and external parameters used for light field camera calibration are determined.

[0183] Optionally, a light field camera calibration unit comprises:

[0184] The light field camera calibration subunit is used to perform a primary optimization on the initial internal and external parameters to obtain the internal and external parameters after the primary optimization, and determine the internal and external parameters after the initial optimization as the calibration internal and external parameters.

[0185] Optionally, the light field camera calibration subunit is specifically used for:

[0186] Get the 3D world coordinates of any point in the calibration plate in the world coordinate system, and use the initial internal and external parameters to convert the 3D world coordinates into 3D camera coordinates in the camera coordinate system;

[0187] Obtain the coefficient matrix of the light field camera, where the coefficient matrix represents the transformation relationship between the three-dimensional position of any point in the calibration plate in the world coordinate system and the measured value of the light field ray;

[0188] Obtain a calibration matrix, set the nonlinear parameters in the calibration matrix to fixed values, and obtain an approximate calibration matrix, where the calibration matrix represents the transformation relationship between the parameters of the light field rays in the light field camera and the light on the image plane;

[0189] Under the first-order distortion coefficient, the approximate calibration matrix, the external parameters to be optimized and the first-order distortion coefficient form the parameter vector to be solved. The result of multiplying the coefficient matrix and the parameter vector to be solved is used to construct a linear equation with the three-dimensional camera coordinates. The linear equation is solved using the singular value decomposition method to obtain the preliminary parameter estimation result of the parameter vector to be solved.

[0190] According to the calibration matrix and the first-order distortion coefficient in the preliminary parameter estimation result, combined with the second-order distortion coefficient, the measurement value of the light field ray reaching the three-dimensional camera coordinate is predicted to obtain the first coordinate prediction value;

[0191] With the goal of minimizing the reprojection error between the first coordinate prediction value and the three-dimensional camera coordinate, the Gauss-Newton iterative algorithm is used to iteratively update the preliminary parameter estimation result to obtain an updated parameter estimation result;

[0192] According to the calibration matrix, the first-order distortion coefficient, the second-order distortion coefficient in the updated parameter estimation result, combined with the third-order distortion coefficient, the measurement value of the light field ray reaching the three-dimensional camera coordinate is predicted to obtain the second coordinate prediction value;

[0193] The goal is to minimize the reprojection error between the second coordinate prediction value and the 3D camera coordinate, and the objective function and the bias vector of the parameter are used as the gradient to perform trust region optimization on the updated parameter estimation results to obtain the final parameter estimation results. The final parameter estimation results are used to obtain the optimized internal and external parameters.

[0194] The external parameters in the final parameter estimation result are extracted, and the internal parameters in the initial internal and external parameters are extracted, and the external parameters are combined with the internal parameters to obtain the optimized internal and external parameters of the light field camera.

[0195] Optionally, the subview pair determination unit includes:

[0196] An image feature extraction subunit, used to determine image feature information corresponding to all subviews according to the light field component data;

[0197] The adjacent subview matching subunit is used to obtain the adjacent subviews of any subview, perform feature point matching between the image feature information of the subview and the image feature information of each adjacent subview, and determine the adjacent subview that matches the subview as the matching subview;

[0198] The subview pair determination subunit is used to take a subview and a matching subview as a subview pair, traverse all subviews, and obtain at least one matching subview pair.

[0199] Optionally, the matching point pair determining unit includes:

[0200] A corner point detection subunit is used to detect corner points of the calibration plate image contained in the first subview and the second subview of the subview pair, and obtain a first group of corner points corresponding to the first subview and a first corner point position corresponding to each corner point, and a second group of corner points corresponding to the second subview and a second corner point position corresponding to each corner point;

[0201] The corner point matching subunit is used to match each corner point of the first group of corner points with each corner point of the second group of corner points one by one, and determine the matched corner points as a matching point pair, wherein the matching point pair includes a first corner point position of the corresponding corner point in the first group of corner points and a second corner point position of the corresponding corner point in the second group of corner points.

[0202] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0203] Figure 7 This is a schematic diagram of the structure of a light field camera provided in Example 6 of the present application. Figure 7 As shown, the light field camera of this embodiment includes: at least one processor ( Figure 7 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned optimization method embodiments for light field camera calibration are implemented.

[0204] The light field camera may include, but is not limited to, a processor and a memory. A person skilled in the art will understand that Figure 7The light field camera is merely an example and does not constitute a limitation on the light field camera. The light field camera may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0205] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0206] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of the light field camera, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be the hard disk of the light field camera, and in other embodiments, it may also be an external storage device of the light field camera, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the light field camera. Furthermore, the memory may also include both an internal storage unit of the light field camera and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0207] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0208] The present application implements all or part of the processes in the above-mentioned method embodiments, and may also be completed through a computer program product. When the computer program product runs on a light field camera, the light field camera can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0209] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0210] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0211] In the embodiments provided in the present application, it should be understood that the disclosed device / light field camera and method can be implemented in other ways. For example, the device / light field camera embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An optimization method for light field camera calibration, characterized in that: The optimization method for light field camera calibration includes: Acquire a light field image of a calibration plate scene acquired by a light field camera, determine calibration internal and external parameters for calibration of the light field camera based on the light field image, extract feature points from the light field image, select a preset number of feature points as control points, and determine the image coordinates of each control point; Taking any pixel point in the light field image as a target point, constructing an image coordinate difference between the target point and each control point based on a radial basis function, performing a linear transformation on the image coordinate of the target point using a linear transformation function to obtain a linear transformation coordinate, and performing a weighted summation on all image coordinate differences to obtain a summation result; Constructing the distorted coordinates of the target point according to the summation result and the linear transformation coordinates, determining the reprojection error according to the image coordinates of the target point and the distorted coordinates, constraining the weight corresponding to each image coordinate difference using constraint conditions to obtain a regularization term, and taking the sum of the reprojection error and the regularization term as an energy function; Iteratively optimizing the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the condition of the minimum energy function is satisfied, thereby obtaining optimized linear parameters and optimized weights, and calculating the distorted coordinates of the feature points according to the optimized linear parameters and the optimized weights to obtain corrected coordinates; The image coordinates of the feature points are projected using the calibration internal and external parameters to obtain calibration coordinates, and the calibration internal and external parameters are optimized with the goal of minimizing the difference between the correction coordinates and the calibration coordinates to obtain optimized calibration internal and external parameters.

2. The optimization method for light field camera calibration according to claim 1, characterized in that: The iterative optimization of the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the condition of the minimum energy function is satisfied to obtain the optimized linear parameters and the optimized weights includes: Using a finite element method to discretize pixels of the light field image to obtain discrete pixel points, and respectively calculating partial derivatives of linear parameters of the discrete pixel points in the linear transformation function and weights corresponding to each image coordinate difference to obtain partial derivatives of each linear parameter and weight; The partial derivatives of each linear parameter and weight are used as elements of the Jacobian matrix, and the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference are iteratively optimized using the gradient descent method until the reprojection error converges to a preset threshold or reaches a maximum number of iterations, thereby obtaining optimized linear parameters and optimized weights.

3. The optimization method for light field camera calibration according to any one of claims 1 to 2, characterized in that: The step of determining calibration internal and external parameters for calibrating the light field camera according to the light field image includes: Acquire initial light field data corresponding to the light field image, perform frequency domain conversion on the directional plane coordinates in the initial light field data to obtain frequency domain light field data, and filter the frequency domain light field data on the condition that the horizontal frequency domain value or the vertical frequency domain value is greater than a frequency domain threshold to obtain light field component data; Using the image captured by the acquisition unit of the light field camera as a sub-view, and pairing all the sub-views according to the light field component data to obtain at least one matching pair of sub-views; For any subview pair, pair them according to the corner point positions of the calibration plate collected in the subview pair to generate matching point pairs, and traverse all subview pairs to obtain all matching point pairs; The basic matrix formed by each matching point pair and the internal and external parameters is used to construct the epipolar constraint equation of the corresponding matching point pair. The initial internal and external parameters are calculated with the goal of minimizing all the epipolar constraint equations. According to the initial internal and external parameters, the calibration internal and external parameters used for the light field camera calibration are determined.

4. The optimization method for light field camera calibration according to claim 3, characterized in that: The step of determining calibration internal and external parameters for calibrating the light field camera according to the initial internal and external parameters includes: The initial internal and external parameters are initially optimized to obtain internal and external parameters after initial optimization, and the internal and external parameters after initial optimization are determined as calibration internal and external parameters.

5. The optimization method for light field camera calibration according to claim 4, characterized in that: The initial internal and external parameters are initially optimized to obtain the internal and external parameters after initial optimization, including: Obtaining the three-dimensional world coordinates of any point in the calibration plate in the world coordinate system, and converting the three-dimensional world coordinates into three-dimensional camera coordinates in the camera coordinate system using the initial internal and external parameters; Acquire a coefficient matrix of the light field camera, wherein the coefficient matrix represents a transformation relationship between a three-dimensional position of any point in the calibration plate in a world coordinate system and a measured value of a light field ray; Acquire a calibration matrix, set nonlinear parameters in the calibration matrix to fixed values, and obtain an approximate calibration matrix, wherein the calibration matrix represents a transformation relationship between parameters of light field rays in the light field camera and light on an image plane; Under the first-order distortion coefficient, the approximate calibration matrix, the external parameter to be optimized and the first-order distortion coefficient form a parameter vector to be solved, a linear equation is constructed by multiplying the coefficient matrix and the parameter vector to be solved and the three-dimensional camera coordinates, and the linear equation is solved by using the singular value decomposition method to obtain a preliminary parameter estimation result of the parameter vector to be solved; According to the calibration matrix and the first-order distortion coefficient in the preliminary parameter estimation result, combined with the second-order distortion coefficient, the measurement value of the light field ray reaching the three-dimensional camera coordinate is predicted to obtain a first coordinate prediction value; With the goal of minimizing the reprojection error between the first coordinate prediction value and the three-dimensional camera coordinate, the preliminary parameter estimation result is iteratively updated using a Gauss-Newton iterative algorithm to obtain an updated parameter estimation result; According to the calibration matrix, the first-order distortion coefficient, the second-order distortion coefficient in the update parameter estimation result, combined with the third-order distortion coefficient, the measurement value of the light field ray reaching the three-dimensional camera coordinate is predicted to obtain a second coordinate prediction value; Taking the minimum reprojection error between the second coordinate prediction value and the three-dimensional camera coordinate as the goal, and taking the objective function and the bias vector of the parameter as the gradient, performing trust region optimization on the updated parameter estimation result to obtain a final parameter estimation result, wherein the final parameter estimation result is used to obtain optimized internal and external parameters; The external parameters in the final parameter estimation result are extracted, and the internal parameters in the initial internal and external parameters are extracted, and the external parameters are combined with the internal parameters to obtain the optimized internal and external parameters of the light field camera.

6. The optimization method for light field camera calibration according to claim 3, characterized in that: The step of pairing all sub-views according to the light field component data to obtain at least one pair of matching sub-views comprises: Determining image feature information corresponding to all sub-views according to the light field component data; For any subview, obtain an adjacent subview of the subview, perform feature point matching between image feature information of the subview and image feature information of each adjacent subview, and determine the adjacent subview that matches the subview as a matching subview; The subview and the matching subview are regarded as a subview pair, and all subviews are traversed to obtain at least one matching subview pair.

7. The optimization method for light field camera calibration according to claim 3, characterized in that: The step of pairing the corner points of the calibration plate collected in the subview pair to generate matching point pairs includes: Detecting corner points of the calibration plate image included in the first subview and the second subview of the subview pair, and obtaining a first group of corner points corresponding to the first subview and a first corner point position corresponding to each corner point, and a second group of corner points corresponding to the second subview and a second corner point position corresponding to each corner point; Each corner point of the first group of corner points is matched one by one with each corner point of the second group of corner points, and the matched corner points are determined to be a matching point pair, wherein the matching point pair includes a first corner point position of a corresponding corner point in the first group of corner points and a second corner point position of a corresponding corner point in the second group of corner points.

8. An optimization device for light field camera calibration, characterized in that: The optimization device for light field camera calibration comprises: A feature extraction module is used to obtain a light field image in a calibration plate scene acquired by a light field camera, determine calibration internal and external parameters for calibration of the light field camera based on the light field image, extract feature points from the light field image, select a preset number of feature points as control points, and determine the image coordinates of each control point; A coordinate transformation module is used to take any pixel point in the light field image as a target point, obtain the image coordinate difference between the target point and each control point based on a radial basis function, use a linear transformation function to linearly transform the image coordinates of the target point to obtain linear transformation coordinates, and perform weighted summation on all image coordinate differences to obtain a summation result; An energy function module is used to construct the distorted coordinates of the target point according to the summation result and the linear transformation coordinates, determine the reprojection error according to the image coordinates of the target point and the distorted coordinates, use constraint conditions to constrain the weight corresponding to each image coordinate difference to obtain a regularization term, and use the sum of the reprojection error and the regularization term as an energy function; A correction coordinate calculation module, used for iteratively optimizing the linear parameters in the linear transformation function and the weights corresponding to each image coordinate difference until the condition of the minimum energy function is satisfied, obtaining the optimized linear parameters and the optimized weights, and calculating the distorted coordinates of the feature points according to the optimized linear parameters and the optimized weights to obtain the correction coordinates; The calibration parameter optimization module is used to use the calibration internal and external parameters to project the image coordinates of the feature points to obtain calibration coordinates, and optimize the calibration internal and external parameters with the goal of minimizing the difference between the correction coordinates and the calibration coordinates to obtain optimized calibration internal and external parameters.

9. A light field camera, characterized in that: The light field camera comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the optimization method for light field camera calibration according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the optimization method for light field camera calibration according to any one of claims 1 to 7 is implemented.

Images