A method, system, device and medium for projected single-pose calibration

Abstract

The present invention discloses a method, system, device and medium for projection single-pose calibration, which relates to the field of computer vision and projection interaction technology. The method includes: generating multiple calibration patterns, projecting them onto a calibration plane, collecting grayscale images, performing corner detection and sub-pixel refinement processing, obtaining the corner point correspondence between the projector image and the camera image, iteratively solving the projection matrix of the projector and the camera, separating the intrinsic parameter matrix and the extrinsic parameter matrix, calculating the basic matrix and the essential matrix, and decomposing the rotation matrix and translation vector of the camera relative to the projector; estimating the camera imaging distortion, obtaining the distortion correction coefficient, and performing distortion correction; statistically calculating the reprojection error generated during the correction process, and adjusting the parameters of the projector and camera until the error meets the preset accuracy requirements. Projector parameter calculation is completed by projection single-pose calibration, and the calibration process does not require posture adjustment, thereby improving the degree of automation and reducing manual intervention.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and projection interaction technology, and in particular to a method, system, device and medium for projection single posture calibration. Background Art

[0002] Traditional projection calibration methods rely on multiple shots of a high-precision calibration plate with multi-angle projection pose adjustments. This method relies heavily on manual intervention, is inefficient, and is prone to errors. Existing monocular projection pose estimation methods have limited accuracy and are unable to meet the increasingly precise market demands for projection. Traditional projection pose estimation solutions rely on feature point recognition, but poor matching results are observed in brightly lit environments, limiting the use cases of projectors. Existing projection pose estimation technologies rely heavily on the projector's hardware, significantly limiting the scope of options. Summary of the Invention

[0003] Based on the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a method, system, device and medium for projection single-pose calibration to solve the above-mentioned technical problems.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for projected single pose calibration, comprising:

[0005] S1: Dynamically generate multiple spatially linearly independent calibration patterns, project the calibration patterns onto the calibration plane through a projector, and collect grayscale images;

[0006] S2: Perform corner detection and sub-pixel refinement on the grayscale image to obtain the corresponding relationship between the corners of the projector image and the camera image;

[0007] S3: Based on the corresponding relationship of the corner points, the projection matrix of the projector and camera is iteratively solved to separate the intrinsic parameter matrix and the extrinsic parameter matrix;

[0008] S4: Calculate the fundamental matrix and the essential matrix according to the epipolar geometry constraint, and decompose them to obtain the rotation matrix and translation vector of the camera relative to the projector;

[0009] S5: Estimate the camera imaging distortion based on the external and internal parameters to obtain a distortion correction coefficient, and perform distortion correction on the camera image based on the distortion correction coefficient;

[0010] S6: Count the reprojection errors generated during the correction process, and dynamically adjust the parameters of the projector and camera according to the reprojection errors until the reprojection errors meet the preset accuracy requirements.

[0011] The present invention is further configured such that the first calibration pattern is composed of ArUco marks at the four corners and a checkerboard in the center, and the remaining calibration patterns are generated from the nonlinear image transformation of the first one. Different calibration patterns must ensure that the object points to be identified are linearly independent in three-dimensional space.

[0012] The present invention is further configured such that S2 includes:

[0013] By calculating the discrete second-order derivative field of the pixel point and the neighborhood grayscale value on the grayscale image, a nonlinear corner response function is constructed to reflect the second-order curvature characteristics of the candidate corner point area;

[0014] Preliminary corner point screening is performed based on the response function value and the preset threshold. The sub-pixel offset of the grayscale extreme point is obtained by local quadratic surface fitting in the neighborhood of each preliminary corner point to achieve the refinement of the corner point position.

[0015] According to the checkerboard or ArUco coding information in the deformed pattern, the projector pattern coding corner points are mapped and matched with the refined camera corner points using the same index rule to establish the corner point correspondence between the projector pattern and the camera image.

[0016] The present invention is further configured such that S3 includes:

[0017] Based on the corner point correspondence, the 3D homogeneous object points and the corresponding camera pixel coordinates are substituted into the homogeneous projection equation, which is expanded into linear constraints in the horizontal and vertical directions of the pixels. The linear constraints are combined row by row to generate the homogeneous linear equation system of the camera projection matrix.

[0018] Perform singular value decomposition on the homogeneous linear equations of the camera projection matrix, and set the eigenvector corresponding to the minimum singular value as the camera projection matrix; extract the first to third columns of the camera projection matrix as matrix blocks, perform RQ decomposition, and obtain the camera intrinsic parameter matrix and extrinsic parameter matrix.

[0019] The present invention is further configured such that S4 includes:

[0020] For each pair of projector and camera images, the corresponding coordinates of the sub-pixel refined corner points are centered and scaled using affine transformation on the projector pixel coordinates and the camera pixel coordinates respectively.

[0021] The normalized projector pixel coordinates and the normalized camera pixel coordinates are used to construct a homogeneous linear equation system of the basic matrix through the epipolar constraint equation. The linear equation system is subjected to singular value decomposition, and the eigenvector corresponding to the minimum singular value is extracted to construct the intrinsic basic matrix. The basic matrix is ​​subjected to a rank-two constraint to eliminate the influence of noise, and the constrained normalized basic matrix is ​​denormalized back to the original pixel coordinate space to obtain the basic matrix;

[0022] Based on the basic matrix, the projector intrinsic parameter matrix and the camera intrinsic parameter matrix, the essential matrix is ​​obtained by matrix product, and the essential matrix is ​​constrained to have equal singular values ​​and reconstructed so that the essential matrix only retains two equal singular values ​​and one zero singular value;

[0023] Perform singular value decomposition on the constrained essential matrix, and combine the predefined orthogonal transformation matrix and the vectors obtained by the essential matrix decomposition to generate four sets of rotation matrix and translation vector candidate solutions;

[0024] For the four sets of candidate solutions, any three-dimensional object point is placed in the projector and camera coordinate systems, and it is determined whether the projected depth values ​​are all greater than zero, and the rotation matrix and translation vector that satisfy the depth positivity constraint are retained.

[0025] The present invention is further configured such that S5 includes:

[0026] The three-dimensional homogeneous coordinates of each calibrated corner point are mapped to a normalized plane based on the camera intrinsic parameters and extrinsic parameters. The pixel coordinates of the same corner point are denormalized to the normalized plane using the camera intrinsic parameters to obtain the ideal normalized coordinates and the actual normalized coordinates.

[0027] Based on the deviation between the ideal normalized coordinates and the actual normalized coordinates, a linear equation system involving radial distortion coefficients and tangential distortion coefficients is constructed using high-order radial polynomials and cross-tangential polynomials. The radial distortion coefficients and tangential distortion coefficients are obtained through singular value decomposition.

[0028] The coordinates of any pixel in the image are first denormalized to the distorted normalized coordinates based on the camera intrinsic parameters. The distorted normalized coordinates are used as the initial values, and the corresponding ideal normalized coordinates are obtained through iterative solution. The ideal normalized coordinates are projected back to the pixel plane according to the camera intrinsic parameters to complete the distortion correction of the camera image.

[0029] The present invention is further configured such that S6 includes:

[0030] The calibrated 3D object points are mapped to the projector pixel plane using the current projector intrinsic and extrinsic parameters, and then mapped to the camera pixel plane using the current camera intrinsic, extrinsic, and distortion coefficients. The difference between the projector predicted pixel coordinates and the actual calibrated corner pixel coordinates, and the difference between the camera predicted pixel coordinates and the actual calibrated corner pixel coordinates are calculated to construct the reprojection residual vector.

[0031] The Jacobian matrix is ​​constructed based on the residual vector, and the partial derivatives of the residual with respect to the intrinsic parameters, extrinsic parameters and distortion coefficients of the projector and camera are arranged in columns. The incremental equation is formed by combining the preset weight matrix and the regularization term.

[0032] Solve the incremental equation to obtain the updated amount of the projector and camera parameters, and apply the updated amount to the projector and camera parameters;

[0033] Based on the updated parameters, all 3D object points are remapped to the projector and camera pixel planes, and the sum of the reprojection errors is calculated. When the reprojection error is greater than the preset threshold and the number of iterations has not reached the upper limit, the residual is calculated, the Jacobian is constructed, and the parameters are updated repeatedly until the reprojection error is less than or equal to the preset threshold or the maximum number of iterations is reached.

[0034] The present invention also provides a device for projection single-pose calibration, which is used to implement the above-mentioned projection single-pose calibration method, comprising:

[0035] Calibration generation module: dynamically generates multiple spatially linearly independent calibration patterns, projects the calibration patterns onto the calibration plane through a projector, and collects grayscale images;

[0036] Correspondence acquisition module: performs corner detection and sub-pixel refinement on the grayscale image to obtain the corner correspondence between the projector image and the camera image;

[0037] Matrix separation module: Based on the corner point correspondence, it iteratively solves the projection matrix of the projector and camera to separate the intrinsic parameter matrix and the extrinsic parameter matrix;

[0038] Matrix calculation module: Calculates the basic matrix and essential matrix according to the epipolar geometry constraints, and decomposes them to obtain the rotation matrix and translation vector of the camera relative to the projector;

[0039] Distortion correction module: estimates the camera imaging distortion based on external and internal parameters, obtains the distortion correction coefficient, and performs distortion correction on the camera image based on the distortion correction coefficient.

[0040] The present invention further provides an electronic device, comprising:

[0041] one or more processors;

[0042] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements a method for projection single posture calibration as described in any one of the above items.

[0043] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer is caused to execute a method for projection single-pose calibration as described in any one of the above items.

[0044] The present invention provides a method, system, device and medium for projector single-pose calibration. The method dynamically generates multiple spatially linearly independent calibration patterns, projects the calibration patterns onto a calibration plane through a projector, and collects a grayscale image; performs corner point detection and sub-pixel refinement processing on the grayscale image to obtain a corner point correspondence relationship between the projector image and the camera image; iteratively solves the projection matrix of the projector and the camera based on the corner point correspondence relationship, and separates the intrinsic parameter matrix and the extrinsic parameter matrix; calculates the basic matrix and the essential matrix according to the epipolar geometry constraint, and decomposes them to obtain the rotation matrix and translation vector of the camera relative to the projector; estimates the camera imaging distortion based on the extrinsic and intrinsic parameters to obtain a distortion correction coefficient, and performs distortion correction on the camera image based on the distortion correction coefficient; calculates the reprojection error generated during the correction process, and dynamically adjusts the parameters of the projector and the camera based on the reprojection error until the reprojection error meets the preset accuracy requirement. The beneficial effects produced include:

[0045] 1. The projector parameters are calculated through single-pose calibration. The calibration process does not require posture adjustment, which improves the degree of automation and reduces manual intervention.

[0046] 2. A specialized projection pose estimation algorithm is developed for projection to improve the recognition accuracy of calibration patterns, increase accuracy under lighting conditions, and reduce dependence on projection hardware.

[0047] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. In the drawings:

[0049] Figure 1 This is a flow chart of a method for projected single pose calibration shown as an exemplary embodiment of the present invention;

[0050] Figure 2 A 3D schematic diagram of a method for projected single pose calibration is shown as an exemplary embodiment of the present invention;

[0051] Figure 3 A single-pose calibration flow chart of a projection single-pose calibration method according to an exemplary embodiment of the present invention is shown;

[0052] Figure 4 A calibration pattern diagram is shown for an exemplary embodiment of the present invention;

[0053] Figure 5 The figure is a schematic structural diagram of a device for projected single pose calibration according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0055] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0056] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0057] Example 1:

[0058] First of all, it should be noted that the invention consists of the following main parts:

[0059] Camera sensor (Camera): responsible for capturing the current projection image and transmitting it to the chip controller (SoC); the field of view (FOV) of this camera must be larger than the field of view (FOV) of the projector lens (Projector);

[0060] LCD display (Panel): Responsible for displaying several calibration patterns generated by the current chip controller (SoC). The display timing of this LCD must be approximately 3 to 4 frames earlier than the camera's acquisition timing (related to the SoC processing performance).

[0061] Projector: Responsible for projecting the image on the LCD screen. The calibration pattern displayed on the LCD screen is projected through the projection lens, forming a clear projection image on the wall or screen. This image can be fully captured by the camera sensor and maintain a fixed distance from the wall or screen.

[0062] Chip controller (SoC): responsible for processing the entire projection system and performing a projection single pose calibration method.

[0063] A method of projective single pose calibration, such as Figure 1 and Figure 3 Shown, including:

[0064] S1: Dynamically generate multiple spatially linearly independent calibration patterns, project the calibration patterns onto the calibration plane through a projector, and collect grayscale images;

[0065] S2: Perform corner detection and sub-pixel refinement on the grayscale image to obtain the corresponding relationship between the corners of the projector image and the camera image;

[0066] S3: Based on the corresponding relationship of the corner points, the projection matrix of the projector and camera is iteratively solved to separate the intrinsic parameter matrix and the extrinsic parameter matrix;

[0067] S4: Calculate the fundamental matrix and the essential matrix according to the epipolar geometry constraint, and decompose them to obtain the rotation matrix and translation vector of the camera relative to the projector;

[0068] S5: Estimate the camera imaging distortion based on the external and internal parameters to obtain a distortion correction coefficient, and perform distortion correction on the camera image based on the distortion correction coefficient;

[0069] S6: Count the reprojection errors generated during the correction process, and dynamically adjust the parameters of the projector and camera according to the reprojection errors until the reprojection errors meet the preset accuracy requirements.

[0070] like Figure 2 As shown, the rectangular box on the left represents the projector, where O1 represents the optical center of the projector (that is, the position in the external parameters of the projection matrix), and multiple rays emitted from O1 point to different three-dimensional object points on the calibration plane (gray plane). The figure takes the four corner points x1, x2, x3, x4 and the plane center x0 as examples to demonstrate how the projector maps the pre-generated spatial linearly independent calibration pattern to the calibration plane. The positions of these object points x1–x4 and x0 on the calibration plane are the results of "dynamically generating and projecting the calibration pattern" in step S1. The projector pixel coordinates can be mapped to three-dimensional object point coordinates through the projection matrix (intrinsic parameter matrix and external parameter matrix);

[0071] The slightly tilted grid plane on the right, parallel to the calibration plane but slightly set back, represents the image plane seen by the camera (CCD or CMOS sensor). The optical center of the camera is marked as O2 (which actually corresponds to the rotation matrix and translation vector in the solved camera extrinsics). The rays extending upward from the object points x1–x4 and x0 on the calibration plane eventually converge to the optical center of the camera O2, and then refract to several image points on the camera image plane. For example, in the figure, the corresponding points can be assumed to be u1, u2, u3, and u4 (on the grid background, compared with the world coordinate grid). These image points are the "corner coordinates of the camera image" obtained through corner detection and sub-pixel refinement;

[0072] The grid behind the gray plane (a sparse grid of vertical and horizontal squares) represents the coordinate system of the calibration plane in the world coordinate system. The solid points on the gray plane (the small squares) correspond to the corner points or spatial sampling points of the calibration pattern displayed on the plane and captured by the camera. x1, x2, x3, and x4 are located at the four corners, indicating that the encoded corner points constructed using the first ArUco image and the checkerboard are fixed at the four corners; x0 is an internal point at the center of the plane, which is used to illustrate that the subsequent object points must be linearly independent in three-dimensional space (the three-dimensional coordinates formed by the line connecting the projector's optical center) to meet the "spatial linear independence" constraint.

[0073] like Figure 4 As shown, the present invention is further configured such that the first calibration pattern is composed of ArUco marks at the four corners and a checkerboard in the center, and the remaining calibration patterns are generated from the nonlinear image transformation of the first one. Different calibration patterns must ensure that the object points to be identified are linearly independent in three-dimensional space.

[0074] The present invention is further configured such that S2 includes:

[0075] By calculating the discrete second-order derivative field of the pixel point and the neighborhood grayscale value on the grayscale image, a nonlinear corner response function is constructed to reflect the second-order curvature characteristics of the candidate corner point area;

[0076] Preliminary corner point screening is performed based on the response function value and the preset threshold. The sub-pixel offset of the grayscale extreme point is obtained by local quadratic surface fitting in the neighborhood of each preliminary corner point to achieve the refinement of the corner point position.

[0077] According to the checkerboard or ArUco coding information in the deformed pattern, the projector pattern coding corner points are mapped and matched with the refined camera corner points using the same index rule to establish the corner point correspondence between the projector pattern and the camera image.

[0078] Assume the homogeneous coordinates of the three-dimensional object point , which is projected onto the two-dimensional image points of the projector and camera The imaging relationship is: ,in, and is the 3×4 projection matrix of the projector and camera, and is a 3×3 internal parameter matrix, including parameters such as focal length and principal point, and are the external parameters 3×3 rotation matrix and 3×1 translation vector, is the scale factor of the homogeneous coordinates.

[0079] Assuming there is no distortion in the initial state, the projector display resolution is (pixels), the pixel pitch is and , the equivalent focal length is , then the initial internal parameter matrix for: , assuming that the projector coordinate system is aligned with the world coordinate system (no rotation, no translation), the initial projection matrix is ​​obtained by combining the internal and external parameters : , the initial camera installation position is known (rotation , with translation ), similarly, the initial camera projection matrix can be obtained: .

[0080] The present invention is further configured such that S3 includes:

[0081] Based on the corner point correspondence, the 3D homogeneous object points and the corresponding camera pixel coordinates are substituted into the homogeneous projection equation, which is expanded into linear constraints in the horizontal and vertical directions of the pixels. The linear constraints are combined row by row to generate the homogeneous linear equation system of the camera projection matrix.

[0082] Perform singular value decomposition on the homogeneous linear equations of the camera projection matrix, and set the eigenvector corresponding to the minimum singular value as the camera projection matrix; extract the first to third columns of the camera projection matrix as matrix blocks, perform RQ decomposition, and obtain the camera intrinsic parameter matrix and extrinsic parameter matrix.

[0083] Specifically, based on the obtained projector prior projection matrix (When not iterated, it is the initial projection matrix ), solve the intrinsic parameters of the camera projection matrix , input: (1) point set , by the projector Back projection Calculated, where represents the pseudo-inverse operation, (2) corresponds to the pixel set , extract sub-pixel coordinates from the camera image; for each corresponding point, based on the projection equation , which is expanded into a linear constraint ( form): ,in, express No. Row vector, pair Point construction Coefficient matrix , for the matrix Perform singular value decomposition ,Pick The last column (corresponding to the smallest singular value) of Projection matrix, you can get the camera's predicted projection matrix , separate the internal reference by RQ decomposition With external reference .

[0084] The present invention is further configured such that S4 includes:

[0085] For each pair of projector and camera images, the corresponding coordinates of the sub-pixel refined corner points are centered and scaled using affine transformation on the projector pixel coordinates and the camera pixel coordinates respectively.

[0086] The normalized projector pixel coordinates and the normalized camera pixel coordinates are used to construct a homogeneous linear equation system of the basic matrix through the epipolar constraint equation. The linear equation system is subjected to singular value decomposition, and the eigenvector corresponding to the minimum singular value is extracted to construct the intrinsic basic matrix. The basic matrix is ​​subjected to a rank-two constraint to eliminate the influence of noise, and the constrained normalized basic matrix is ​​denormalized back to the original pixel coordinate space to obtain the basic matrix;

[0087] Based on the basic matrix, the projector intrinsic parameter matrix and the camera intrinsic parameter matrix, the essential matrix is ​​obtained by matrix product, and the essential matrix is ​​constrained to have equal singular values ​​and reconstructed so that the essential matrix only retains two equal singular values ​​and one zero singular value;

[0088] Perform singular value decomposition on the constrained essential matrix, and combine the predefined orthogonal transformation matrix and the vectors obtained by the essential matrix decomposition to generate four sets of rotation matrix and translation vector candidate solutions;

[0089] For the four sets of candidate solutions, any three-dimensional object point is placed in the projector and camera coordinate systems, and it is determined whether the projected depth values ​​are all greater than zero, and the rotation matrix and translation vector that satisfy the depth positivity constraint are retained.

[0090] Specifically, the projector is regarded as a reverse camera, based on the obtained (When not iterated, it is the camera's initial projection matrix ), solve the intrinsic parameters of the updated projection matrix , input: (1) point set , by camera Back projection Calculated, where + represents pseudo-inverse operation, (2) corresponds to the pixel set , the sub-pixel coordinates are directly defined in the projector calibration pattern; for each corresponding point, based on the projection equation , which is expanded into a linear constraint ( The solution logic is the same as above, and the predicted projection matrix of the projector can be obtained. , separate the internal reference by RQ decomposition With external reference .

[0091] Based on the intrinsic parameters of the predicted projection matrix and , using epipolar geometry to solve the essential matrix With the basic moment , input: (1) image point set of the projector , the sub-pixel coordinates are directly defined in the projector calibration pattern, (2) the point set corresponding to the camera , extract sub-pixel coordinates from the camera image, and for each corresponding point, based on the epipolar constraint , which is expanded into a linear constraint ( form): , ; The solution process is the same as the above calculation logic, and the basic matrix can be obtained , the essential matrix , and decomposed by SVD ,in, correspond , get the motion of the camera relative to the projector : ,in, , , is an antisymmetric matrix operator; align the projector coordinate system with the world coordinate system again, and combine the internal and external parameters to obtain the posterior projection matrix and : , .

[0092] The present invention is further configured such that S5 includes:

[0093] The three-dimensional homogeneous coordinates of each calibrated corner point are mapped to a normalized plane based on the camera intrinsic parameters and extrinsic parameters. The pixel coordinates of the same corner point are denormalized to the normalized plane using the camera intrinsic parameters to obtain the ideal normalized coordinates and the actual normalized coordinates.

[0094] Based on the deviation between the ideal normalized coordinates and the actual normalized coordinates, a linear equation system involving radial distortion coefficients and tangential distortion coefficients is constructed using high-order radial polynomials and cross-tangential polynomials. The radial distortion coefficients and tangential distortion coefficients are obtained through singular value decomposition.

[0095] The coordinates of any pixel in the image are first denormalized to the distorted normalized coordinates based on the camera intrinsic parameters. The distorted normalized coordinates are used as the initial values, and the corresponding ideal normalized coordinates are obtained through iterative solution. The ideal normalized coordinates are projected back to the pixel plane according to the camera intrinsic parameters to complete the distortion correction of the camera image.

[0096] Specifically, a polynomial model is used to describe the distortion coefficient ,in is radial distortion, 、 is the tangential distortion. Let the normalized plane coordinates be , the coordinates after distortion are , the distortion model is: , input: (1) ideal normalized plane coordinate set , reprojected from the coordinates of the calibration pattern Calculated, the distortion normalized plane coordinate set { }, by the sub-pixel coordinates of the acquired image, back-projection The calculation is to expand the distortion polynomial for each corresponding point into a linear constraint (in the form of Ax=b): ,right Point construction 2 ×5 coefficient matrix , and 2 ×1 vector b, pair matrix Perform singular value decomposition . Solve the least squares solution through singular value decomposition ,in, is the pseudo-inverse matrix of the singular values, , retain the previous non-zero singular values ​​and truncate small singular values ​​to enhance numerical stability.

[0097] The present invention is further configured such that S6 includes:

[0098] The calibrated 3D object points are mapped to the projector pixel plane using the current projector intrinsic and extrinsic parameters, and then mapped to the camera pixel plane using the current camera intrinsic, extrinsic, and distortion coefficients. The difference between the projector predicted pixel coordinates and the actual calibrated corner pixel coordinates, and the difference between the camera predicted pixel coordinates and the actual calibrated corner pixel coordinates are calculated to construct the reprojection residual vector.

[0099] The Jacobian matrix is ​​constructed based on the residual vector, and the partial derivatives of the residual with respect to the intrinsic parameters, extrinsic parameters and distortion coefficients of the projector and camera are arranged in columns. The incremental equation is formed by combining the preset weight matrix and the regularization term.

[0100] Solve the incremental equation to obtain the updated amount of the projector and camera parameters, and apply the updated amount to the projector and camera parameters;

[0101] Based on the updated parameters, all 3D object points are remapped to the projector and camera pixel planes, and the sum of the reprojection errors is calculated. When the reprojection error is greater than the preset threshold and the number of iterations has not reached the upper limit, the residual is calculated, the Jacobian is constructed, and the parameters are updated repeatedly until the reprojection error is less than or equal to the preset threshold or the maximum number of iterations is reached.

[0102] Specifically, based on the distortion parameters after iteration , projection matrix and , calculate the reprojection error Error, , where δ(D) is the distortion correction function.

[0103] Example 2:

[0104] See also Figure 5 The exemplary device for projected single pose calibration is used to implement the above-mentioned method for projected single pose calibration, including:

[0105] Calibration generation module: dynamically generates multiple spatially linearly independent calibration patterns, projects the calibration patterns onto the calibration plane through a projector, and collects grayscale images;

[0106] Correspondence acquisition module: performs corner detection and sub-pixel refinement on the grayscale image to obtain the corner correspondence between the projector image and the camera image;

[0107] Matrix separation module: Based on the corner point correspondence, it iteratively solves the projection matrix of the projector and camera to separate the intrinsic parameter matrix and the extrinsic parameter matrix;

[0108] Matrix calculation module: Calculates the basic matrix and essential matrix according to the epipolar geometry constraints, and decomposes them to obtain the rotation matrix and translation vector of the camera relative to the projector;

[0109] Distortion correction module: estimates the camera imaging distortion based on external and internal parameters, obtains the distortion correction coefficient, and performs distortion correction on the camera image based on the distortion correction coefficient.

[0110] It should be noted that the apparatus for single-pose projection calibration provided in the above embodiment and the method for single-pose projection calibration provided in the above embodiment share the same concept. The specific manner in which each module and unit performs operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the apparatus for single-pose projection calibration provided in the above embodiment can, as needed, allocate the above functions to different functional modules, i.e., divide the internal structure of the system into different functional modules to perform all or part of the functions described above. This is not a limitation herein.

[0111] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements a method for projection single posture calibration provided in the above-mentioned embodiments.

[0112] The embodiment of the present application further provides a computer system for an electronic device. It should be noted that the computer system for an electronic device is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present application.

[0113] Specifically, a computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes, such as the methods described in the above embodiments, based on programs stored in read-only memory (ROM) or programs loaded from a storage unit into random access memory (RAM). RAM also stores various programs and data required for system operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0114] The following components are connected to the I / O interface: an input section including a keyboard and mouse; an output section including cathode ray tubes (CRTs), liquid crystal displays (LCDs), and speakers; a storage section including a hard disk; and a communication section including network interface cards such as LAN (Local Area Network) cards and modems. The communication section performs communication processing via a network such as the Internet. Drives are also connected to the I / O interface as needed. Removable media such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memories are installed in the drive as needed, allowing computer programs read from these media to be installed in the storage section as needed.

[0115] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component and / or installed from removable media. When executed by a central processing unit (CPU), the computer program performs the various functions defined in the system of the present application.

[0116] It should be noted that the computer-readable medium described in the embodiments of this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. This propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0117] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0118] Another aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for projected single-pose calibration. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0119] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method for projected single pose calibration provided in each of the above embodiments.

[0120] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, any equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A method for projected single pose calibration, characterized in that: include: S1: Dynamically generate multiple spatially linearly independent calibration patterns, project them onto the calibration plane using a projector, and acquire grayscale images. The first calibration pattern consists of ArUco markers at the four corners and a checkerboard in the center. The remaining calibration patterns are generated from the nonlinear image transformation of the first one. Different calibration patterns must ensure that the object points to be identified are linearly independent in three-dimensional space. S2: Perform corner detection and sub-pixel refinement on the grayscale image to obtain the corresponding relationship between the corners of the projector image and the camera image; including: calculating the discrete second-order derivative field of the pixel points and the grayscale values ​​of the neighborhood on the grayscale image to construct a nonlinear corner response function to reflect the second-order curvature characteristics of the candidate corner point area; performing preliminary corner point screening based on the response function value and the preset threshold, obtaining the sub-pixel offset of the grayscale extreme point by local quadratic surface fitting in the neighborhood of each preliminary corner point, and realizing the refinement of the corner point position; mapping and matching the projector pattern coded corner points with the refined camera corner points using the same index rule according to the checkerboard or ArUco coding information in the deformed pattern, and establishing the corresponding relationship between the corner points of the projector pattern and the camera image; S3: Based on the corresponding relationship of the corner points, the projection matrix of the projector and camera is iteratively solved to separate the intrinsic parameter matrix and the extrinsic parameter matrix; S4: Calculate the fundamental matrix and the essential matrix according to the epipolar geometry constraint, and decompose them to obtain the rotation matrix and translation vector of the camera relative to the projector; S5: Estimate the camera imaging distortion based on the external and internal parameters to obtain a distortion correction coefficient, and perform distortion correction on the camera image based on the distortion correction coefficient; S6: Count the reprojection errors generated during the correction process, and dynamically adjust the parameters of the projector and camera according to the reprojection errors until the reprojection errors meet the preset accuracy requirements.

2. The method for projected single pose calibration according to claim 1, characterized in that: S3 includes: Based on the corner point correspondence, the 3D homogeneous object points and the corresponding camera pixel coordinates are substituted into the homogeneous projection equation, which is expanded into linear constraints in the horizontal and vertical directions of the pixels. The linear constraints are combined row by row to generate the homogeneous linear equation system of the camera projection matrix. Perform singular value decomposition on the homogeneous linear equations of the camera projection matrix, and set the eigenvector corresponding to the minimum singular value as the camera projection matrix; extract the first to third columns of the camera projection matrix as matrix blocks, perform RQ decomposition, and obtain the camera intrinsic parameter matrix and extrinsic parameter matrix.

3. The method for projection single pose calibration according to claim 1, characterized in that: S4 include: For each pair of projector and camera images, the corresponding coordinates of the sub-pixel refined corner points are centered and scaled using affine transformation on the projector pixel coordinates and the camera pixel coordinates respectively. The normalized projector pixel coordinates and the normalized camera pixel coordinates are used to construct a homogeneous linear equation system of the basic matrix through the epipolar constraint equation. The linear equation system is subjected to singular value decomposition, and the eigenvector corresponding to the minimum singular value is extracted to construct the intrinsic basic matrix. The basic matrix is ​​subjected to a rank-two constraint to eliminate the influence of noise, and the constrained normalized basic matrix is ​​denormalized back to the original pixel coordinate space to obtain the basic matrix; Based on the basic matrix, the projector intrinsic parameter matrix and the camera intrinsic parameter matrix, the essential matrix is ​​obtained by matrix product, and the essential matrix is ​​constrained to have equal singular values ​​and reconstructed so that the essential matrix only retains two equal singular values ​​and one zero singular value; Perform singular value decomposition on the constrained essential matrix, and combine the predefined orthogonal transformation matrix and the vectors obtained by the essential matrix decomposition to generate four sets of rotation matrix and translation vector candidate solutions; For the four sets of candidate solutions, any three-dimensional object point is placed in the projector and camera coordinate systems, and it is determined whether the projected depth values ​​are all greater than zero, and the rotation matrix and translation vector that satisfy the depth positivity constraint are retained.

4. The method for projected single pose calibration according to claim 1, wherein S5 include: The three-dimensional homogeneous coordinates of each calibrated corner point are mapped to a normalized plane based on the camera intrinsic parameters and extrinsic parameters. The pixel coordinates of the same corner point are denormalized to the normalized plane using the camera intrinsic parameters to obtain the ideal normalized coordinates and the actual normalized coordinates. Based on the deviation between the ideal normalized coordinates and the actual normalized coordinates, a linear equation system involving radial distortion coefficients and tangential distortion coefficients is constructed using high-order radial polynomials and cross-tangential polynomials. The radial distortion coefficients and tangential distortion coefficients are obtained through singular value decomposition. The coordinates of any pixel in the image are first denormalized to the distorted normalized coordinates based on the camera intrinsic parameters. The distorted normalized coordinates are used as the initial values, and the corresponding ideal normalized coordinates are obtained through iterative solution. The ideal normalized coordinates are projected back to the pixel plane according to the camera intrinsic parameters to complete the distortion correction of the camera image.

5. The method for projected single pose calibration according to claim 1, wherein S6 include: The calibrated 3D object points are mapped to the projector pixel plane using the current projector intrinsic and extrinsic parameters, and then mapped to the camera pixel plane using the current camera intrinsic, extrinsic, and distortion coefficients. The difference between the projector predicted pixel coordinates and the actual calibrated corner pixel coordinates, and the difference between the camera predicted pixel coordinates and the actual calibrated corner pixel coordinates are calculated to construct the reprojection residual vector. The Jacobian matrix is ​​constructed based on the residual vector, and the partial derivatives of the residual with respect to the intrinsic parameters, extrinsic parameters and distortion coefficients of the projector and camera are arranged in columns. The incremental equation is formed by combining the preset weight matrix and the regularization term. Solve the incremental equation to obtain the updated amount of the projector and camera parameters, and apply the updated amount to the projector and camera parameters; Based on the updated parameters, all 3D object points are remapped to the projector and camera pixel planes, and the sum of the reprojection errors is calculated. When the reprojection error is greater than the preset threshold and the number of iterations has not reached the upper limit, the residual is calculated, the Jacobian is constructed, and the parameters are updated repeatedly until the reprojection error is less than or equal to the preset threshold or the maximum number of iterations is reached.

6. A device for projecting single pose calibration, used to implement the method for projecting single pose calibration according to any one of claims 1 to 5, characterized in that: include: Calibration generation module: dynamically generates multiple spatially linearly independent calibration patterns, projects the calibration patterns onto the calibration plane through a projector, and collects grayscale images; Correspondence acquisition module: performs corner detection and sub-pixel refinement on the grayscale image to obtain the corner correspondence between the projector image and the camera image; Matrix separation module: Based on the corner point correspondence, it iteratively solves the projection matrix of the projector and camera to separate the intrinsic parameter matrix and the extrinsic parameter matrix; Matrix calculation module: Calculates the basic matrix and essential matrix according to the epipolar geometry constraints, and decomposes them to obtain the rotation matrix and translation vector of the camera relative to the projector; Distortion correction module: estimates the camera imaging distortion based on external and internal parameters, obtains the distortion correction coefficient, and performs distortion correction on the camera image based on the distortion correction coefficient; Parameter adjustment module: Counts the reprojection error generated during the correction process, and dynamically adjusts the parameters of the projector and camera according to the reprojection error until the reprojection error meets the preset accuracy requirements.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement a method for projection single-pose calibration as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute a method for projection single pose calibration according to any one of claims 1 to 5.

Images