Cuboid-based structured light system calibration method, device and system and storage medium

Abstract

The invention particularly relates to a structured light system calibration method, device and system based on a cuboid and a storage medium, and the calibration method comprises the steps: obtaining a target image which is an image obtained by photographing a single cuboid by a camera; acquiring a measurement matrix and camera internal parameters according to the target image; acquiring a camera external reference matrix according to the measurement matrix and the camera internal reference matrix; according to the target image and the camera projection matrix, obtaining three-dimensional point coordinates corresponding to the two-dimensional points on the image surface of the camera; according to a multi-frequency fringe projection technology, obtaining a projector plane coordinate of a two-dimensional point of a camera image plane; and obtaining internal and external parameters of the projector according to the three-dimensional point coordinates corresponding to the image surface of the camera and the plane coordinates of the projector. In the whole system calibration process, the position and the size of a single cuboid do not need to be known, the calibration of the structured light system can be realized only by fixedly placing the single cuboid at any position and projecting and shooting the single cuboid, the operation steps are simplified, and the calibration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional computer vision and three-dimensional mapping technology, and in particular to a calibration method, device, system and storage medium for a cuboid-based structured light system. Background Art

[0002] Structured light system calibration technology refers to the process of using a camera to capture the structured light image projected by a projector onto the surface of an object, and then solving the relevant parameters of the structured light system. It can provide support for subsequent reverse engineering, cultural relic reconstruction and industrial measurement.

[0003] Traditional structured light system calibration requires prefabricated calibration plates. Subsequent calibration requires the precise location and dimensions of each feature point on the plate. For each shot, the projector projects a specific pattern onto the plate, requiring the plate's position or orientation to be changed. This method of calibrating a structured light system using a calibration plate requires at least five repetitive steps, which is time-consuming, labor-intensive, and inflexible.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0005] The present invention provides a structured light system calibration method, device, system and storage medium based on a cuboid, which are used to solve the technical problems of the existing calibration method based on a calibration plate, such as complicated operation, time-consuming operation and low flexibility.

[0006] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0007] According to a first aspect of the present invention, there is provided a calibration method for a structured light system based on a cuboid, the method comprising camera calibration and projector calibration;

[0008] The camera calibration includes:

[0009] Acquire a target image, where the target image is an image captured by a camera of a single cuboid;

[0010] According to the target image, obtain the coordinates of the six corner points of the cuboid in the camera pixel coordinate system;

[0011] Obtain the measurement matrix by obtaining the coordinates of the six corner points of the cuboid in the camera pixel coordinate system;

[0012] According to the measurement matrix, obtain the camera measurement matrix sub-matrix and intrinsic parameter matrix;

[0013] Obtain the camera rotation matrix based on the obtained camera intrinsic parameter matrix and measurement matrix;

[0014] Obtain the transformation matrix based on the obtained camera measurement matrix sub-matrix, camera intrinsic parameter matrix and camera rotation matrix;

[0015] Obtain the camera projection parameter matrix according to the obtained measurement matrix and transformation matrix;

[0016] Obtain the camera translation matrix based on the obtained camera projection parameter matrix, camera intrinsic parameter matrix and camera rotation matrix;

[0017] The projector calibration includes:

[0018] According to the target image and the camera projection parameter matrix, the three-dimensional point coordinates corresponding to the two-dimensional point on the camera image plane are obtained;

[0019] According to the multi-frequency fringe projection technology, the projector image plane coordinates of the two-dimensional points on the camera image plane are obtained;

[0020] According to the three-dimensional point coordinates corresponding to the two-dimensional point on the camera image plane and the projector image plane coordinates, the projector intrinsic parameter matrix and extrinsic parameter matrix are obtained.

[0021] In some exemplary embodiments, the six corner points of the cuboid include:

[0022] The origin of the world coordinate system, the other corner point of the side of the cuboid where the X axis is located, the other corner point of the side of the cuboid where the Y axis is located, the other corner point of the side of the cuboid where the Z axis is located, the corner point of the cuboid between the X axis and the Y axis, and the corner point of the cuboid between the Y axis and the Z axis;

[0023] The origin of the world coordinate system is the first selected corner point of a single cuboid, the X-axis of the world coordinate system points from the first selected corner point of the single cuboid to the second selected corner point of the single cuboid, the Y-axis of the world coordinate system points from the first selected corner point of the single cuboid to the third selected corner point of the single cuboid, and the Z-axis of the world coordinate system points from the first selected corner point of the single cuboid to the fourth selected corner point of the single cuboid.

[0024] In some exemplary embodiments, the acquisition of the measurement matrix based on the coordinates of the six corner points of the cuboid in the camera pixel coordinate system is specifically determined according to the following formula:

[0025]

[0026] Among them, x i ,y i The coordinates of the eight vertices of the cuboid in the camera pixel coordinate system, and the measurement matrix W represents M in the above formula cD, λ1λ2…λ8 are the scaling factors of the homogeneous coordinates of the i-th vertex.

[0027] Substituting the six corner points into the eight vertices in the above formula can determine the measurement matrix W.

[0028] In some exemplary embodiments, obtaining a camera measurement matrix sub-matrix and an intrinsic parameter matrix according to the measurement matrix includes:

[0029] According to the measurement matrix, select its upper left 3×3 matrix as the measurement matrix sub-matrix;

[0030] Based on the measurement matrix submatrix and its relationship with the camera intrinsic parameter matrix and the cuboid parameter matrix, three independent equations are constructed to determine the camera intrinsic parameters. Among them, the three columns of the measurement matrix submatrix are the three vanishing points consistent with the X, Y, and Z directions in the world coordinate system.

[0031] In some exemplary embodiments, obtaining the three-dimensional point coordinates corresponding to the two-dimensional point on the camera image plane according to the target image and the camera projection parameter matrix includes:

[0032] According to the target image and the camera projection parameter matrix, the homography matrix from the three-dimensional points in each plane to the two-dimensional points in the target image is obtained;

[0033] Obtaining the homography matrix from the two-dimensional points in each plane to the three-dimensional points in the target image according to the homography matrix from the three-dimensional points in each plane to the two-dimensional points in the target image;

[0034] According to the target image and the homography matrix from the two-dimensional points of each plane to the three-dimensional points, the three-dimensional point coordinates corresponding to the two-dimensional points on each camera plane are obtained.

[0035] In some exemplary embodiments, obtaining a homography matrix from three-dimensional points in each plane to two-dimensional points in the target image based on the target image and the camera projection parameter matrix includes:

[0036] According to the target image, obtain the two-dimensional point coordinates of the three planes XOY, XOZ, and YOZ;

[0037] According to the two-dimensional points on the camera image plane and the camera projection parameter matrix, the camera imaging model is deformed to obtain the deformed camera projection model, which is the homography matrix of the plane.

[0038] In some exemplary embodiments, obtaining the projector image plane coordinates of corresponding image points based on the multi-frequency fringe projection technology includes:

[0039] According to the two-dimensional point in the camera image plane coordinate system and the phase of the phase-shift fringe pattern, the coordinates of the two-dimensional points on the three planes XOY, XOZ, and YOZ in the projector image plane coordinate system are obtained.

[0040] In some exemplary embodiments, obtaining the projector intrinsic parameter matrix and extrinsic parameter matrix according to the three-dimensional point coordinates corresponding to the camera image plane and the projector image plane coordinates includes:

[0041] Obtaining the projection parameter matrix of the projector according to the three-dimensional point coordinates of the corresponding image point and the image plane coordinates of the projector;

[0042] According to the projection parameter matrix of the projector, the intrinsic parameter matrix and rotation matrix of the projector are obtained;

[0043] Obtain the translation matrix of the projector according to the projection parameter matrix of the projector, the intrinsic parameter matrix of the projector, and the rotation matrix.

[0044] In a second aspect, a computer device is provided, comprising a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute a computer program or instruction stored in the memory, so that the electronic device implements a rectangular parallelepiped-based structured light system calibration method as described in any one of the first aspects.

[0045] In a third aspect, a structured light system is provided, comprising a projector, a camera, and the computer device according to the second aspect.

[0046] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is run on a computer device, the computer device executes the rectangular parallelepiped-based structured light system calibration method as described in any one of the first aspects.

[0047] In a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to execute the rectangular parallelepiped-based structured light system calibration method as described in any one of the first aspects.

[0048] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0049] The rectangular parallelepiped-based structured light system calibration method provided by the embodiments of the present invention does not require the position and size of a single rectangular parallelepiped to be known during the entire system calibration process. Calibration of the structured light system can be achieved by simply placing the single rectangular parallelepiped in an arbitrary position and projecting and photographing the single rectangular parallelepiped. In contrast, in traditional calibration methods based on calibration plates, it is necessary to change the position or posture of the calibration plate and perform projection and photography. This operation is cumbersome and time-consuming, and has low accuracy and robustness. Therefore, compared with traditional calibration plate-based structured light system calibration methods, the method provided by this application can simplify the operating steps and improve calibration efficiency.

[0050] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0052] Figure 1 is a schematic diagram of a structured light system in the prior art;

[0053] Figure 2 A schematic diagram of another structured light system provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a multi-frequency phase-shift fringe pattern provided in an embodiment of the present application;

[0055] Figure 4 A flowchart of the overall solution provided for the embodiment of this application;

[0056] Figure 5 A schematic flow chart of a structured light system calibration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0058] In addition, the accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0059] The traditional structured light system calibration is usually achieved by projecting onto a calibration plate, but this method is cumbersome and time-consuming. In view of this problem, an embodiment of the present application provides a structured light system calibration method: in this method, a single cuboid is used instead of the traditional calibration plate, the projector projects onto the single cuboid, and the camera transmits the captured single cuboid and the projected image to the computer device, so that the computer device can obtain the internal and external parameters of the camera and projector using the image captured by the camera, thereby achieving the calibration of the structured light system. Since the entire calibration process only requires placing a single cuboid in one position and executing a "projection-shooting" process once, there is no need to project and shoot onto multiple cuboids or multiple times a single cuboid at different positions. Therefore, the calibration steps of the structured light system are greatly simplified. In addition, since there is no specific restriction on the position of the single cuboid, and it is not necessary to know the spatial position of the single cuboid in advance, it is only necessary to capture the projection image of the single cuboid, so the calibration flexibility of the present application is better.

[0060] The following is an illustrative description of the rectangular parallelepiped-based structured light system calibration method, device, system, and storage medium solution of the present application with reference to the accompanying drawings.

[0061] Figure 2 This is a schematic diagram of a structured light system provided in an embodiment of the present application. Figure 2 As shown, the system may include a single rectangular parallelepiped, a projector, a camera, and a computer. The optical axes of the projector and camera are arranged at a predetermined angle. The camera and computer are connected via a wired or wireless connection. The optical centers of the projector and camera's optical axes are not constrained and can be at the same or different heights.

[0062] The size of a single cuboid does not need to be precisely known. For example, the surface of a single cuboid can be a single color (such as white) or a mixed color. The surface of the cuboid does not need to be mapped or printed with any feature points. During the projection and shooting process, the cuboid can be projected and shot in a natural light scene. It should be noted that if Figure 2 As shown, a single cuboid needs to be placed within the common field of view of the camera and the projector, but the position of the single cuboid does not have to be known.

[0063] The projector is used to project a specific pattern onto a single cuboid. The specific pattern can be Figure 3 The multi-frequency phase-shifted fringe pattern, the single-frequency phase-shifted fringe pattern + Gray code fringe pattern, or the single-frequency single-frame fringe pattern shown may also use other patterns. It should be noted that the following embodiments are illustrated using an example of a projector projecting a multi-frequency phase-shifted fringe pattern onto a single rectangular parallelepiped, and do not limit the embodiments of this application.

[0064] The camera is used to capture an image of a single cuboid. For example, when a projector projects a multi-frequency phase-shifted fringe pattern onto a single cuboid, the camera captures the image of the single cuboid. For another example, when the projector is not projecting onto the single cuboid, the camera captures an original image of the single cuboid in a natural light scene.

[0065] The computer device is used to obtain the internal and external parameters of the camera and projector based on the images taken by the camera, thereby realizing the calibration of the structured light system.

[0066] Figure 4 This is a flow chart of the overall solution provided by the embodiment of this application. Figure 4 As shown in the figure, a single cuboid is placed manually or automatically within the common field of view of the camera and projector and fixed in position. The projector then sequentially projects a multi-frequency phase-shifted fringe pattern onto the surface of the cuboid, and the camera sequentially captures the captured multi-frequency phase-shifted fringe pattern. Meanwhile, while the position of the cuboid remains unchanged, the camera captures the cuboid without projecting onto it, obtaining its original image. Corner points are then selected from the image to obtain the camera parameters. The captured multi-frequency phase-shifted fringe pattern of the cuboid is processed using a fringe analysis algorithm to obtain the phase of the fringe pattern, and then the projector parameters are obtained, achieving structured light system calibration.

[0067] During the entire system calibration process, the position of a single cuboid does not need to be known. It only needs to be fixed in an arbitrary position by the single cuboid and projected and photographed. In contrast, in traditional calibration plate-based methods, the calibration plate needs to be placed in multiple positions and postures for photography and projection. Therefore, compared with traditional calibration plate-based methods, the method provided in this application can significantly simplify the calibration process and improve calibration efficiency.

[0068] Figure 5 This is a flow chart of a rectangular parallelepiped-based structured light system calibration method provided in an embodiment of the present application. The execution subject of this method can be a computer device or a functional module in the computer device, which is not limited in this embodiment of the present application.

[0069] The method includes camera and projector calibration, wherein camera calibration includes obtaining the camera's intrinsic and extrinsic parameters, and projector calibration includes obtaining the projector's intrinsic and extrinsic parameters:

[0070] (1) Camera calibration

[0071] 1) Acquire the image

[0072] A target image is acquired, where the target image is an image captured by a camera of a single cuboid projected by a projector and an unprojected original image.

[0073] 2) Get corner points

[0074] At least six corner points of a cuboid are selected on the target image manually or automatically.

[0075] 3) Calculate the measurement matrix

[0076] Obtain the measurement matrix based on the selected corner points of the cuboid.

[0077] 4) Camera parameter calibration

[0078] ① Internal reference calculation

[0079] According to the obtained measurement matrix and the characteristics of the transformation matrix of the cuboid, the camera intrinsic parameters are obtained.

[0080] ② External parameter calculation

[0081] According to the obtained camera internal participation measurement matrix, the rotation matrix is ​​obtained; according to the measurement matrix, its upper left 3×3 sub-matrix is ​​selected to obtain the measurement matrix sub-matrix; according to the measurement matrix sub-matrix, the camera intrinsic parameter matrix and the camera rotation matrix, the transformation matrix sub-matrix is ​​obtained; according to the obtained measurement matrix sub-matrix and transformation matrix sub-matrix, the measurement matrix and the transformation matrix are obtained, wherein the transformation matrix is ​​used to decompose the cuboid vertex coordinate matrix, and the transformation matrix sub-matrix is ​​the matrix obtained by taking the upper left 3×3 sub-matrix of the transformation matrix; according to the measurement matrix and the transformation matrix, the camera projection parameter matrix is ​​obtained; according to the obtained camera projection parameter matrix, the intrinsic parameter matrix and the rotation matrix, the camera translation matrix is ​​obtained, and the camera calibration in the structured light system is completed.

[0082] (2) Projector calibration

[0083] 1) Obtaining the homography matrix

[0084] According to the target image and the camera projection parameter matrix, the homography matrix from the three-dimensional points to the two-dimensional points on each plane of the cuboid in the target image is obtained.

[0085] 2) Construct the correspondence between the projector plane coordinates and the three-dimensional coordinates

[0086] According to the target image and the homography matrix from the two-dimensional points to the three-dimensional points on each plane, the three-dimensional points corresponding to the two-dimensional points on each camera plane are obtained; based on the multi-frequency fringe projection technology, the projector image plane coordinates corresponding to the two-dimensional points on the camera plane are obtained; based on the three-dimensional points corresponding to the two-dimensional points on the camera plane and the projector image plane coordinates, the correspondence between the projector image plane coordinates and the three-dimensional coordinates is constructed.

[0087] 3) Calculate the projection parameter matrix of the projector

[0088] According to the correspondence between the projector image plane coordinates and the three-dimensional coordinates, the projector projection parameter matrix is ​​obtained.

[0089] 4) Calculate the internal and external parameters of the projector

[0090] According to the projector projection parameter matrix, the intrinsic parameter matrix and rotation matrix of the projector are obtained; according to the projector projection parameter matrix, intrinsic parameter matrix and rotation matrix, the translation matrix of the projector is obtained, thus completing the projector calibration in the structured light system.

[0091] In the above scheme, during the entire system calibration process, the position and size of a single cuboid do not need to be known. The structured light system can be calibrated by simply placing the cuboid in a fixed position and projecting and photographing it. Traditional calibration methods based on calibration plates require changing the position or posture of the plate and performing projection and photographing. This operation is cumbersome and time-consuming, and has low accuracy and robustness. Therefore, compared to traditional calibration plate-based structured light system calibration methods, the method provided in this application can simplify the operation steps and improve calibration efficiency.

[0092] like Figure 5 As shown, the method may include the following steps S1 to S6.

[0093] S1: The computer device obtains a target image, which is an image obtained by a camera shooting a single cuboid.

[0094] In one approach, the target image is a phase-shifted fringe pattern captured by a camera while the projector is projecting onto a single rectangular block. This captured phase-shifted fringe pattern is then used to obtain the absolute phases of the horizontal and vertical fringes and establish a mapping relationship between the projector image plane coordinates and the camera image plane coordinates. Once the camera image plane coordinates are known, the absolute phases of the horizontal and vertical fringes can be used to directly obtain the corresponding projector image plane coordinates, supporting the subsequent acquisition of the projector's internal and external parameters.

[0095] In another approach, the target image is the original image captured by the camera when the projector is not projecting onto a single cuboid, and is used to colorize the subsequent reconstructed point cloud.

[0096] S2: The computer device obtains the measurement matrix and camera intrinsic parameters based on the target image.

[0097] In some embodiments, the intrinsic parameters of the camera include: the focal length of the camera lens and the coordinates of the camera's principal point, where the principal point coordinates are the coordinates of the intersection of the camera's lens optical axis and the camera's image plane in the camera's image plane coordinate system. The origin of the camera's image plane coordinate system is the principal point of the camera's image plane, the X-axis of the camera's image plane coordinate system is the horizontal direction of the camera's image plane, and the Y-axis of the camera's image plane coordinate system is the vertical direction of the camera's image plane.

[0098] (1) The computer device obtains the coordinates of the vertices of a single cuboid in the camera pixel coordinate system based on the target image.

[0099] The computer extracts the camera pixel coordinates of the corner points of the target image to obtain the pixel coordinates of the six corner points of the cuboid in the target image. The six corner points are, in order, the origin in world coordinates, the corner point at the other end of the cuboid side where the X axis is located, the corner point at the other end of the cuboid side where the Y axis is located, the corner point at the other end of the cuboid side where the Z axis is located, the corner point of the cuboid between the X and Y axes, and the corner point of the cuboid between the Y and Z axes.

[0100] (2) The computer device obtains the measurement matrix based on the coordinates of the vertices of the single cuboid in the camera pixel coordinate system.

[0101] The eight vertex coordinates of the cuboid can be represented by the matrix V

[0102]

[0103] Among them, l1, l2, and l3 represent the lengths of the three sides of the cuboid respectively.

[0104] The cuboid vertex coordinate matrix V can be decomposed with the help of the transformation matrix D into

[0105]

[0106] The vertices of the cuboid projected onto the camera plane can be expressed as

[0107]

[0108] Among them, x i ,y i (i=1, 2, ..., 8) are the coordinates of the eight vertices of the cuboid in the camera pixel coordinate system, and the measurement matrix W represents M in equation (3) c D. λ1λ2…λ8 is the scaling factor of the homogeneous coordinates of the i-th vertex, corresponding to the 8 vertices of the cuboid, and its function is to map the three-dimensional homogeneous coordinates to the two-dimensional image plane.

[0109] In one implementation, obtaining a measurement matrix based on the corner points of a cuboid selected on a target image includes: obtaining image coordinates of six corner points of the cuboid based on the target image; and obtaining a cuboid measurement matrix based on the image coordinates of the six corner points of the cuboid.

[0110] The corner points of the cuboid are extracted from the target image to obtain the 6 corner points of the cuboid, which can support the solution of the 11 constraints of the measurement matrix.

[0111] Specifically, the coordinates obtained in S2(1) are used to solve the 11 unknowns in the measurement matrix W to obtain the measurement matrix W.

[0112] (3) The computer device obtains the camera intrinsic parameters based on the measurement matrix.

[0113] Based on the measurement matrix, the upper-left 3×3 matrix is ​​selected as the measurement matrix submatrix. Based on the relationship between the measurement matrix submatrix, the camera intrinsic parameter matrix, and the cuboid parameter matrix, three independent equations are constructed to determine the camera intrinsic parameters. The three columns of the measurement matrix submatrix are the three vanishing points that coincide with the X, Y, and Z directions in the world coordinate system.

[0114] In one implementation, the camera intrinsic parameters are obtained based on the acquired measurement matrix and the characteristics of the cuboid's transformation matrix. This includes: selecting the upper-left 3×3 matrix of the measurement matrix as a measurement matrix submatrix; and constructing three independent equations based on the relationship between the measurement matrix submatrix, the camera intrinsic parameter matrix, and the cuboid parameter matrix to determine the camera intrinsic parameters. The three columns of the measurement matrix submatrix are the three vanishing points that correspond to the X, Y, and Z directions in the world coordinate system.

[0115] In one implementation, the internal parameters of the camera include three parameters: the focal length of the camera lens and the principal point coordinates of the camera. The principal point coordinates are the coordinates of the intersection of the camera's optical axis and the imaging plane in the image plane coordinate system.

[0116] Correspondingly, according to the measurement matrix, the upper left 3×3 matrix thereof is selected as the measurement matrix submatrix, including: selecting the upper left 3×3 matrix of the 3×4 measurement matrix to construct the measurement matrix submatrix.

[0117] In one implementation, three independent equations are obtained based on the measurement matrix submatrix and its relationship with the camera intrinsic parameter matrix and the cuboid parameter transformation matrix to determine the camera intrinsic parameters, including:

[0118] In one implementation, the cuboid parameter transformation matrix is ​​a matrix that normalizes the cuboid parameter matrix. By defining the unit length of the unit cuboid as 1, the normalized coordinates of the cuboid corners are obtained, and a coordinate matrix of the cuboid corners is generated. The cuboid parameter matrix is ​​decomposed to obtain the transformation matrix and the corner normalization matrix. The submatrices of the corner normalization matrix have arbitrary scale factors.

[0119] In one implementation, the product of the transpose of the transformation matrix submatrix and the transformation matrix submatrix is ​​a diagonal matrix, and the value of the off-diagonal matrix is ​​zero.

[0120] In one implementation, submatrices of the measurement matrix are first extracted, corresponding to the first, second, and third columns. The transposed matrix of the measurement matrix submatrix is ​​then calculated. Next, the inverse matrix of the camera intrinsic parameter matrix and the inverse matrix of its transpose are calculated. The off-diagonal values ​​of the transformation matrix are then obtained by multiplying the transposed matrix of the measurement matrix submatrix, the inverse matrix of the transposed camera intrinsic parameters, the inverse matrix of the camera intrinsic parameters, and another measurement matrix submatrix. This process is repeated three times, corresponding to different submatrix combinations. Next, three equations related to the camera intrinsic parameters and the measurement matrix submatrix are established based on the assumption that the off-diagonal region of the product of the transpose of the transformation matrix submatrix and itself is zero.

[0121] Finally, by solving these three equations, the specific values ​​of the camera intrinsic parameters are determined.

[0122] Specifically, the camera projection parameter matrix M c Further decomposition is performed, and the measurement matrix W can be expressed as

[0123] W=M c D = K[RT]D. (4) Take the upper left 3 × 3 matrix of W, and equation (4) can be transformed into

[0124] w=KRd.(5)

[0125] Where w and d are the upper left 3×3 matrices of W and D respectively.

[0126] Transform Equation (5)

[0127]

[0128] Among them, d T d is a diagonal matrix with all off-diagonal elements being zero.

[0129] Due to d T d is a symmetric matrix. According to formula (6), we can get three equations:

[0130]

[0131] Among them, w i (i=1, 2, 3) are the vanishing points corresponding to the X, Y, and Z directions in the world coordinate system, and are defined as

[0132] The orthogonality constraint between mutually orthogonal vanishing points is implemented.

[0133] Since the camera intrinsic parameters include three unknown quantities: focal length and principal point coordinates, the camera intrinsic parameters can be obtained by solving (7) using the least squares method.

[0134] S3: The computer device obtains the camera extrinsic parameter matrix according to the measurement matrix and the camera intrinsic parameter matrix.

[0135] In some embodiments, the external parameters of the camera include: a rigid transformation of the camera coordinate system relative to the world coordinate system, that is, a rotation matrix and a translation vector between the two coordinate systems.

[0136] (1) The computer device obtains the camera rotation matrix based on the measurement matrix and the camera intrinsic parameter matrix.

[0137] In one implementation, obtaining a camera rotation matrix according to the obtained measurement matrix and the camera intrinsic parameter matrix includes: multiplying the inverse matrix of the camera intrinsic parameter matrix by the measurement matrix submatrix according to the camera intrinsic parameter matrix and the measurement matrix submatrix, and then dividing by the modulus of the product to obtain a rotation matrix.

[0138] Specifically, the camera rotation matrix R c It can be calculated by the following formula

[0139]

[0140] where r i (i=1,2,3) is the camera rotation matrix R c Each column of

[0141] The camera rotation matrix can be obtained by using the matrix.

[0142] (2) The computer device obtains the camera transformation matrix D based on the measurement matrix, the camera intrinsic parameter matrix and the camera rotation matrix.

[0143] In one implementation, a transformation matrix is ​​obtained based on the obtained measurement matrix, camera intrinsic parameter matrix, and camera rotation matrix, including: multiplying the inverse matrix of the camera rotation matrix, the inverse matrix of the camera intrinsic parameter matrix, and a submatrix of the measurement matrix in sequence to obtain a transformation matrix submatrix; and adding a row and a column to the transformation matrix submatrix to obtain a transformation matrix.

[0144] Specifically, transform equation (5) into

[0145] d=R -1 K -1 w.(9) Since d is the upper left 3×3 matrix of the camera transformation matrix D, D is

[0146]

[0147] Obtain the transformation matrix D based on the known measurement matrix, camera intrinsic parameter matrix and camera rotation matrix.

[0148] (3) The computer device obtains the camera projection parameter matrix based on the measurement matrix and the transformation matrix.

[0149] In one implementation, obtaining a projection parameter matrix of the camera according to the obtained measurement matrix and transformation matrix includes: multiplying the measurement matrix by an inverse matrix of the transformation matrix to obtain the projection parameter matrix of the camera.

[0150] Specifically, transform equation (4) into

[0151] M c =WD -1 .(11)

[0152] Get the camera projection parameter matrix based on the known measurement matrix and transformation matrix.

[0153] (4) The computer device obtains the camera translation vector based on the camera projection parameter matrix, intrinsic parameter matrix and rotation matrix.

[0154] In one implementation, obtaining a camera translation matrix based on the obtained camera projection parameter matrix, intrinsic parameter matrix, and rotation matrix includes: sequentially multiplying the inverse matrix of the camera rotation matrix, the inverse matrix of the camera intrinsic parameter matrix, and the camera projection parameter matrix to obtain the camera translation matrix. The camera translation matrix has an arbitrary scale factor.

[0155] Specifically, according to formula (4), the camera translation vector T C for

[0156] T C =R C -1 K -1 M C . (12)

[0157] S4: The computer device obtains the three-dimensional point coordinates corresponding to the two-dimensional point on the camera image plane according to the target image and the camera projection parameter matrix.

[0158] In one implementation, based on the camera projection parameter matrix, the homography matrix from the three-dimensional points of each plane to the two-dimensional points in the target image is obtained, including: defining a certain coordinate value of the plane to be 0, setting the corresponding coordinate of the corresponding three-dimensional point to 0, deforming the camera imaging model, and obtaining the deformed camera projection model, which is the homography matrix of the plane.

[0159] In one implementation, the deformation is as follows: for example, for the XOY plane, the Z-axis coordinate is 0, and the Z-axis coordinate can be directly set to 0 in the camera imaging model to simplify the camera imaging model. The same applies to the XOZ and YOZ planes.

[0160] In one implementation, based on the homography matrix from the three-dimensional points of each plane to the two-dimensional points in the target image, the homography matrix from the two-dimensional points of each plane to the three-dimensional points is obtained, including: based on the invertibility of the homography matrix, multiplying the inverse matrix of the homography matrix from the three-dimensional points of each plane to the two-dimensional points with the two-dimensional points to obtain the homography matrix from the two-dimensional points of each plane to the three-dimensional points.

[0161] In one implementation, the 3D points corresponding to the 2D points on each camera plane in the target image are obtained based on the target image and the homography matrices from the 2D points on each plane to the 3D points. This includes selecting a 2D point on a plane in the target image and multiplying the homogeneous coordinates of the 2D point on the plane by the homography matrix from the 2D point to the 3D point on the plane to obtain the 3D homogeneous coordinates corresponding to the selected 2D point on the plane. Similar operations are performed on all three planes to obtain the 3D homogeneous coordinates corresponding to the 2D points on the three planes.

[0162] Specifically include:

[0163] (1) The computer device obtains the two-dimensional coordinates of the camera image plane based on the target image.

[0164] According to the target image, obtain the two-dimensional point coordinates of the three planes XOY, XOZ, and YOZ.

[0165] (2) The computer device obtains the three-dimensional point coordinates corresponding to the two-dimensional point on the camera image plane based on the two-dimensional point on the camera image plane and the camera projection parameter matrix.

[0166] Taking the XOY plane as an example, the Z axis of the three-dimensional point corresponding to the plane in the target image is 0, so the camera imaging model can be expressed as

[0167]

[0168] Among them, x cij , x cij (i, j = 1, 2, 3, ..., N) is the camera image plane coordinate, m i , i=1, 2, 3, 4 are the camera projection parameter matrices M c Column vector of .

[0169] Matrix H XOY It is the homography matrix introduced by the XOY plane, which is the transformation matrix between the target vector and the coordinates in the camera image plane coordinate system. The target vector is composed of the X-axis and Y-axis coordinates of all intersection points in the world coordinate system. XOY It is reversible, so the correspondence between the two-dimensional point and the three-dimensional point on the XOY plane can be expressed. Therefore, the corresponding three-dimensional point on the XOY plane can be expressed as

[0170]

[0171] Similarly, the three-dimensional points on the XOZ plane and the YOZ plane can be expressed as

[0172]

[0173] Among them, x pij , x pij (i, j = 1, 2, 3, ..., N) are the coordinates of the projector image plane. According to the above formula, the three-dimensional point sets corresponding to the two-dimensional point coordinates on the three planes XOY, XOZ, and YOZ can be obtained respectively.

[0174] S5: The computer device obtains the projector image plane coordinates of the two-dimensional points on the camera image plane based on the multi-frequency fringe projection technology.

[0175] In one implementation, obtaining the projector image plane coordinates of a corresponding image point using multi-frequency fringe projection technology includes calculating the wrapped phase of a rectangular multi-frequency phase-shifted fringe pattern and applying phase unwrapping techniques to the wrapped phase to obtain the absolute phase of the rectangular fringe. Based on the point on the camera imaging plane and the phase of the phase-shifted fringe pattern, the corresponding two-dimensional coordinates of the point on the projector imaging plane are obtained.

[0176] Specifically, according to the phase of the two-dimensional point and the phase shift fringe pattern in the camera image plane coordinate system, the coordinates of the two-dimensional points on the three planes XOY, XOZ, and YOZ in the projector image plane coordinate system are obtained. According to the phase shift fringe pattern obtained in S1, the absolute phases of the horizontal and vertical phase shift fringe patterns can be obtained respectively. Combined with the coordinates in the camera image plane coordinate system obtained in S4, the corresponding projector image plane coordinates can be obtained by the following formula

[0177]

[0178] Among them, x pij , x pij (i, j = 1, 2, 3, ..., N) is the projector image plane coordinate, are the absolute phases of the vertical and horizontal phase-shifted fringe patterns of the pixel, respectively, and f V and f H are the frequencies of the projected horizontal and vertical stripes, respectively.

[0179] S6: The computer device obtains the internal and external parameters of the projector according to the three-dimensional point coordinates corresponding to the camera image plane and the projector image plane coordinates.

[0180] In one implementation, obtaining a projector projection parameter matrix based on the three-dimensional point coordinates of corresponding image points and the projector image plane coordinates includes: constructing an equation using the obtained three-dimensional point coordinates and their corresponding projector image plane coordinates, and obtaining the projector projection parameter matrix through direct linear transformation (DLT). The projector projection parameter matrix includes internal and external parameters of the projector, the X-axis of the projector image plane coordinate system is the horizontal direction of the projector image plane, and the Y-axis of the projector image plane coordinate system is the vertical direction of the projector image plane.

[0181] In one implementation, the intrinsic parameter matrix and rotation matrix of the projector are obtained according to the projector projection parameter matrix, including: taking the upper left 3×3 matrix of the projector projection parameter matrix, performing QR decomposition on it, and obtaining the projector intrinsic parameter matrix and rotation matrix.

[0182] In one implementation, the translation matrix of the projector is obtained based on the projection parameter matrix, the intrinsic parameter matrix and the rotation matrix of the projector, including: taking the 4th column of the projector projection parameter matrix, taking the upper left 3×3 matrix of the 4th column in combination with the projector projection parameter matrix, calculating the coordinate matrix of the camera center in the world coordinate system, multiplying the rotation matrix by the coordinate matrix of the camera center in the world coordinate system, and then multiplying by -1 to obtain the translation matrix.

[0183] In one implementation, the color information of the target is obtained based on the unprojected original image of the target; and the colored three-dimensional point cloud of the target is obtained based on the color information of the target and the three-dimensional point cloud of the target.

[0184] Specifically, the following equations can be established through S4 and S5, and M can be directly solved by using least squares algorithms P .

[0185]

[0186] Among them, M P is the parameter matrix of the projector.

[0187] To M P Take the upper left 3×3 submatrix and perform QR decomposition to obtain the projector intrinsic parameter matrix and rotation matrix. P The fourth column of is used to obtain the coordinates of the origin of the camera coordinate system in the world coordinate system, and then the translation matrix of the projector is obtained.

[0188] By adopting the structured light system calibration method provided in the above embodiment, it is only necessary to place a single cuboid at a single position for projection shooting to achieve the acquisition of camera and projector parameters, avoiding the tedious process of repeated projection and shooting at multiple different positions required by conventional calibration methods, which can greatly simplify the calibration process. At the same time, there is no need to know the placement position of a single cuboid, so it has better flexibility.

[0189] An embodiment of the present application further provides a computer device, including a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute a computer program or instruction stored in the memory, so that the computer device implements the methods in the above embodiments.

[0190] The present application also provides a computer-readable storage medium having computer instructions stored therein; when the computer-readable storage medium is executed on a computer device, the computer device executes the method described above. The computer instructions may be stored in the computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0191] An embodiment of the present application further provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0192] The present application also provides a chip coupled to a memory, configured to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments. The chip may be a general-purpose processor or a dedicated processor. In some embodiments, the chip includes a processor, a transceiver, and a storage medium. The processor is configured to support a computer device in executing the technical solution described above. The transceiver is configured to accept control of the processor and support the computer device in executing the technical solution described above.

[0193] It should be noted that the chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits that can perform the various functions described throughout this application.

[0194] The computer device, computer-readable storage medium, computer program product, and chip provided in the above-mentioned embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0195] It should be understood that the above is only to help those skilled in the art better understand the embodiments of the present application, and is not intended to limit the scope of the embodiments of the present application. Based on the above examples given, those skilled in the art can obviously make various equivalent modifications or changes. For example, certain steps in each embodiment of the above detection method may be unnecessary, or certain new steps may be added. Or a combination of any two or any multiple embodiments described above. Such modifications, changes, or combined solutions also fall within the scope of the embodiments of the present application.

[0196] It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced with each other. For the sake of brevity, they will not be repeated here.

[0197] It should also be understood that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0198] It should also be understood that in the embodiments of the present application, "pre-setting" and "pre-definition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including an electronic device), and the present application does not limit its specific implementation method.

[0199] It should also be understood that the division of the modes, situations, categories and embodiments in the embodiments of the present application is only for the convenience of description and should not constitute a special limitation. The features of various modes, categories, situations and embodiments can be combined without contradiction.

[0200] It should also be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0201] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A structured light system calibration method based on a cuboid, characterized in that: The method includes camera calibration and projector calibration; The camera calibration includes: Acquire a target image, where the target image is an image captured by a camera of a single cuboid; According to the target image, obtain the coordinates of the six corner points of the cuboid in the camera pixel coordinate system; Obtain the measurement matrix by obtaining the coordinates of the six corner points of the cuboid in the camera pixel coordinate system; According to the measurement matrix, obtain the camera measurement matrix sub-matrix and intrinsic parameter matrix; Obtain the camera rotation matrix based on the obtained camera intrinsic parameter matrix and measurement matrix; Obtain the transformation matrix based on the obtained camera measurement matrix sub-matrix, camera intrinsic parameter matrix and camera rotation matrix; Obtain the camera projection parameter matrix according to the obtained measurement matrix and transformation matrix; Obtain the camera translation matrix based on the obtained camera projection parameter matrix, camera intrinsic parameter matrix and camera rotation matrix; The projector calibration includes: According to the target image and the camera projection parameter matrix, the three-dimensional point coordinates corresponding to the two-dimensional point on the camera image plane are obtained; According to the multi-frequency fringe projection technology, the projector image plane coordinates of the two-dimensional points on the camera image plane are obtained; According to the three-dimensional point coordinates corresponding to the two-dimensional point on the camera image plane and the projector image plane coordinates, the projector intrinsic parameter matrix and extrinsic parameter matrix are obtained.

2. The method according to claim 1, characterized in that The six corner points of the cuboid include: The origin of the world coordinate system, the other corner point of the side of the cuboid where the X axis is located, the other corner point of the side of the cuboid where the Y axis is located, the other corner point of the side of the cuboid where the Z axis is located, the corner point of the cuboid between the X axis and the Y axis, and the corner point of the cuboid between the Y axis and the Z axis; The origin of the world coordinate system is the first selected corner point of a single cuboid, the X-axis of the world coordinate system points from the first selected corner point of the single cuboid to the second selected corner point of the single cuboid, the Y-axis of the world coordinate system points from the first selected corner point of the single cuboid to the third selected corner point of the single cuboid, and the Z-axis of the world coordinate system points from the first selected corner point of the single cuboid to the fourth selected corner point of the single cuboid.

3. The method according to claim 2, characterized in that The measurement matrix is ​​obtained by obtaining the coordinates of the six corner points of the cuboid in the camera pixel coordinate system, which is specifically determined according to the following formula: Among them, x i ,y i The coordinates of the eight vertices of the cuboid in the camera pixel coordinate system, and the measurement matrix W represents M in the above formula c D, λ1λ2…λ8 are the scaling factors of the homogeneous coordinates of the i-th vertex. Substituting the six corner points into the eight vertices in the above formula can determine the measurement matrix W.

4. The method according to claim 1, wherein The method of obtaining the camera measurement matrix sub-matrix and the intrinsic parameter matrix according to the measurement matrix includes: According to the measurement matrix, select its upper left 3×3 matrix as the measurement matrix sub-matrix; Based on the measurement matrix submatrix and its relationship with the camera intrinsic parameter matrix and the cuboid parameter matrix, three independent equations are constructed to determine the camera intrinsic parameters. Among them, the three columns of the measurement matrix submatrix are the three vanishing points consistent with the X, Y, and Z directions in the world coordinate system.

5. The method according to claim 1, wherein According to the target image and the camera projection parameter matrix, the 3D coordinates of the corresponding 2D points on the camera image plane are obtained, including: According to the target image and the camera projection parameter matrix, the homography matrix from the three-dimensional points in each plane to the two-dimensional points in the target image is obtained; Obtaining the homography matrix from the two-dimensional points in each plane to the three-dimensional points in the target image according to the homography matrix from the three-dimensional points in each plane to the two-dimensional points in the target image; According to the target image and the homography matrix from the two-dimensional points of each plane to the three-dimensional points, the three-dimensional point coordinates corresponding to the two-dimensional points on each camera plane are obtained.

6. The method according to claim 5, characterized in that The step of obtaining a homography matrix from three-dimensional points in each plane to two-dimensional points in the target image according to the target image and the camera projection parameter matrix includes: According to the target image, obtain the two-dimensional point coordinates of the three planes XOY, XOZ, and YOZ; According to the two-dimensional points on the camera image plane and the camera projection parameter matrix, the camera imaging model is deformed to obtain the deformed camera projection model, which is the homography matrix of the plane.

7. The method according to claim 1, characterized in that The method of obtaining the projector image plane coordinates of corresponding image points based on the multi-frequency fringe projection technology includes: According to the two-dimensional point in the camera image plane coordinate system and the phase of the phase-shift fringe pattern, the coordinates of the two-dimensional points on the three planes XOY, XOZ, and YOZ in the projector image plane coordinate system are obtained.

8. The method according to claim 1, characterized in that The step of obtaining the projector intrinsic parameter matrix and extrinsic parameter matrix according to the three-dimensional point coordinates corresponding to the camera image plane and the projector image plane coordinates includes: Obtaining the projection parameter matrix of the projector according to the three-dimensional point coordinates of the corresponding image point and the image plane coordinates of the projector; According to the projection parameter matrix of the projector, the intrinsic parameter matrix and rotation matrix of the projector are obtained; Obtain the translation matrix of the projector according to the projection parameter matrix of the projector, the intrinsic parameter matrix of the projector, and the rotation matrix.

9. A computer device, characterized in that: The device comprises a processor and a processor, the processor being coupled to the memory, the processor being used to execute a computer program or instruction stored in the memory, so that the computer device implements the rectangular parallelepiped-based structured light system calibration method as described in any one of claims 1 to 8.

10. A structured light system, characterized in that: The structured light system includes a projector, a camera, and the computer device according to claim 9, wherein the optical axes of the projector and the camera are arranged at a certain angle, and the camera is connected to the computer device via a wired or wireless connection.

Images