Structured Light System Calibration Method, Device, System and Storage Medium

Abstract

An embodiment of the present application provides a method, device, system, and storage medium for calibrating a structured light system, which relates to the fields of three-dimensional computer vision and three-dimensional measurement technologies. In the structured light system calibration solution provided by the present application, a single sphere is used to replace the traditional calibration board. The projector projects onto the single sphere, and the camera sends the captured image of the single sphere and the projected image to the computer device. Thus, the computer device can obtain the internal parameters and external parameters of the projector by using the image collected by the camera and the known internal parameters of the camera, thereby realizing the calibration of the structured light system.

Description

Technical Field

[0001] This application relates to the field of three-dimensional computer vision and three-dimensional measurement technologies, and particularly relates to a structured light system calibration method, device, system, and storage medium. Background Art

[0002] Structured light system calibration is the basis and prerequisite for three-dimensional reconstruction.

[0003] Generally, a structured light system includes a projector, a camera, a calibration board, and a computer device as shown in Figure 1 . The structured light system calibration process is as follows: Move the calibration board to a certain starting position, the projector projects a specific pattern onto the calibration board, and the camera sequentially takes pictures of the projected images on the calibration board. Then, manually change the position or pose of the calibration board, repeat the above projection and shooting processes. After repeating multiple times, the computer uses the multiple sets of images taken for structured light system calibration.

[0004] However, in the above method of calibrating a structured light system by projecting onto a calibration board, the camera and the projector need to repeat the projection, shooting, and data processing processes at least 5 times, resulting in cumbersome operation and time consumption. Summary of the Invention

[0005] This application provides a structured light system calibration method, device, system, and storage medium, which solves the technical problem of cumbersome operation and time consumption in the method of calibrating a structured light system by projecting onto a calibration board.

[0006] To achieve the above object, this application adopts the following technical solutions:

[0007] In a first aspect, an embodiment of this application provides a structured light system calibration method. The method includes:

[0008] Obtain a target image, where the target image is an image obtained by the camera taking a picture of a single sphere; obtain the external parameters of the camera according to the target image and the internal parameters of the camera; on each of the N intersection lines of the surface of the single sphere and the N planes of the N planes perpendicular to the Z-axis of the world coordinate system, where the Z-axis of the world coordinate system is determined according to the center of the single sphere and the center of the camera lens, and both N and M are integers greater than or equal to 3; obtain the coordinates of all the intersections in the camera image plane coordinate system according to all the obtained intersections, the internal parameters of the camera, and the external parameters of the camera; obtain the internal parameters and external parameters of the projector according to the coordinates of all the intersections in the camera image plane coordinate system and the phase of the phase-shifted fringe pattern, where the phase-shifted fringe pattern is an image obtained by the projector projecting onto the single sphere and the camera taking a picture.

[0009] In the above solution, through algorithms such as fringe analysis, edge extraction, and ellipse fitting, the multi-frequency phase-shifted fringe pattern obtained by shooting and the original image of a single small ball are processed respectively to obtain the phase of the fringe pattern, the coordinates of the ellipse center in the camera image plane coordinate system, and the length of the major semi-axis of the ellipse. Then, combined with the internal parameters of the camera, parameters such as the external parameters of the camera, the internal parameters and external parameters of the projector can be obtained to realize the calibration of the structured light system. During the entire system calibration process, the position of the single small ball does not need to be known. It only needs to fix the single small ball at an arbitrary position and project and shoot the single small ball. In the traditional calibration plate-based method, the calibration plate needs to be placed at multiple different positions and postures for shooting and projection. Therefore, compared with the traditional calibration plate-based method, the method provided in this application can significantly simplify the calibration process and improve the calibration efficiency.

[0010] In one implementation, according to the target image and the internal parameters of the camera, the external parameters of the camera are obtained, including: obtaining the coordinates of the center of a single sphere in the camera coordinate system according to the target image and the internal parameters of the camera; obtaining the parameter matrix of the camera according to the coordinates of the center of the single sphere in the camera coordinate system, and the parameter matrix includes the internal parameters and external parameters of the camera. Wherein, the origin of the camera coordinate system is the lens center of the camera, the Z-axis of the camera coordinate system is the optical axis of the camera lens, the X-axis of the camera coordinate system is the horizontal direction of the camera image plane, and the Y-axis of the camera coordinate system is the vertical direction of the camera image plane.

[0011] In one implementation, the internal parameters of the camera include the focal length of the camera lens and the coordinates of the principal point of the camera, and the coordinates of the principal point are the coordinates of the intersection of the optical axis of the camera lens and the camera image plane in the camera image plane coordinate system.

[0012] Correspondingly, according to the target image and the internal parameters of the camera, obtaining the coordinates of the center of a single sphere in the camera coordinate system includes: performing edge extraction of a single sphere on the target image to obtain the coordinates of each edge point in the camera image plane coordinate system, and performing ellipse fitting on the coordinates of each edge point to obtain an ellipse containing each edge point, and determining the coordinates of the center point of the ellipse in the camera image plane coordinate system and the length of the major semi-axis of the ellipse; determining the coordinates of the center point of the ellipse in the camera coordinate system according to the coordinates of the center point of the ellipse in the camera image plane coordinate system, the focal length of the camera lens, and the coordinates of the principal point of the camera; determining the target ratio according to the length of the major semi-axis of the ellipse, the coordinates of the center point of the ellipse in the camera coordinate system, and the radius of the single sphere; taking the product of the coordinates of the center point of the ellipse in the camera coordinate system and the target ratio as the coordinates of the center of the single sphere in the camera coordinate system.

[0013] In one implementation, the radius of the single sphere is the real radius of the single sphere or a preset virtual radius.

[0014] It should be understood that when the radius of a single small ball is the true radius, the coordinates of the estimated center of the ball in the camera coordinate system are the true physical coordinates. Based on these coordinates for subsequent processing, the obtained external parameters of the camera and the projector parameters are the true parameters. And based on the calibrated system for three-dimensional reconstruction of the scene, the obtained three-dimensional reconstruction result is consistent with the true size of the scene. When the radius of a single small ball is the preset virtual radius, the coordinates of the estimated center of the ball in the camera coordinate system differ from the true physical coordinates by a certain scaling ratio. The rotation matrix in the external parameters of the camera obtained based on these coordinates is consistent with the true rotation matrix of the camera, but the translation vector in the external parameters of the camera differs from the true value by a certain scaling ratio. At the same time, the obtained internal parameters and external parameters of the projector are also inconsistent with the true parameters. However, based on the calibration results of the camera and the projector obtained with the preset virtual radius, the scene can still be three-dimensionally reconstructed. Only the reconstruction result differs from the true size of the scene by a certain scaling ratio, but there is a structural similarity between the two, only differing by a certain scale change.

[0015] In one implementation, according to the coordinates of the center of a single sphere in the camera coordinate system, the parameter matrix of the camera is obtained. The parameter matrix includes the internal parameters and external parameters of the camera, including: normalizing the coordinates of the center of a single sphere in the camera coordinate system to obtain a unit vector, and determining a first angle and a second angle based on the unit vector; generating a first rotation matrix of the unit vector around the X-axis of the camera coordinate system according to the first angle; and generating a second rotation matrix of the unit vector around the Y-axis of the camera coordinate system according to the second angle; multiplying the first rotation matrix and the second rotation matrix to obtain the transposed matrix of the rotation matrix of the camera; transposing the transposed matrix of the rotation matrix of the camera to obtain the rotation matrix of the camera; generating the parameter matrix of the camera according to the internal parameters of the camera, the rotation matrix of the camera, and the coordinates of the center point of the ellipse in the camera image plane coordinate system.

[0016] In one implementation, according to all the obtained intersection points, the internal parameters of the camera, and the external parameters of the camera, the coordinates of all the intersection points in the camera image plane coordinate system are obtained, including: determining a homography matrix according to the Z coordinates of all the intersection points in the world coordinate system, and the internal parameters and external parameters of the camera. The homography matrix is a transformation matrix between the target vector and the coordinates in the camera image plane coordinate system, and the target vector is composed of the X-axis and Y-axis coordinates of all the intersection points in the world coordinate system; obtaining the coordinates of all the intersection points in the camera image plane coordinate system according to the product of the homography matrix and the target vector.

[0017] In one implementation, based on the coordinates of all intersection points in the camera image plane coordinate system and the phase of the phase-shifted fringe pattern, the internal parameters and external parameters of the projector are obtained, including: obtaining the coordinates of all intersection points in the projector image plane coordinate system based on the coordinates of all intersection points in the camera image plane coordinate system and the phase of the phase-shifted fringe pattern; obtaining the parameter matrix of the projector based on the coordinates of all intersection points in the world coordinate system and the coordinates of all intersection points in the projector image plane coordinate system, where the parameter matrix of the projector includes the internal parameters and external parameters of the projector. Among them, the origin of the projector image plane coordinate system is the principal point of the projector image plane, the X-axis of the projector image plane coordinate system is the horizontal direction of the projector image plane, and the Y-axis of the projector image plane coordinate system is the vertical direction of the projector image plane.

[0018] In one implementation, the target image is a phase-shifted fringe pattern obtained by the camera when the projector projects onto a single sphere. The captured phase-shifted fringe pattern is used to subsequently obtain the absolute phases of the horizontal and vertical fringes and establish the mapping relationship between the projector image plane coordinates and the camera image plane coordinates. After the camera image plane coordinates are known, the corresponding projector image plane coordinates can be directly obtained through the absolute phases of the above-mentioned horizontal and vertical fringes, supporting the subsequent acquisition of the projector parameter matrix.

[0019] In one implementation, the target image is an original image obtained by the camera when the projector does not project onto a single sphere. The captured original image is subjected to edge extraction and ellipse fitting of a single small ball to obtain the coordinates of the ellipse center in the camera image plane coordinate system and the major axis of the ellipse, supporting the acquisition of the external parameters of the camera. Although the average brightness map based on the captured phase-shifted fringe pattern can also be used for ellipse edge extraction, due to the directivity of projection, it is easy to cause shadows in the image of a single sphere, resulting in problems such as missing ellipse edges. Therefore, an additional original image of a single small ball is captured separately when the projector is not projecting to accurately obtain the ellipse edge and ensure the accuracy of subsequent calibration.

[0020] In one implementation, the origin of the world coordinate system is the center of a single sphere, the X-axis of the world coordinate system is the horizontal direction of the camera image plane, and the Y-axis of the world coordinate system is the vertical direction of the camera image plane.

[0021] In one implementation, the origin of the camera image plane coordinate system is the principal point of the camera image plane, the X-axis of the camera image plane coordinate system is the horizontal direction of the camera image plane, and the Y-axis of the camera image plane coordinate system is the vertical direction of the camera image plane.

[0022] In one implementation, the external parameters of the camera include: the rigid transformation of the camera coordinate system relative to the world coordinate system.

[0023] In a second aspect, a computer device is provided, including a processor and a memory, the processor being coupled to the memory, and the processor being configured to execute a computer program or instructions stored in the memory to enable the electronic device to implement the structured light system calibration method according to any one of the first aspect.

[0024] In a third aspect, a structured light system is provided, including a projector, a camera, and the computer device according to the second aspect.

[0025] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program runs on a computer device, it enables the computer device to execute the structured light system calibration method according to any one of the first aspect.

[0026] In a fifth aspect, a computer program product is provided. When the computer program product runs on a computer, it enables the computer to execute the structured light system calibration method according to any one of the first aspect.

[0027] It can be understood that the beneficial effects of the above second aspect to fifth aspect can be referred to the relevant descriptions in the above first aspect, and will not be elaborated here. Description of the Drawings

[0028] Figure 1 A schematic diagram of a structured light system provided by an embodiment of the present application;

[0029] Figure 2 A schematic diagram of another structured light system provided by an embodiment of the present application;

[0030] Figure 3 A schematic diagram of a multi-frequency phase-shifted fringe pattern provided by an embodiment of the present application;

[0031] Figure 4 A flowchart of the overall solution provided by an embodiment of the present application;

[0032] Figure 5 A schematic diagram of a world coordinate system, a camera coordinate system, and a camera image plane coordinate system provided by an embodiment of the present application;

[0033] Figure 6 A flowchart of a structured light system calibration method provided by an embodiment of the present application;

[0034] Figure 7 A schematic diagram of sphere center coordinate estimation provided by an embodiment of the present application;

[0035] Figure 8 A schematic diagram of the sphere center coordinates provided by an embodiment of the present application;

[0036] Figure 9Schematic diagram for obtaining the intersection line between the i-th plane provided by the embodiment of the present application and the surface of a single sphere;

[0037] Figure 10 Schematic diagram for obtaining the coordinates of all intersection points in the camera image plane coordinate system provided by the embodiment of the present application. Detailed implementation manners

[0038] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application.

[0039] In the description of the present application, unless otherwise specified, " / " means "or". For example, A / B may represent A or B. In the description of the present application, "and / or" is merely an association relationship describing associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0040] In the description of the present application, terms such as "first" and "second" in the specification and claims are used to distinguish different objects or different treatments of the same object, rather than to describe a specific order of the objects.

[0041] The reference to "an embodiment" or "some embodiments" etc. described in the specification of the present application means that specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Thus, statements such as "in an embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. Terms such as "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0042] In a traditional structured light system, calibration of the structured light system is usually performed by projecting onto a calibration board. However, this method has the technical problems of being cumbersome and time-consuming. In view of this technical problem, an embodiment of the present application provides a method for calibrating a structured light system. In this method, a single sphere is used instead of the traditional calibration board. The projector projects onto the single sphere, and the camera sends the images of the single sphere it captures and the projected images to a computer device. Thus, the computer device can obtain the external parameters of the camera, the internal parameters and external parameters of the projector by using the images collected by the camera and the known internal parameters of the camera, thereby realizing the calibration of the structured light system. Since the entire calibration process only requires placing a single sphere at one position and performing one "projection - shooting" process, instead of projecting and shooting multiple small balls or single small balls at multiple different positions. Therefore, the calibration steps of the structured light system are greatly simplified. In addition, since there is no specific restriction on the position where the single sphere is placed and the spatial position of the single sphere does not need to be known in advance, only the real radius or virtual radius of the single sphere needs to be known. Therefore, the calibration of the present application has better flexibility.

[0043] Figure 2 FIG. is a schematic diagram of a structured light system provided by an embodiment of the present application. As Figure 2 shown, the system may include a single sphere, a projector, a camera, and a computer device. The optical axis angles of the projector and the camera are set at a certain angle. The camera is connected to the computer device by a wired or wireless method. Among them, the optical centers of the optical axes of the projector and the camera may be at the same height or at different heights.

[0044] The single sphere is a sphere with any radius. For example, the single small ball can be a sphere with a single color (such as white) surface, or a sphere with other mixed color surfaces. There is no need to map or print any feature points on the sphere surface. During the projection and shooting process, the small ball can be projected and shot in a natural light scene. It should be noted that, as Figure 2 shown, the single small ball needs to be placed within the common field of view of the camera and the projector, but the position of the single small ball does not need to be known.

[0045] The projector is used to project a specific pattern onto the single sphere. The specific pattern can be Figure 3 the multi - frequency phase - shift fringe pattern, single - frequency phase - shift fringe pattern + Gray - code fringe pattern, or single - frequency single - frame fringe pattern shown, and of course other patterns are also possible. It should be noted that the following embodiments take the projector projecting a multi - frequency phase - shift fringe pattern onto a single sphere as an example for illustration, which does not limit the embodiments of the present application.

[0046] The camera is used to capture images of a single sphere. For example, in the case where the projector projects a multi-frequency phase-shift fringe pattern onto a single sphere, the camera captures an image of the single sphere. Another example is that in the case where the projector does not project onto the single sphere, the camera captures the original image of the single sphere in a natural light scene.

[0047] The computer device is used to obtain the external parameters of the camera, the internal parameters and external parameters of the projector based on the images captured by the camera and the known internal parameters of the camera, so as to realize the calibration of the structured light system.

[0048] Figure 4 This is the overall scheme flowchart provided by the embodiment of the present application. As Figure 4 shown, the single sphere is placed in the common field of view of the camera and the projector and fixed in position manually, automatically or in other ways. Then, the projector projects multi-frequency phase-shift fringe patterns onto the surface of the single sphere in sequence, and the camera captures images in sequence to obtain the captured multi-frequency phase-shift fringe patterns. At the same time, keeping the position of the single small sphere unchanged, when the projector does not project onto the single small sphere, the camera captures the single small sphere to obtain the original image of the single small sphere. Through algorithms such as fringe analysis, edge extraction and ellipse fitting, the captured multi-frequency phase-shift fringe patterns and the original image of the single small sphere are processed respectively to obtain the phase of the fringe pattern, the coordinates of the ellipse center in the camera image plane coordinate system and the length of the major axis of the ellipse. Then, combined with the internal parameters of the camera (abbreviation: internal parameters), the external parameters of the camera (abbreviation: external parameters), the internal parameters and external parameters of the projector (abbreviation: internal and external parameters) and other parameters can be obtained to realize the calibration of the structured light system.

[0049] During the calibration process of the whole system, the position of the single small sphere does not need to be known. It only needs to fix the single small sphere at an arbitrary position and project and capture the single small sphere. In the traditional calibration plate-based method, the calibration plate needs to be placed at multiple different positions and postures for shooting and projection. Therefore, compared with the traditional calibration plate-based method, the method provided by the present application can significantly simplify the calibration process and improve the calibration efficiency.

[0050] The embodiment of the present application involves multiple coordinate systems, such as the world coordinate system, the camera coordinate system, the camera image plane coordinate system and the projector image plane coordinate system. To understand these coordinate systems more clearly, the following is combined with Figure 5 for example illustration.

[0051] Figure 5 This is the schematic diagram of the world coordinate system, the camera coordinate system and the camera image plane coordinate system provided by the embodiment of the present application.

[0052] Camera coordinate system O C -X C -Y C -Z C: The origin of the camera coordinate system is the lens center O of the camera. C , the X-axis of the camera coordinate system is the horizontal direction of the camera image plane, the Y-axis of the camera coordinate system is the vertical direction of the camera image plane, and the Z-axis of the camera coordinate system is the optical axis of the camera lens.

[0053] World coordinate system O W -X W -Y W -Z W : The origin of the world coordinate system is the center O of a single sphere. W , the X-axis of the world coordinate system is the intersection line of plane P0 and plane O W O C X C . Among them, plane P0 is the plane formed by the points tangent to the spherical surface of a single small ball from the camera lens center O C . Similarly, the Y-axis of the world coordinate system is the intersection line of plane P0 and plane O W O C Y C , and the Z-axis of the world coordinate system is determined according to the center O of a single sphere W and the lens center O of the camera C . The Z-axis of the world coordinate system is specifically the direction where the line O W O C is located.

[0054] Camera image plane coordinate system o0-x C -y C : The origin of the camera image plane coordinate system is the principal point o0 of the camera image plane. The X-axis of the camera image plane coordinate system is the horizontal direction of the camera image plane, and the Y-axis of the camera image plane coordinate system is the vertical direction of the camera image plane.

[0055] In the embodiments of the present application, the principal point of the camera image plane refers to the intersection point of the optical axis of the camera lens and the camera image plane.

[0056] It should be noted that Figure 5 only shows a method for establishing a world coordinate system, a camera coordinate system, and a camera image plane coordinate system. Other methods can also be used to establish a world coordinate system, a camera coordinate system, and a camera image plane coordinate system. For example, the world coordinate system can be a coordinate system with the lens center as the origin O C , X, Y, and Z axis directions unchanged O C -X W -Y W -Z W , and the camera image plane coordinate system can be a coordinate system with the upper left corner o of the image UL as the origin and X and Y axis directions unchanged o UL -x C -y C .

[0057] In addition, Figure 5 The projector image plane coordinate system is not shown. In one implementation, the origin of the projector image plane coordinate system is the principal point of the projector image plane, the X-axis of the projector image plane coordinate system is the horizontal direction of the projector image plane, and the Y-axis of the projector image plane coordinate system is the vertical direction of the projector image plane. Herein, the projector image plane refers to the plane where the projector DLP chip or LCD is located, and the principal point of the projector image plane refers to the intersection point of the optical axis of the projector lens and the camera image plane. It should be understood that the projector image plane coordinate system can also be established in other ways.

[0058] Figure 6 The flowchart of a structured light system calibration method provided by an embodiment of the present application. The execution subject of this method can be a computer device or a functional module in the computer device, which is not limited in the embodiments of the present application. As Figure 6 shown, this method may include the following S1 to S5.

[0059] S1. The computer device acquires a target image, where the target image is an image obtained by the camera shooting a single sphere.

[0060] In one way, the target image is a phase-shifted fringe pattern obtained by the camera shooting when the projector projects onto a single sphere. The captured phase-shifted fringe pattern is used to obtain the absolute phases of the horizontal and vertical fringes subsequently and establish the mapping relationship between the projector image plane coordinates and the camera image plane coordinates. After the camera image plane coordinates are known, the corresponding projector image plane coordinates can be directly obtained through the absolute phases of the above horizontal and vertical fringes, supporting the subsequent acquisition of the projector parameter matrix.

[0061] In another way, the target image is an original image obtained by the camera shooting when the projector does not project onto a single sphere. The captured original image is subjected to edge extraction and ellipse fitting of a single small ball, which is used to obtain the coordinates of the ellipse center in the camera image plane coordinate system and the major axis of the ellipse, supporting the acquisition of the camera external parameters. Although the average brightness map based on the captured phase-shifted fringe pattern can also be used for ellipse edge extraction, due to the directivity of projection, it is easy to cause shadows in the image of a single sphere, resulting in problems such as missing ellipse edges. Therefore, an additional original image of a single small ball when the projector does not project is captured separately to accurately obtain the ellipse edge and ensure the accuracy of subsequent calibration.

[0062] S2. The computer device acquires the external parameters of the camera according to the target image and the internal parameters of the camera.

[0063] In some embodiments, the internal parameters of the camera include: the focal length of the camera's lens and the coordinates of the principal point of the camera, where the principal point coordinates are the coordinates of the intersection point of the camera's lens optical axis and the camera image plane in the camera image plane coordinate system. The external parameters of the camera include: the rigid transformation of the camera coordinate system relative to the world coordinate system, i.e., the rotation matrix and translation vector between the two coordinate systems.

[0064] (1) According to the target image and the internal parameters of the camera, obtain the coordinates of the center of a single sphere in the camera coordinate system.

[0065] Specifically, it is achieved through the following ① - ④:

[0066] ① The computer device performs edge extraction of a single sphere on the target image to obtain the coordinates of each edge point in the camera image plane coordinate system, performs ellipse fitting on the coordinates of each edge point to obtain an ellipse containing each edge point, and determines the center point o of the ellipse. e The coordinates (x e , y e ) of the center point o in the camera image plane coordinate system and the length a of the major semi - axis of the ellipse.

[0067] Assume that the target image is the original image captured by the camera when the projector does not project onto a single small ball, denoted as I S . Process I through the edge extraction algorithm S to obtain the image edge points p Sm , m = 1, 2, …, J being the number of edge points. The coordinates of each obtained edge point in the camera image plane coordinate system are (x Sm , y Sm ), m = 1, 2, …, J. Perform ellipse fitting on all edge points p Sm to obtain the center point o of the ellipse e and the length a of the major semi - axis. The coordinates of the obtained ellipse center point in the camera image plane coordinate system are (x e , y e ).

[0068] ② The computer device determines the coordinates of the center point of the ellipse in the camera coordinate system according to the coordinates (x e , y e ) of the center point of the ellipse in the camera image plane coordinate system, the focal length of the camera's lens, and the coordinates of the principal point of the camera.

[0069] Denote the focal length of the camera lens and the principal point coordinates as f and (x0, y0) respectively. According to Equation (1), the coordinates X Ce = [X Ce , Y Ce , Z Ce of the center point of the ellipse in the camera coordinate system can be calculated.

[0070]

[0071] ③The computer device determines the target ratio k according to the length a of the major semi-axis of the ellipse and the coordinates X of the center point of the ellipse in the camera coordinate system. The calculation process is shown in Equation (2). Ce , and the radius R of a single sphere, to determine the target ratio k. The calculation process is shown in Equation (2).

[0072]

[0073] ④The computer device takes the product of the coordinates of the center point of the ellipse in the camera coordinate system and the target ratio as the coordinates of the center of a single sphere in the camera coordinate system.

[0074] Denote the coordinates of the center of a single sphere in the camera coordinate system as X CS = [X CS , Y CS , Z CS , then its coordinates can be calculated by Equation (3).

[0075]

[0076] In some embodiments, the radius R of a single sphere is the true radius of a single sphere or a preset virtual radius.

[0077] When the radius of a single small sphere is the true radius, the coordinates of the estimated center of the sphere in the camera coordinate system are real physical coordinates. Based on these coordinates for subsequent processing, the obtained external parameters of the camera and the projector parameters are real parameters. And based on the calibrated system for three-dimensional reconstruction of the scene, the obtained three-dimensional reconstruction result is consistent with the true size of the scene.

[0078] When the radius of a single small sphere is a preset virtual radius, the coordinates of the estimated center of the sphere in the camera coordinate system differ from the real physical coordinates by a certain scaling ratio. The rotation matrix in the obtained external parameters of the camera is consistent with the true rotation matrix of the camera, but the translation vector in the external parameters of the camera differs from the real value by a certain scaling ratio. At the same time, the obtained internal parameters and external parameters of the projector are also inconsistent with the real parameters. However, based on the calibration results of the camera and the projector obtained with the preset virtual radius, the scene can still be three-dimensionally reconstructed. Only the reconstruction result differs from the true size of the scene by a certain scaling ratio, but there is a similarity in structure between the two, only differing by a certain scale change.

[0079] Exemplarily, Figure 7 shows a schematic diagram of obtaining the coordinates of the center of the sphere. As Figure 7 shown, after inputting the original image I S of a single small sphere, after edge extraction, the image edge points p Sm, where \(m = 1, 2, \ldots, J\). Ellipse fitting is performed on these edge points to obtain the coordinates \((x e , y e )\) of the ellipse center in the camera image plane coordinate system, as well as the length \(a\) of the major semi - axis of the ellipse. Then, according to steps ② and ③, the coordinates \(X Ce \) of the ellipse center point in the camera coordinate system and the target ratio \(k\) are determined respectively. Finally, multiplying the two gives the coordinates of the center of a single sphere in the camera coordinate system as \(X CS .

[0080] Exemplarily, Figure 8 a schematic diagram of the center - of - sphere coordinates is shown. As Figure 8 shown, the coordinates \(X Ce = [X Ce , Y Ce , Z Ce \) of the ellipse center point in the camera coordinate system can be calculated according to Equation (1). And the center of a single small sphere is on the same straight line as the ellipse center point. Therefore, the coordinates \(X CS = [X CS , Y CS , Z CS \) of the center of the sphere in the camera coordinate system has a proportional relationship with the coordinates \(X Ce \), and this proportional relationship is the target ratio \(k\). Therefore, \(X CS \) can be directly obtained through formula (3).

[0081] (2) According to the coordinates of the center of a single sphere in the camera coordinate system, obtain the parameter matrix of the camera, which includes the internal parameters and external parameters of the camera.

[0082] Specifically, it is achieved through the following ① - ④:

[0083] ① The computer device normalizes the coordinates of the center of a single sphere in the camera coordinate system to obtain a unit vector, and determines the first angle and the second angle according to the unit vector.

[0084] The unit vector \(V\) after normalizing the coordinates of the center of a single sphere in the camera coordinate system is

[0085]

[0086] According to the unit vector \(V\), the first angle \(\theta x \) and the second angle \(\theta y \) are calculated respectively through Equation (5).

[0087]

[0088] ② According to the first angle \(\theta x, generate the first rotation matrix \(R\) of the unit vector rotating around the \(X\)-axis of the camera coordinate system CX ; and according to the second angle \(\theta\) y , generate the second rotation matrix \(R\) of the unit vector rotating around the \(Y\)-axis of the camera coordinate system CY . As shown in Equation (6).

[0089]

[0090] ③ Multiply the first rotation matrix \(R\) CX and the second rotation matrix \(R\) CY to obtain the transpose matrix of the rotation matrix \(R\) of the camera C . See Equation (7).

[0091]

[0092] Transpose the above product to obtain the final rotation matrix \(R\) C .

[0093] ④ According to the internal parameters of the camera, the rotation matrix of the camera, and the coordinates of the center point of the ellipse in the camera image plane coordinate system, generate the parameter matrix of the camera, as shown in Equation (8). This parameter matrix \(M\) C includes the internal parameters \(K\) of the camera C and the external parameters \(R\) of the camera C and \(X\) CS .

[0094]

[0095] Where \(I\) is a \(3\times3\) identity matrix.

[0096] S3. On each of the \(N\) intersection lines of the surface of a single sphere and \(N\) planes, obtain \(M\) intersection points. Among them, the \(N\) planes are planes perpendicular to the \(Z\)-axis of the world coordinate system. Both \(N\) and \(M\) are integers greater than or equal to 3.

[0097] Denote the diameter of a single small ball as \(R\), then the expression of a single small ball in the world coordinate system is Equation (9).

[0098]

[0099] The expression of the \(i\)-th plane is \(Z\) W \(=Z\) Wi , \(i = 1, 2, \cdots, N\), then the intersection point coordinates \(X\) Wi,j \(=[X\) Wi,j , \(Y\) Wi,j , \(Z\) Wi,j , \(j = 1, 2, \cdots, M\), and can be obtained through Equation (10).

[0100]

[0101] Exemplarily, Figure 9 a schematic diagram of obtaining the intersection line between the i-th plane and the surface of a single sphere is shown. As Figure 9 shown, for the i = 1 plane, its expression is Z W = Z W1 , and its intersection line with the surface of a single sphere is C1. The coordinates of all points on the intersection line C1 simultaneously satisfy the expressions of the i = 1 plane and the surface of a single sphere. Therefore, by combining the expressions of the i = 1 plane and the surface of a single sphere and solving, the coordinates of all points on the intersection line C1 in the world coordinate system can be obtained, as shown in Equation (10). Similarly, for the i = 2 plane, its intersection line with the surface of a single sphere is C2. The coordinates of all points on the intersection line C2 are also obtained by solving the system of equations shown in Equation (10). By analogy, the coordinates of all points on the intersection lines between all N planes and the surface of a single sphere can be obtained through similar steps. Thus, the coordinates of the M intersection points on the N intersection lines of all N planes in the world coordinate system are obtained.

[0102] S4. The computer device obtains the coordinates of all intersection points in the camera image plane coordinate system according to all the obtained intersection points, the internal parameters of the camera, and the external parameters of the camera.

[0103] Specifically, it is achieved through the following ① - ②:

[0104] ① According to the Z coordinates of all intersection points in the world coordinate system, as well as the internal parameters of the camera and the external parameters of the camera, a homography matrix is determined. Among them, the homography matrix is a transformation matrix between the target vector and the coordinates in the camera image plane coordinate system, and the target vector is composed of the X-axis and Y-axis coordinates of all intersection points in the world coordinate system.

[0105] The homography matrix H corresponding to the i-th plane i can be obtained through Equation (11)

[0106]

[0107] where γ i is expressed as

[0108]

[0109] ② According to the product of the homography matrix and the target vector, the coordinates of all intersection points in the camera image plane coordinate system are obtained.

[0110] The coordinates of the intersection points between the i-th plane and the surface of a single sphere after imaging onto the camera image plane are x Ci,j = [x Ci,j , y Ci,j , j = 1, 2,..., M, and can be obtained through Equation (13).

[0111]

[0112] For example, Figure 10 FIG. 1 shows a schematic diagram of obtaining the coordinates of all intersection points in the camera image plane coordinate system. Figure 10 As shown, for the point on the intersection line C1 of the i=1th plane and the spherical surface of a single ball, its coordinate is X W1,j =[X W1,j ,Y W1,j ,Z W1,j ]. Ellipse C e1 is the image of the intersection line C1 on the camera image plane. According to the homography matrix H1, C1 can be directly mapped to C in the camera image plane through formula (13): e1 , thus obtaining the coordinates of the intersection point in the camera image plane coordinate system.

[0113] S5. The computer device obtains the internal parameters and external parameters of the projector according to the coordinates of all intersection points in the camera image plane coordinate system and the phase of the phase shift fringe pattern.

[0114] This is achieved through the following ①-②:

[0115] ①According to the coordinates of all intersection points in the camera image plane coordinate system and the phase of the phase-shift fringe pattern, the coordinates of all intersection points in the projector image plane coordinate system are obtained.

[0116] According to the phase shift fringe pattern obtained in S1, the absolute phases of the horizontal and vertical phase shift fringe patterns can be obtained, which are recorded as and Combine the coordinates of all the intersection points obtained in S2 in the camera image plane coordinate system, and obtain the corresponding projector image coordinates according to formula (14).

[0117]

[0118] Among them, f V and f H are the frequencies of the known (preset) horizontal and vertical stripes respectively.

[0119] ②According to the coordinates of all intersection points in the world coordinate system and the coordinates of all intersection points in the projector image plane coordinate system, the parameter matrix of the projector is obtained. The parameter matrix of the projector includes the internal parameters and the external parameters of the projector.

[0120] The parameter matrix of the projector is M P , the following equations can be established for each intersection point. By combining the equations of all intersection points shown in equation (15), M can be directly solved by the least squares algorithm. P .

[0121]

[0122] By using the structured light system calibration method provided in the above embodiments, it is only necessary to place a single small ball at a single position for projection and shooting to obtain the parameters of the camera and the projector, avoiding the cumbersome process of repeated projection and shooting at multiple different positions in the conventional calibration method, which can greatly simplify the calibration process. At the same time, the placement position of the single small ball does not need to be known, so it has good flexibility.

[0123] An embodiment of the present application also provides a computer device, including a processor and a memory. The processor is coupled to the memory, and the processor is configured to execute computer programs or instructions stored in the memory so that the computer device implements the methods in the above embodiments.

[0124] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium runs on a computer device, the computer device is caused to execute the method as shown above. The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0125] An embodiment of the present application also provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer is caused to execute the methods in the above embodiments.

[0126] An embodiment of the present application also provides a chip, which is coupled to the memory. The chip is configured to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments. The chip can be a general-purpose processor or a dedicated processor. In some embodiments, the chip includes a processor, a transceiver, and a storage medium. The processor is used to support the computer device to execute the technical solutions as shown above. The transceiver is used to accept the control of the processor and support the computer device to execute the technical solutions as shown above.

[0127] It should be noted that the chip can be implemented by using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing various functions described throughout the present application.

[0128] The computer device, computer-readable storage medium, computer program product, and chip provided in the embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the methods provided above, and will not be elaborated here.

[0129] It should be understood that the above is only to help those skilled in the art better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application. Those skilled in the art can obviously make various equivalent modifications or changes according to the above examples given. For example, in the various embodiments of the above detection method, some steps may not be necessary, or some steps may be newly added, etc. Or any combination of any two or any multiple of the above embodiments. Such modified, changed, or combined solutions also fall within the scope of the embodiments of the present application.

[0130] It should also be understood that the description of the embodiments of the present application above focuses on emphasizing the differences between the various embodiments. The same or similar parts not mentioned can be referred to each other. For the sake of brevity, they will not be elaborated here.

[0131] It should also be understood that the size of the serial numbers of the above processes does not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0132] It should also be understood that in the embodiments of the present application, "pre-set" and "pre-defined" can be implemented by pre-saving corresponding codes, tables, or other ways that can be used to indicate relevant information in a device (for example, including an electronic device). The present application does not limit its specific implementation manner.

[0133] It should also be understood that the division of the manners, situations, categories, and embodiments in the embodiments of the present application is only for the convenience of description and should not constitute a special limitation. The features in various manners, categories, situations, and embodiments can be combined without conflict.

[0134] It should also be understood that in the various embodiments of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referred to each other. The technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0135] Finally, it should be noted that the above description content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any change or replacement within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for calibrating a structured light system, characterized in that, The method includes: Obtaining a target image, where the target image is an image obtained by a camera photographing a single sphere; Obtaining the external parameters of the camera according to the target image and the internal parameters of the camera; On each of the N intersection lines of the surface of the single sphere and N planes, obtaining M intersection points, where the N planes are planes perpendicular to the Z-axis of the world coordinate system, and the Z-axis of the world coordinate system is determined according to the center of the single sphere and the center of the camera lens, and both N and M are integers greater than or equal to 3; Obtaining the coordinates of all the intersection points in the camera image plane coordinate system according to all the obtained intersection points, the internal parameters of the camera, and the external parameters of the camera; Obtaining the internal parameters and external parameters of the projector according to the coordinates of all the intersection points in the camera image plane coordinate system and the phase of the phase-shifted fringe pattern, where the phase-shifted fringe pattern is an image obtained by the projector projecting onto the single sphere and photographed by the camera.

2. The method according to claim 1, characterized in that, The obtaining the external parameters of the camera according to the target image and the internal parameters of the camera includes: Obtaining the coordinates of the center of the single sphere in the camera coordinate system according to the target image and the internal parameters of the camera; Obtaining the parameter matrix of the camera according to the coordinates of the center of the single sphere in the camera coordinate system, where the parameter matrix includes the internal parameters and the external parameters of the camera; Wherein, the origin of the camera coordinate system is the center of the camera lens, the Z-axis of the camera coordinate system is the optical axis of the camera lens, the X-axis of the camera coordinate system is the horizontal direction of the camera image plane, and the Y-axis of the camera coordinate system is the vertical direction of the camera image plane.

3. The method according to claim 2, characterized in that, The internal parameters of the camera include the focal length of the camera lens and the coordinates of the principal point of the camera, and the coordinates of the principal point are the coordinates of the intersection point of the optical axis of the camera lens and the camera image plane in the camera image plane coordinate system; The obtaining the coordinates of the center of the single sphere in the camera coordinate system according to the target image and the internal parameters of the camera includes: Performing edge extraction of the single sphere on the target image to obtain the coordinates of each edge point in the camera image plane coordinate system, performing ellipse fitting on the coordinates of each edge point to obtain an ellipse containing each edge point, and determining the coordinates of the center point of the ellipse in the camera image plane coordinate system and the length of the major semi-axis of the ellipse; Determining the coordinates of the center point of the ellipse in the camera coordinate system according to the coordinates of the center point of the ellipse in the camera image plane coordinate system, the focal length of the camera lens, and the coordinates of the principal point of the camera; Determining a target ratio according to the length of the major semi-axis of the ellipse, the coordinates of the center point of the ellipse in the camera coordinate system, and the radius of the single sphere; Taking the product of the coordinates of the ellipse center point in the camera coordinate system and the target ratio as the coordinates of the center of the single sphere in the camera coordinate system.

4. The method according to claim 3, characterized in that, The radius of the single sphere is the true radius of the single sphere or a preset virtual radius.

5. The method according to claim 3, characterized in that, Based on the coordinates of the center of the single sphere in the camera coordinate system, obtain the parameter matrix of the camera, where the parameter matrix includes the internal parameters and external parameters of the camera, including: Normalize the coordinates of the center of the single sphere in the camera coordinate system to obtain a unit vector, and determine a first angle and a second angle based on the unit vector; Generate a first rotation matrix for the unit vector to rotate around the X-axis of the camera coordinate system according to the first angle; and generate a second rotation matrix for the unit vector to rotate around the Y-axis of the camera coordinate system according to the second angle; Multiply the first rotation matrix and the second rotation matrix to obtain the transpose matrix of the rotation matrix of the camera; Transpose the transpose matrix of the rotation matrix of the camera to obtain the rotation matrix of the camera; Generate the parameter matrix of the camera based on the internal parameters of the camera, the rotation matrix of the camera, and the coordinates of the center point of the ellipse in the camera image plane coordinate system.

6. The method according to claim 1, characterized in that, The obtaining the coordinates of all the intersection points in the camera image plane coordinate system according to all the obtained intersection points, the internal parameters of the camera, and the external parameters of the camera includes: Determine a homography matrix based on the Z coordinates of all the intersection points in the world coordinate system, as well as the internal parameters of the camera and the external parameters of the camera. The homography matrix is a transformation matrix between the target vector and the coordinates in the camera image plane coordinate system, and the target vector is composed of the X-axis and Y-axis coordinates of all the intersection points in the world coordinate system; Obtain the coordinates of all the intersection points in the camera image plane coordinate system according to the product of the homography matrix and the target vector.

7. The method according to claim 1, characterized in that, The obtaining the internal parameters and external parameters of the projector according to the coordinates of all the intersection points in the camera image plane coordinate system and the phase of the phase-shifted fringe pattern includes: Obtain the coordinates of all the intersection points in the projector image plane coordinate system according to the coordinates of all the intersection points in the camera image plane coordinate system and the phase of the phase-shifted fringe pattern; Obtain the parameter matrix of the projector according to the coordinates of all the intersection points in the world coordinate system and the coordinates of all the intersection points in the projector image plane coordinate system. The parameter matrix of the projector includes the internal parameters and external parameters of the projector; Wherein, the origin of the projector image plane coordinate system is the principal point of the projector image plane, the X-axis of the projector image plane coordinate system is the horizontal direction of the projector image plane, and the Y-axis of the projector image plane coordinate system is the vertical direction of the projector image plane.

8. The method according to any one of claims 1 to 7, characterized in that, The target image is the phase-shifted fringe pattern obtained by the camera when the projector projects onto the single sphere; or, the target image is the original image obtained by the camera when the projector does not project onto the single sphere.

9. The method according to any one of claims 1 to 7, characterized in that, The origin of the world coordinate system is the center of the single sphere, the X-axis of the world coordinate system is the horizontal direction of the camera image plane, and the Y-axis of the world coordinate system is the vertical direction of the camera image plane; The origin of the camera image plane coordinate system is the principal point of the camera image plane. The X-axis of the camera image plane coordinate system is the horizontal direction of the camera image plane, and the Y-axis of the camera image plane coordinate system is the vertical direction of the camera image plane.

10. The method according to any one of claims 1 to 7, characterized in that, The external parameters of the camera include the rigid transformation of the camera coordinate system relative to the world coordinate system.

11. A computer device, characterized in that, It includes a processor and a processor. The processor is coupled to the memory, and the processor is configured to execute the computer program or instructions stored in the memory, so that the computer device implements the structured light system calibration method according to any one of claims 1 to 10.

12. A structured light system, characterized in that, The structured light system includes a projector, a camera, and the computer device according to claim 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program. When the computer program runs on a computer device, the computer device is caused to execute the structured light system calibration method according to any one of claims 1 to 10.

Images