Active light camera calibration parameter calculation method, device, equipment and storage medium

Abstract

The present invention relates to the field of camera calibration technology, and discloses a calibration parameter calculation method, device, equipment and storage medium of an active light camera. The method is applied to an optical motion capture system, and the system includes multiple active light rigid bodies including multiple feature points and multiple active light cameras. The method includes controlling the active light rigid bodies to emit light and triggering the active light cameras to take pictures of the active light rigid bodies to obtain light-emitting images to determine the image coordinates of the feature points and the active light camera to be recalibrated; creating a spatial coordinate system according to the relative positions of other active light cameras; based on the spatial coordinate system, calculating the coordinates of the feature points and the projection point coordinates of the feature points on the active light camera to be recalibrated; calculating the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the projection point coordinates to determine the calibration parameters of the active light camera to be recalibrated. The present invention automatically calibrates the system without affecting the normal operation of the multi-camera system.

Description

Technical Field

[0001] The present invention relates to the field of camera calibration technology, and in particular to a calibration parameter calculation method, device, equipment and storage medium for an active light camera. Background Art

[0002] In existing optical motion capture systems, there is a certain relationship between the three-dimensional position of an object in real space and the two-dimensional position obtained by camera imaging. By establishing a geometric model between the two and migrating the relevant parameters in the geometric model to the camera, the three-dimensional position of the object is associated with the two-dimensional position in the camera. In a multi-camera system, the accuracy of the internal and external parameters of each camera will directly affect the accuracy of the entire system in capturing objects in real space. Therefore, before the optical motion capture system is operated, it is necessary to determine the status of all cameras themselves and the relative positional relationship between different cameras. This step is called "camera calibration."

[0003] The existing camera calibration technologies mainly include chessboard calibration, one-dimensional calibration rod calibration, etc. The calibration principles and methods related to the two are very mature and will not be described here. However, the environment is constantly changing during the operation of the system, affecting the internal parameters of the camera; the camera installation environment inevitably vibrates, which causes the initial position of the camera and the relative position between the cameras to shift. Therefore, for the calibration of the chessboard and the one-dimensional calibration rod, the current work needs to be stopped or the system restarted to recalibrate, which will affect the work efficiency of the system operation. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem that dynamic calibration cannot be performed during the operation of a multi-camera system.

[0005] A first aspect of the present invention provides a calibration parameter calculation method for an active light camera, which is applied to an optical motion capture system, wherein the optical motion capture system comprises: a plurality of active light rigid bodies and a plurality of active light cameras, wherein the active light rigid bodies have a plurality of feature points, and the calibration parameter calculation method comprises:

[0006] Controlling the active light rigid body to emit light, and triggering the active light camera to take a picture of the active light rigid body to obtain a corresponding light-emitting image;

[0007] Determining, according to the luminous image, image coordinates corresponding to the feature points and an active light camera to be recalibrated in the active light cameras;

[0008] Creating a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0009] Based on the spatial coordinate system, the feature point coordinates corresponding to the feature point are calculated, and the projection point coordinates corresponding to the feature point coordinates are calculated using the active light camera to be recalibrated as a projection surface;

[0010] Calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0011] According to the projection difference, calibration parameters of the active light camera to be recalibrated are determined.

[0012] Optionally, in a first implementation manner of the first aspect of the present invention, determining, according to the luminous image, the image coordinates corresponding to the feature points and the active light camera to be recalibrated in the active light camera includes:

[0013] Determining image coordinates corresponding to the feature points according to the luminescent image;

[0014] Creating a three-dimensional coordinate system according to the relative positions of the active light cameras;

[0015] Based on the three-dimensional coordinate system, calculating the three-dimensional coordinates of each of the feature points and the two-dimensional coordinates corresponding to the three-dimensional coordinates, wherein the two-dimensional coordinates are obtained by projecting the three-dimensional coordinates on each active light camera;

[0016] An active light camera to be recalibrated among the active light cameras is determined according to the image coordinates and the two-dimensional coordinates.

[0017] Optionally, in a second implementation manner of the first aspect of the present invention, determining the active light camera to be recalibrated in the active light cameras according to the image coordinates and the two-dimensional coordinates includes:

[0018] Calculating a first distance value between the image coordinates and the two-dimensional coordinates;

[0019] Determining whether the first distance value is less than a preset distance value threshold;

[0020] If the first distance value is less than a preset distance value threshold, the corresponding active light camera is used as the active light camera to be recalibrated.

[0021] Optionally, in a third implementation manner of the first aspect of the present invention, if the first distance value is less than a preset distance value threshold, taking the corresponding active light camera as the active light camera to be recalibrated includes:

[0022] If the first distance value is less than a preset distance value threshold, determining whether there are multiple active light cameras whose first distance values ​​are greater than the preset distance threshold;

[0023] If so, sort the first distance values ​​to obtain a sorting result;

[0024] According to the sorting result, an active light camera to be recalibrated is selected from the multiple active light cameras.

[0025] Optionally, in a fourth implementation manner of the first aspect of the present invention, before calculating the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates, the method further includes:

[0026] Calculating a second distance value between the image coordinates corresponding to the active light camera to be recalibrated and the coordinates of each projection point;

[0027] The second distance values ​​are sorted, and the projection point coordinates and image coordinates corresponding to the second distance values ​​in the sorting result are matched one to one.

[0028] Optionally, in a fifth implementation manner of the first aspect of the present invention, the calibration parameters include a rotation matrix and a translation matrix, and determining the calibration parameters of the active light camera to be recalibrated according to the projection difference includes:

[0029] Determining whether the projection difference is less than a preset projection difference threshold;

[0030] If yes, then inversely project the projection difference to obtain a rotation matrix and a translation matrix of the active light camera to be recalibrated;

[0031] If not, the process jumps to executing the step of determining the active light camera to be recalibrated among the active light cameras.

[0032] Optionally, in a sixth implementation of the first aspect of the present invention, the calibration parameter calculation method further includes:

[0033] Marking the active light camera to obtain a corresponding optimization number;

[0034] At every preset period, the active light cameras are sequentially used as active light cameras to be recalibrated according to the optimization numbers.

[0035] A second aspect of the present invention provides a calibration parameter calculation device for an active light camera, comprising:

[0036] A photographing module, used for controlling the active light rigid body to emit light, and triggering the active light camera to take a photo of the active light rigid body to obtain a corresponding luminous image;

[0037] A camera determination module, used to determine the image coordinates corresponding to the feature points and the active light camera to be recalibrated in the active light camera according to the luminous image;

[0038] A creation module, used for creating a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0039] A first calculation module, used to calculate the feature point coordinates corresponding to the feature point based on the spatial coordinate system, and calculate the projection point coordinates corresponding to the feature point coordinates using the active light camera to be recalibrated as a projection surface;

[0040] A second calculation module is used to calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0041] The parameter determination module is used to determine the calibration parameters of the active light camera to be recalibrated according to the projection difference.

[0042] Optionally, in a first implementation manner of the second aspect of the present invention, the camera determination module further includes:

[0043] An image coordinate determination unit, used to determine the image coordinates corresponding to the feature points according to the luminous image;

[0044] A three-dimensional coordinate calculation unit, used to create a three-dimensional coordinate system according to the relative positions between the active light cameras; based on the three-dimensional coordinate system, calculate the three-dimensional coordinates of each of the feature points and the two-dimensional coordinates corresponding to the three-dimensional coordinates, wherein the two-dimensional coordinates are obtained by projecting the three-dimensional coordinates on each active light camera;

[0045] A camera determination unit is used to determine the active light camera to be recalibrated among the active light cameras according to the image coordinates and the two-dimensional coordinates.

[0046] Optionally, in a second implementation of the second aspect of the present invention, the camera determination unit is further used to:

[0047] a distance calculation subunit, configured to calculate a first distance value between the image coordinates and the two-dimensional coordinates;

[0048] A distance judgment subunit, used to judge whether the first distance value is less than a preset distance value threshold;

[0049] The camera determination subunit is configured to use the corresponding active light camera as the active light camera to be recalibrated if the first distance value is less than a preset distance value threshold.

[0050] Optionally, in a third implementation manner of the second aspect of the present invention, the camera determination subunit is further used to:

[0051] If the first distance value is less than a preset distance value threshold, determining whether there are multiple active light cameras whose first distance values ​​are greater than the preset distance threshold;

[0052] If so, sort the first distance values ​​to obtain a sorting result;

[0053] According to the sorting result, an active light camera to be recalibrated is selected from the multiple active light cameras.

[0054] Optionally, in a fourth implementation manner of the second aspect of the present invention, the calibration parameter calculation device further includes a matching module, and the matching module is used to:

[0055] Calculating a second distance value between the image coordinates corresponding to the active light camera to be recalibrated and the coordinates of each projection point;

[0056] The second distance values ​​are sorted, and the projection point coordinates and image coordinates corresponding to the second distance values ​​in the sorting result are matched one to one.

[0057] Optionally, in a fifth implementation manner of the second aspect of the present invention, the calibration parameters include a rotation matrix and a translation matrix, and the parameter determination module further includes:

[0058] A projection difference judging unit, used to judge whether the projection difference is less than a preset projection difference threshold;

[0059] a parameter determination unit, configured to perform a back-projection on the projection difference to obtain a rotation matrix and a translation matrix of the active light camera to be recalibrated;

[0060] The loop unit is used for, if no, jumping to the step of determining the active light camera to be recalibrated in the active light cameras.

[0061] Optionally, in a sixth implementation manner of the second aspect of the present invention, the calibration parameter calculation device further includes a loop optimization module, and the loop optimization module is further used to:

[0062] Marking the active light camera to obtain a corresponding optimization number;

[0063] At every preset period, the active light cameras are sequentially used as active light cameras to be recalibrated according to the optimization numbers.

[0064] A third aspect of the present invention provides a calibration parameter calculation device for an active light camera, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via a line; the at least one processor calls the instructions in the memory so that the calibration parameter calculation device for the active light camera executes the above-mentioned calibration parameter calculation method for the active light camera.

[0065] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned method for calculating calibration parameters of an active light camera.

[0066] In the technical solution provided by the present invention, an active light rigid body is controlled to emit light and an active light camera is triggered to take pictures at the same time. The three-dimensional structure of the active light rigid body is determined by feature point coordinates according to the light-emitting images captured by multiple active light cameras. The three-dimensional structure is then projected into each active light camera and compared with the light-emitting image captured by the camera itself. The active light camera to be recalibrated can be determined and its calibration parameters can be recalculated. The entire process does not require the system to be stopped, and the multi-camera system can be recalibrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of a first embodiment of a calibration parameter calculation method for an active light camera of the present invention;

[0068] Figure 2 Schematic diagram of a second embodiment of a calibration parameter calculation method for an active light camera of the present invention;

[0069] Figure 3 Schematic diagram of a third embodiment of a calibration parameter calculation method for an active light camera of the present invention;

[0070] Figure 4 Schematic diagram of a fourth embodiment of a calibration parameter calculation method for an active light camera of the present invention;

[0071] Figure 5 A schematic diagram of an embodiment of a calibration parameter calculation device for an active light camera of the present invention;

[0072] Figure 6 It is a schematic diagram of another embodiment of a calibration parameter calculation device for an active light camera of the present invention;

[0073] Figure 7 FIG. 4 is a schematic diagram of an embodiment of a calibration parameter calculation device for an active light camera in the present invention. DETAILED DESCRIPTION

[0074] The embodiments of the present invention provide a calibration parameter calculation method, device, equipment and storage medium for an active light camera, which obtains a luminous image by controlling an active light rigid body to emit light and triggering an active light camera to take a picture of the active light rigid body; then, the image coordinates of a feature point are determined according to the luminous image, and the active light camera to be recalibrated is further determined; then, according to the relative positions of other active light cameras, a control coordinate system is created to calculate the feature point coordinates of the feature point, and the projection point coordinates of the feature point on the active light camera to be recalibrated are calculated; finally, the calibration parameters of the active light camera to be recalibrated are determined by the projection difference between the image coordinates of the active light camera to be recalibrated and the projection point coordinates, so that the system can be automatically calibrated without affecting the normal operation of the multi-camera system.

[0075] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0076] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the calibration parameter calculation method of the active light camera in the embodiment of the present invention includes:

[0077] 101. Control the active light rigid body to emit light, and trigger the active light camera to take a picture of the active light rigid body to obtain a corresponding light-emitting image;

[0078] It is understandable that the execution subject of the present invention can be a calibration parameter calculation device of an active light camera, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as an execution subject as an example. The calibration parameter calculation method of the active light camera is applied to an optical motion capture system, which includes: a plurality of active light rigid bodies and a plurality of active light cameras, wherein the active light rigid body has a plurality of feature points.

[0079] In this embodiment, the active optical rigid body (hereinafter referred to as the rigid body) refers to a rigid body carrying a light source, where the light source is preferably an LED (Light Emitting Diode) that emits infrared light, and the active optical camera (hereinafter referred to as the camera) is also preferably an infrared camera.

[0080] Here, a synchronous trigger signal is generated by the server and sent to the rigid body and the camera at the same time, so that the rigid body emits light and the camera is triggered to expose and shoot, so as to ensure that the camera can capture the light signal of the light source on the rigid body. That is, the rigid body emits light and the camera shoots are always synchronized.

[0081] When the camera collects the luminous image of a rigid body, the light source is directly mapped onto the camera lens, forming a feature point in the luminous image. Therefore, the luminous image captured by the camera uses the light source as the feature point. All feature points constitute the main content of the luminous image and record the relative position coordinates of different light sources.

[0082] 102. Determine, according to the luminous image, image coordinates corresponding to the feature points and an active light camera to be recalibrated in the active light cameras;

[0083] In this embodiment, the luminous image is established on the camera coordinate system, recording the two-dimensional position relationship of the luminous source in space relative to the camera focus center, that is, the camera focus center is the origin of the camera coordinate system, the x-axis and y-axis are parallel to the X and Y axes of the image, and the Z axis is the camera optical axis, which is perpendicular to the luminous image plane. The object can be mapped to the camera through Z-axis translation and rotation to form a luminous image. Therefore, the relative coordinates of each feature point in the camera coordinate system, that is, the image coordinates, can be determined from the luminous image.

[0084] The luminous images captured by multiple cameras are used to determine the three-dimensional position coordinates of the feature points.

[0085] During the calibration process, the camera periodically photographs the rigid body to obtain a time-series luminous image. Here, the luminous images taken by different cameras at the same time node are first converted into the three-dimensional coordinate system of each camera according to the relative position relationship between the different cameras placed in advance, that is, the relative coordinates of the camera are converted into world coordinates.

[0086] Then, the three-dimensional coordinate system between the cameras and the captured luminous image are combined to calculate the feature point coordinates of each feature point, which are spatial coordinates in the spatial coordinate system. This method is a common technical means in the field of optical motion capture technology and will not be described in detail here.

[0087] Then, the feature point coordinates are combined with the three-dimensional coordinates of each camera, and the feature points are projected into the camera coordinate system of the camera to determine the two-dimensional coordinates of the feature points in the camera;

[0088] Finally, the Euclidean distance between the corresponding feature points is determined by comparing the two-dimensional coordinates of the feature points with the image coordinates. When the Euclidean distance between the two-dimensional coordinates of the feature points and the image coordinates is greater than the preset first distance value, it can be determined that the corresponding camera intrinsic or extrinsic parameters have changed and need to be recalibrated, and the camera is used as the camera to be recalibrated.

[0089] 103. Create a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0090] In this embodiment, after determining the camera to be recalibrated, the camera to be recalibrated is shielded, and then a new multi-camera system is established based on other cameras with correct internal and external parameters, and then the relative position relationship between the cameras in the new multi-camera system is converted into world coordinates to create a new spatial coordinate system (i.e., world coordinate system) of the multi-camera system. For example, if camera A is the camera to be recalibrated, camera A is excluded and the spatial coordinate systems of the other cameras except camera A are created.

[0091] 104. Based on the spatial coordinate system, calculate the feature point coordinates corresponding to the feature point, and use the active light camera to be recalibrated as a projection surface to calculate the projection point coordinates corresponding to the feature point coordinates;

[0092] In this embodiment, the new feature point coordinates of the feature points are calculated together according to the spatial coordinate system of the new multi-camera system. When the feature point coordinates of the feature point in the spatial coordinate system are projected onto the camera coordinate system of each camera, there are differences in rotation angle and offset position. Therefore, the current calibration parameters of the camera are required during the projection process so that the feature points can be correctly projected onto the graphic coordinates. Here, the feature point coordinates of the feature point are projected onto the camera coordinate system with the help of the calibration parameters to obtain the projection point coordinates.

[0093] It should be noted that the calibration parameters here include the rotation matrix and the translation matrix.

[0094] 105. Calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0095] In this embodiment, when the calibration parameters are correct, the current calibration parameters of the camera should project the feature coordinates of the feature points onto the graphic coordinates, that is, the projection point coordinates overlap with the image coordinates. However, the calibration parameters of the camera to be recalibrated are wrong, so its projection point coordinates do not overlap with the image coordinates. The projection difference here is the difference between the projection point coordinates and the image coordinates.

[0096] Specifically, for example, the current calibration parameters of feature point A include rotation matrix R0 and translation matrix T0. According to (R0, T0), feature point A is projected onto the camera to be recalibrated to obtain the projection point coordinates; the projection difference between the image coordinates and the projection point coordinates is calculated.

[0097] 106. Determine calibration parameters of the active light camera to be recalibrated according to the projection difference.

[0098] In this embodiment, the feature points are inversely projected onto the camera to be recalibrated according to the calculated projection difference and the image coordinates of the feature points, so as to obtain a new rotation matrix R1 and a new translation matrix T1, and then the process returns to step 102 to re-determine whether the first distance value of the current camera to be relabeled is less than the preset distance threshold. If so, (R1, T1) is used as the calibration parameter of the camera to be recalibrated. If not, the process continues to loop steps 103-106 until the first distance value of the camera to be relabeled is less than the preset distance threshold.

[0099] In this embodiment, the active light rigid body is controlled to emit light, and the active light camera is triggered to take a picture of the active light rigid body to obtain a light-emitting image; then the image coordinates of the feature points are determined according to the light-emitting image, and the active light camera to be recalibrated is further determined; then, according to the relative positions of other active light cameras, a control coordinate system is created to calculate the feature point coordinates of the feature points, so as to calculate the projection point coordinates of the feature points on the active light camera to be recalibrated; finally, the calibration parameters of the active light camera to be recalibrated are determined by the projection difference between the image coordinates and the projection point coordinates, so that the system can be automatically calibrated without affecting the normal operation of the multi-camera system.

[0100] See also Figure 2 The second embodiment of the method for calculating calibration parameters of an active light camera in the embodiment of the present invention includes:

[0101] 201. Control the active light rigid body to emit light, and trigger the active light camera to take a picture of the active light rigid body to obtain a corresponding light-emitting image;

[0102] 202. Determine the image coordinates corresponding to the feature points according to the luminous image;

[0103] In this embodiment, the camera focus center is used as the origin, the optical axis is used as the Z axis of the camera coordinate axis, and the feature point is used as a reference to the Z axis. The feature point is rotated and translated to be mapped to the luminous image. At this time, the feature point is transferred from the spatial coordinate system to the camera coordinate system, that is, the image coordinates of the feature point are plane coordinates. The content of the luminous image is each feature point, and the image coordinates of the feature point can be obtained by simply extracting the coordinate values ​​of each feature point from the image.

[0104] 203. Create a three-dimensional coordinate system according to the relative positions between the active light cameras;

[0105] In this embodiment, the relative positions of multiple cameras are converted into spatial positions, and the three-dimensional coordinate system of the multiple cameras is used as a reference to determine the feature point coordinates of each feature point. The three-dimensional coordinate system here is a world coordinate system, and converting from the relative positions of each camera to the world coordinate system is a common technology and will not be described here.

[0106] 204. Based on the three-dimensional coordinate system, calculate the three-dimensional coordinates of each of the feature points and the two-dimensional coordinates corresponding to the three-dimensional coordinates, wherein the two-dimensional coordinates are obtained by projecting the three-dimensional coordinates onto each active light camera;

[0107] In this example, the three-dimensional coordinates of the feature points in space are reconstructed with reference to the luminous images captured by each camera using the three-dimensional coordinate system of multiple cameras. The least squares method, parallax ranging method and other methods can be used for calculation, which will not be described in detail here.

[0108] The three-dimensional coordinates are then combined with the three-dimensional positions of each space of the multiple cameras (which can be obtained from the three-dimensional coordinate system of the camera), and the feature points are projected into the camera coordinate system of the camera to be recalibrated, that is, the feature point coordinates are converted into two-dimensional coordinates in the camera coordinate system.

[0109] Then, the three-dimensional coordinate system between the cameras and the captured luminous image are combined to calculate the feature point coordinates of each feature point, which are spatial coordinates in the spatial coordinate system. This method is a common technical means in the field of optical motion capture technology and will not be described in detail here.

[0110] Then, the feature point coordinates are combined with the three-dimensional coordinates of each camera, and the feature points are projected into the camera coordinate system of the camera to determine the two-dimensional coordinates of the feature points in the camera. Here, the current calibration parameters of the camera are used for projection.

[0111] 205. Determine an active light camera to be recalibrated among the active light cameras according to the image coordinates and the two-dimensional coordinates;

[0112] According to the image coordinates and the two-dimensional coordinates, the camera to be recalibrated is determined. The specific operation is as follows:

[0113] Calculating a first distance value between the image coordinates and the two-dimensional coordinates;

[0114] Determining whether the first distance value is less than a preset distance value threshold;

[0115] If the first distance value is less than a preset distance value threshold, the corresponding active light camera is used as the active light camera to be recalibrated.

[0116] For example, in camera A, the image coordinates of the feature point are N i (X i1 , Y i1 , Z i1 ), the two-dimensional coordinate is L i (X i2 , Y i2 , Z i2 ), calculate the Euclidean distance d between the image coordinates and the two-dimensional coordinates of the i feature points, and then get the first distance value between the two.

[0117] The distance threshold between the preset image coordinates and the two-dimensional coordinates is less than the threshold, and the calibration parameters of the corresponding camera are correct and no recalibration is required. If the distance is greater than the threshold, the calibration parameters of the corresponding camera are wrong and recalibration is required. The camera is the one to be recalibrated.

[0118] Specifically, when there are multiple cameras to be calibrated, a camera to be recalibrated needs to be further screened.

[0119] If the first distance value is less than a preset distance value threshold, determining whether there are multiple active light cameras whose first distance values ​​are greater than the preset distance threshold;

[0120] If so, sort the first distance values ​​to obtain a sorting result;

[0121] According to the sorting result, an active light camera to be recalibrated is selected from the multiple active light cameras.

[0122] Each time we recalibrate the camera, we only select the camera with the largest first distance value as the camera to be recalibrated, and then recalibrate them in sequence according to the sorting results.

[0123] 206. Create a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0124] 207. Based on the spatial coordinate system, calculate the feature point coordinates corresponding to the feature point, and use the active light camera to be recalibrated as a projection surface to calculate the projection point coordinates corresponding to the feature point coordinates;

[0125] 208. Calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0126] 209. Determine calibration parameters of the active light camera to be recalibrated according to the projection difference.

[0127] In this embodiment, it is described in detail how to fully identify cameras with incorrect calibration parameters from a multi-camera system as active light cameras to be recalibrated. Specifically, the active light rigid body obtained by modeling all active light cameras is reprojected to each active light camera, and then compared with the marked points photographed by the active light camera itself. When the distance difference between the image coordinates of the marked point and the projection point coordinates is too large, it is determined that the calibration parameters of the camera are wrong. The camera with incorrect calibration parameters can be recalibrated without affecting the normal operation of the multi-camera system.

[0128] See also Figure 3 A third embodiment of the method for calculating calibration parameters of an active light camera in an embodiment of the present invention includes:

[0129] 301. Control the active light rigid body to emit light, and trigger the active light camera to take a picture of the active light rigid body to obtain a corresponding light-emitting image;

[0130] 302. Determine, according to the luminous image, image coordinates corresponding to the feature points and an active light camera to be recalibrated in the active light cameras;

[0131] 303. Create a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0132] 304. Based on the spatial coordinate system, calculate the feature point coordinates corresponding to the feature point, and use the active light camera to be recalibrated as a projection surface to calculate the projection point coordinates corresponding to the feature point coordinates;

[0133] 305. Calculate a second distance value between the image coordinates corresponding to the active light camera to be recalibrated and the coordinates of each projection point;

[0134] 306. Sort the second distance values, and make a one-to-one correspondence between the projection point coordinates and the image coordinates corresponding to the second distance values ​​in the sorting result;

[0135] In this embodiment, the image coordinates obtained by mapping the feature points to the camera are relatively small in difference from the projection point coordinates of the feature points, and the corresponding camera is only applicable to recalibration by this method. When the difference between the two is large, this method is not applicable. Here, the second distance between each image coordinate and each projection point is calculated in turn to determine the common feature points of the two. For example, the image coordinates corresponding to the feature point A are N1 (X1, Y1, Z1), and the image coordinates corresponding to the feature point B are N2 (X2, Y2, Z2). The coordinates of the projection points received by the camera to be recalibrated include: L3 (X3, Y3, Z3) and N3 (X3, Y3, Z3). The Euclidean distances d3 and d4 between N1 (X1, Y1, Z1) and L3 (X3, Y3, Z3) and N4 (X4, Y4, Z4) are calculated respectively. If d3 is less than d4, it is determined that the projection point corresponding to L3 (X3, Y3, Z3) corresponds to the feature point corresponding to N1 (X1, Y1, Z1) and belongs to the same light source.

[0136] 307. Calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0137] 308. Determine whether the projection difference is less than a preset projection difference threshold;

[0138] In this embodiment, when the projection difference is less than the preset projection difference threshold, it means that the coordinates of the feature points captured by the camera to be recalibrated and the actual projection point coordinates of the feature points are within the processable range of values, and the calibration parameters of the camera to be recalibrated can be directly calculated according to the projection difference, including the rotation matrix and the translation matrix; otherwise, after the camera is inversely projected on the projection difference, the camera to be recalibrated is adjusted according to the new calibration parameters, and steps 303-308 are executed in a loop.

[0139] 309. If yes, perform inverse projection on the projection difference to obtain a rotation matrix and a translation matrix of the active light camera to be recalibrated;

[0140] In this embodiment, when the projection difference is less than the preset projection difference threshold, the projection difference is reversely projected onto the luminous image A captured by the camera to be recalibrated according to the projection difference, and then the repaired luminous image A is obtained. B , and then according to the repaired luminous image A B , calculate the rotation matrix and translation matrix of the area, and use the corresponding rotation matrix and translation matrix as new calibration parameters of the camera to be recalibrated to adjust the camera.

[0141] If not, the process jumps to executing the step of determining the active light camera to be recalibrated among the active light cameras.

[0142] In this embodiment, when the projection difference is greater than the preset projection difference threshold, the calibration parameters of the camera to be recalibrated are also calculated to adjust the camera, and then steps 303-308 are repeated until the projection difference is less than the preset projection difference. The current latest projection difference is reversely projected onto the luminous image A that has been repaired once or multiple times. N , calculate the rotation matrix and translation matrix of the corresponding area as the new calibration parameters of the camera to be recalibrated, so as to make the final adjustment to the camera.

[0143] In this embodiment, the matching process between the image coordinates and the projection point coordinates, as well as the recalibration process of the calibration parameters are introduced in detail; the former is mainly through calculating the two-dimensional coordinate distance difference between the image coordinates and the projection point coordinates in the camera coordinate system. When the two-dimensional coordinate distance difference between the two is within a certain range, it can be determined that they belong to the same marked point; the latter is through the projection difference between the image coordinates and the projection point coordinates corresponding to the marked point, and the calibration parameters corresponding to the camera to be recalibrated are cyclically calibrated until the projection difference between the two is less than the preset projection difference threshold, thereby improving the accuracy of the recalibrated parameters.

[0144] See also Figure 4 A fourth embodiment of the method for calculating calibration parameters of an active light camera in an embodiment of the present invention includes:

[0145] 401. Mark the active light camera to obtain a corresponding optimization number;

[0146] In this embodiment, the multi-camera system can also be cyclically optimized, where the cameras in the multi-camera system are numbered, the corresponding optimization numbers are obtained, and the numbers are written into the camera optimization table. For example, if a multi-camera system includes five active light cameras, the five light cameras are numbered respectively, the optimization numbers 1-5 are obtained, and the numbers are written into the camera optimization table A.

[0147] If a new camera is added later, the newly added camera will be marked with a new number, and the optimization numbers of other cameras will be adjusted accordingly. For example, if a new active light camera is added to the multi-camera system containing five active light cameras with optimization numbers 1-5, and its optimization order is 3, the optimization number of the newly added active light camera will be set to 3, and the original active light cameras with optimization numbers 3, 4, and 5 will be postponed to 4, 5, and 6.

[0148] 402. Every preset period, according to the optimization number, sequentially use the active light camera as the active light camera to be recalibrated;

[0149] In this embodiment, according to the working characteristics of the multi-camera system, the user can set the optimization cycle, which can be set through the human-computer interaction interface on the application terminal. It is not necessary to determine whether the calibration parameters of the camera are wrong, and adjust the calibration parameters of each camera in the multi-camera system regularly.

[0150] 403. Create a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0151] 404. Calculate the feature point coordinates corresponding to the feature point based on the spatial coordinate system, and calculate the projection point coordinates corresponding to the feature point coordinates by taking the active light camera to be recalibrated as a projection surface;

[0152] 405. Calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0153] 406. Determine calibration parameters of the active light camera to be recalibrated according to the projection difference.

[0154] In this embodiment, it is introduced that the method of the present invention can also periodically optimize the calibration parameters of the cameras in the multi-camera system by means of cyclic optimization. It only needs to set the preset cycle and the order of optimizing the cameras. In this way, the identification parameters of the cameras can be calibrated without affecting the normal operation of the multi-camera system.

[0155] The above describes the calibration parameter calculation method of the active light camera in the embodiment of the present invention. The following describes the calibration parameter calculation device of the active light camera in the embodiment of the present invention. Figure 5 , an embodiment of a calibration parameter calculation device for an active light camera in an embodiment of the present invention includes:

[0156] The photographing module 501 is used to control the active light rigid body to emit light, and trigger the active light camera to take a photo of the active light rigid body to obtain a corresponding light-emitting image;

[0157] A camera determination module 502 is used to determine the image coordinates corresponding to the feature points and the active light camera to be recalibrated in the active light camera according to the luminous image;

[0158] A creation module 503, used to create a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0159] A first calculation module 504 is used to calculate the feature point coordinates corresponding to the feature point based on the spatial coordinate system, and calculate the projection point coordinates corresponding to the feature point coordinates using the active light camera to be recalibrated as a projection surface;

[0160] A second calculation module 505 is used to calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0161] The parameter determination module 506 is used to determine the calibration parameters of the active light camera to be recalibrated according to the projection difference.

[0162] In this embodiment, the active light rigid body is controlled to emit light, and the active light camera is triggered to take a picture of the active light rigid body to obtain a light-emitting image; then the image coordinates of the feature points are determined according to the light-emitting image, and the active light camera to be recalibrated is further determined; then, according to the relative positions of other active light cameras, a control coordinate system is created to calculate the feature point coordinates of the feature points, so as to calculate the projection point coordinates of the feature points on the active light camera to be recalibrated; finally, the calibration parameters of the active light camera to be recalibrated are determined by the projection difference between the image coordinates and the projection point coordinates, so that the system can be automatically calibrated without affecting the normal operation of the multi-camera system.

[0163] See also Figure 6 Another embodiment of the calibration parameter calculation device of the active light camera in the embodiment of the present invention includes:

[0164] The photographing module 501 is used to control the active light rigid body to emit light, and trigger the active light camera to take a photo of the active light rigid body to obtain a corresponding light-emitting image;

[0165] A camera determination module 502 is used to determine the image coordinates corresponding to the feature points and the active light camera to be recalibrated in the active light camera according to the luminous image;

[0166] A creation module 503, used to create a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated;

[0167] A first calculation module 504 is used to calculate the feature point coordinates corresponding to the feature point based on the spatial coordinate system, and calculate the projection point coordinates corresponding to the feature point coordinates using the active light camera to be recalibrated as a projection surface;

[0168] A second calculation module 505 is used to calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates;

[0169] The parameter determination module 506 is used to determine the calibration parameters of the active light camera to be recalibrated according to the projection difference.

[0170] Specifically, the camera determination module 502 further includes:

[0171] An image coordinate determination unit 5021, configured to determine the image coordinates corresponding to the feature points according to the luminous image;

[0172] The three-dimensional coordinate calculation unit 5022 is used to create a three-dimensional coordinate system according to the relative positions between the active light cameras; based on the three-dimensional coordinate system, calculate the three-dimensional coordinates of each of the feature points and the two-dimensional coordinates corresponding to the three-dimensional coordinates, wherein the two-dimensional coordinates are obtained by projecting the three-dimensional coordinates on each active light camera;

[0173] The camera determination unit 5023 is used to determine the active light camera to be recalibrated in the active light cameras according to the image coordinates and the two-dimensional coordinates.

[0174] Specifically, the camera determination unit 5023 is further used for:

[0175] The distance calculation subunit 50231 is used to calculate a first distance value between the image coordinates and the two-dimensional coordinates;

[0176] The distance judging subunit 50232 is used to judge whether the first distance value is less than a preset distance value threshold;

[0177] The camera determination subunit 50233 is configured to use the corresponding active light camera as the active light camera to be recalibrated if the first distance value is less than a preset distance value threshold.

[0178] Specifically, the camera determination subunit 50233 is also used for:

[0179] If the first distance value is less than a preset distance value threshold, determining whether there are multiple active light cameras whose first distance values ​​are greater than the preset distance threshold;

[0180] If so, sort the first distance values ​​to obtain a sorting result;

[0181] According to the sorting result, an active light camera to be recalibrated is selected from the multiple active light cameras.

[0182] Specifically, the calibration parameter calculation device further includes a matching module 507, and the matching module 507 is used to:

[0183] Calculating a second distance value between the image coordinates corresponding to the active light camera to be recalibrated and the coordinates of each projection point;

[0184] The second distance values ​​are sorted, and the projection point coordinates and image coordinates corresponding to the second distance values ​​in the sorting result are matched one to one.

[0185] Specifically, the calibration parameters include a rotation matrix and a translation matrix, and the parameter determination module 506 further includes:

[0186] A projection difference judging unit 5061 is used to judge whether the projection difference is less than a preset projection difference threshold;

[0187] a parameter determination unit 5062, configured to perform a reverse projection on the projection difference to obtain a rotation matrix and a translation matrix of the active light camera to be recalibrated;

[0188] The loop unit 5063 is configured to jump to the step of determining the active light camera to be recalibrated among the active light cameras if no.

[0189] Specifically, the calibration parameter calculation device further includes a loop optimization module 508, and the loop optimization module 508 is further used for:

[0190] Marking the active light camera to obtain a corresponding optimization number;

[0191] At every preset period, the active light cameras are sequentially used as active light cameras to be recalibrated according to the optimization numbers.

[0192] In this embodiment, the active light rigid body is controlled to emit light, and the active light camera is triggered to take a picture of the active light rigid body to obtain a light-emitting image; then the image coordinates of the feature points are determined according to the light-emitting image, and the active light camera to be recalibrated is further determined; then, according to the relative positions of other active light cameras, a control coordinate system is created to calculate the feature point coordinates of the feature points, and the projection point coordinates of the feature points on the active light camera to be recalibrated are calculated; finally, the calibration parameters of the active light camera to be recalibrated are determined by the projection difference between the image coordinates and the projection point coordinates, so that the system can be automatically calibrated without affecting the normal operation of the multi-camera system. In addition, how to fully determine the camera with incorrect calibration parameters from the multi-camera system, the matching process between the image coordinates and the projection point coordinates, and the recalibration process of the calibration parameters are introduced in detail, so that the identification parameters of the camera are recalibrated without affecting the normal operation of the multi-camera system, and the accuracy of the recalibrated parameters is guaranteed at the same time. Finally, it is introduced that the method of the present invention can also periodically optimize the calibration parameters of the cameras in the multi-camera system by means of cyclic optimization. It only needs to set the preset cycle and the order of optimizing the cameras. In this way, the identification parameters of the cameras can be calibrated without affecting the normal operation of the multi-camera system.

[0193] above Figure 5 and Figure 6The calibration parameter calculation device of the active light camera in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The calibration parameter calculation device of the active light camera in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0194] Figure 7 is a schematic diagram of the structure of a calibration parameter calculation device for an active light camera provided by an embodiment of the present invention. The calibration parameter calculation device 700 for an active light camera may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 710 (for example, one or more processors) and a memory 720, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 733 or data 732. Among them, the memory 720 and the storage medium 730 may be temporary storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the calibration parameter calculation device 700 for an active light camera. Furthermore, the processor 710 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the calibration parameter calculation device 700 for an active light camera.

[0195] The calibration parameter calculation device 700 for the active light camera may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input and output interfaces 760, and / or one or more operating systems 731, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 7 The structure of the calibration parameter calculation device for an active light camera shown does not constitute a limitation on the calibration parameter calculation device for an active light camera, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0196] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the steps of the calibration parameter calculation method for the active light camera.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0198] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration parameter calculation method for an active light camera, applied to an optical motion capture system, characterized in that: The optical motion capture system comprises: a plurality of active optical rigid bodies and a plurality of active optical cameras, wherein the active optical rigid bodies have a plurality of feature points, and the calibration parameter calculation method comprises: Controlling the active light rigid body to emit light, and triggering the active light camera to take a picture of the active light rigid body to obtain a corresponding light-emitting image; Determining, according to the luminous image, image coordinates corresponding to the feature points and an active light camera to be recalibrated in the active light cameras; Creating a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated; Based on the spatial coordinate system, the feature point coordinates corresponding to the feature point are calculated, and the projection point coordinates corresponding to the feature point coordinates are calculated using the active light camera to be recalibrated as a projection surface; Calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates; Determining calibration parameters of the active light camera to be recalibrated according to the projection difference; The step of determining the image coordinates corresponding to the feature points and the active light camera to be recalibrated in the active light camera according to the luminous image comprises: Determining image coordinates corresponding to the feature points according to the luminescent image; Creating a three-dimensional coordinate system according to the relative positions of the active light cameras; Based on the three-dimensional coordinate system, calculating the three-dimensional coordinates of each of the feature points and the two-dimensional coordinates corresponding to the three-dimensional coordinates, wherein the two-dimensional coordinates are obtained by projecting the three-dimensional coordinates on each active light camera; An active light camera to be recalibrated among the active light cameras is determined according to the image coordinates and the two-dimensional coordinates.

2. The calibration parameter calculation method according to claim 1, characterized in that: The step of determining the active light camera to be recalibrated in the active light cameras according to the image coordinates and the two-dimensional coordinates comprises: Calculating a first distance value between the image coordinates and the two-dimensional coordinates; Determining whether the first distance value is less than a preset distance value threshold; If the first distance value is less than a preset distance value threshold, the corresponding active light camera is used as the active light camera to be recalibrated.

3. The calibration parameter calculation method according to claim 2, characterized in that: If the first distance value is less than a preset distance value threshold, taking the corresponding active light camera as an active light camera to be recalibrated includes: If the first distance value is less than a preset distance value threshold, determining whether there are multiple active light cameras whose first distance values ​​are greater than the preset distance threshold; If so, sort the first distance values ​​to obtain a sorting result; According to the sorting result, an active light camera to be recalibrated is selected from the multiple active light cameras.

4. The calibration parameter calculation method according to claim 1, characterized in that: Before calculating the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates, the method further includes: Calculating a second distance value between the image coordinates corresponding to the active light camera to be recalibrated and the coordinates of each projection point; The second distance values ​​are sorted, and the projection point coordinates and image coordinates corresponding to the second distance values ​​in the sorting result are matched one to one.

5. The calibration parameter calculation method according to claim 1, characterized in that: The calibration parameters include a rotation matrix and a translation matrix, and determining the calibration parameters of the active light camera to be recalibrated according to the projection difference includes: Determining whether the projection difference is less than a preset projection difference threshold; If yes, then inversely project the projection difference to obtain a rotation matrix and a translation matrix of the active light camera to be recalibrated; If not, the process jumps to executing the step of determining the active light camera to be recalibrated among the active light cameras.

6. The calibration parameter calculation method according to any one of claims 1 to 5, characterized in that: The calibration parameter calculation method also includes: Marking the active light camera to obtain a corresponding optimization number; At every preset period, the active light cameras are sequentially used as active light cameras to be recalibrated according to the optimization numbers.

7. A calibration parameter calculation device for an active light camera, characterized in that: The calibration parameter calculation device of the active light camera includes: A photographing module, used for controlling the active light rigid body to emit light, and triggering the active light camera to take a photo of the active light rigid body to obtain a corresponding luminous image; A camera determination module, used to determine the image coordinates corresponding to the feature points and the active light camera to be recalibrated in the active light camera according to the luminous image; A creation module, used for creating a spatial coordinate system according to the relative positions of active light cameras other than the active light camera to be recalibrated; A first calculation module, used to calculate the feature point coordinates corresponding to the feature point based on the spatial coordinate system, and calculate the projection point coordinates corresponding to the feature point coordinates using the active light camera to be recalibrated as a projection surface; A second calculation module is used to calculate the projection difference between the image coordinates corresponding to the active light camera to be recalibrated and the corresponding projection point coordinates; A parameter determination module, used to determine the calibration parameters of the active light camera to be recalibrated according to the projection difference; The step of determining the image coordinates corresponding to the feature points and the active light camera to be recalibrated in the active light camera according to the luminous image comprises: Determining image coordinates corresponding to the feature points according to the luminescent image; Creating a three-dimensional coordinate system according to the relative positions of the active light cameras; Based on the three-dimensional coordinate system, calculating the three-dimensional coordinates of each of the feature points and the two-dimensional coordinates corresponding to the three-dimensional coordinates, wherein the two-dimensional coordinates are obtained by projecting the three-dimensional coordinates on each active light camera; An active light camera to be recalibrated among the active light cameras is determined according to the image coordinates and the two-dimensional coordinates.

8. A calibration parameter calculation device for an active light camera, characterized in that: The calibration parameter calculation device of the active light camera includes: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected through a line; The at least one processor calls the instruction in the memory to enable the calibration parameter calculation device of the active light camera to execute the calibration parameter calculation method of the active light camera according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the calibration parameter calculation method of the active light camera according to any one of claims 1 to 6 is implemented.

Images