Multi-camera calibration method, system and device and storage medium

Abstract

The invention relates to a multi-camera calibration method, system and device and a storage medium. The multi-camera calibration method comprises the steps of obtaining first detection data of a plurality of optical cameras for an observation calibration rod waving in a view space, and further obtaining calculation internal parameters and calculation external parameters of the plurality of optical cameras through pairwise camera optimization based on the first detection data; acquiring second detection data of each optical camera for the set calibration rod under the real world coordinate system; reconstructing the set calibration rod based on an epipolar geometry principle to obtain simulation setting data, and obtaining offset data between the simulation setting data and basic data of the set calibration rod; and optimizing the calculation internal parameters and the calculation external parameters of the plurality of optical cameras based on the offset data to obtain calibration internal parameters and calibration external parameters of the plurality of optical cameras. The calibration data of the internal reference and the external reference of the optical camera can better adapt to a real world coordinate system, and the accuracy is higher.

Description

Technical Field

[0001] The present invention relates to the field of optical camera control, and particularly to a multi-camera calibration method, system, device and storage medium. Background Art

[0002] Motion capture technology has been fully applied to related industries such as film and television advertising and game production. Among them, a multi-view system composed of multiple infrared cameras is a basic component of optical motion capture technology, which provides high-precision support for related applications of motion capture. The high-precision positioning of the multi-camera system depends on the accuracy and stability of multi-camera calibration. Therefore, multi-camera calibration is an important prerequisite for the stable and effective operation of the multi-camera system.

[0003] The essence of camera calibration is to calculate the internal and external parameters of the camera model, and the purpose is to restore the three-dimensional shape of the object from the two-dimensional image of the object. The internal parameters of the camera are the inherent properties of the camera and generally do not change after the camera leaves the factory. However, photosensitive components such as sensors will change slightly due to environmental temperature and humidity. The external parameters of the camera are the rotation and translation of the camera coordinate system relative to the world coordinate system. After years of development, calibration methods such as three-dimensional calibration, two-dimensional calibration, one-dimensional calibration and self-calibration have also emerged.

[0004] The essential difference between various calibration methods is only the selection of the calibration object. The calibration object used in the three-dimensional calibration method is a three-dimensional calibration block with known geometric information. Essentially, it calculates the internal and external parameters of the camera through the correspondence between the geometric information on the calibration object and the image information. Since the three-dimensional calibration object is generally composed of several intersecting planes and can provide geometric information in different dimensions, theoretically, three-dimensional calibration can achieve higher calibration accuracy, but the calibration manufacturing is more troublesome. The two-dimensional calibration method uses a two-dimensional calibration board for calibration, and the more classic one is the Zhang's calibration method. During the calibration process, the camera takes pictures of the calibration board at different angles, then extracts feature points, establishes a homography relationship, and calculates the internal and external parameters of the camera. Compared with two-dimensional calibration, one-dimensional calibration is more suitable for multi-camera calibration systems because it is not easily blocked. The one-dimensional calibration object is generally composed of more than two small balls fixed on a rod, and the geometric distance between the balls is known. Then wave the rod under the multi-view system and capture multiple images at the same time, and complete the calibration of the camera according to the correspondence between the small balls on the calibration rod and the feature points on the image. Self-calibration realizes camera calibration only relying on the information of the image itself.

[0005] In the existing multi-camera calibration system, the data collection time is relatively long, and the calibration object is relatively heavy, sacrificing a certain degree of simplicity for the sake of calibration accuracy. Summary of the Invention

[0006] In view of the deficiencies of the above-mentioned prior art, one of the objectives of the present invention is to provide a multi-camera calibration method that is both simple and highly accurate, and solve the problem of simultaneous calibration of multiple infrared cameras in a large-space environment of optical motion capture.

[0007] Another objective of the present invention is to provide a multi-camera calibration system.

[0008] A third objective of the present invention is to provide an electronic device.

[0009] A fourth objective of the present invention is to provide a computer-readable storage medium.

[0010] To achieve the above objectives, the present invention adopts the following technical solutions:

[0011] On the one hand, the present invention provides a multi-camera calibration method, including:

[0012] Obtaining first detection data of a plurality of optical cameras for an observation calibration rod waved in a view space, and then obtaining calculation internal parameters and calculation external parameters of the plurality of optical cameras through pairwise camera optimization based on the first detection data;

[0013] Obtaining second detection data of each optical camera for a set calibration rod in a real-world coordinate system;

[0014] Based on the calculation internal parameters, calculation external parameters of the plurality of optical cameras, and the pose information of each optical camera, reconstructing the set calibration rod based on the second detection data according to the epipolar geometry principle to obtain simulated set data, and obtaining offset data between the simulated set data and the basic data of the set calibration rod;

[0015] Optimizing the calculation internal parameters and calculation external parameters of the plurality of optical cameras based on the offset data to obtain calibration internal parameters and calibration external parameters of the plurality of optical cameras.

[0016] Preferably, the observation calibration rod has a plurality of first marking points;

[0017] Obtaining first detection data of a plurality of optical cameras for an observation calibration rod waved in a view space, and then obtaining calculation internal parameters and calculation external parameters of the plurality of optical cameras based on the first detection data specifically includes:

[0018] Acquire image data of a plurality of first calibration points on a swinging observation calibration rod collected by a plurality of infrared cameras; perform image preprocessing on the image data to obtain calibration data; the calibration data includes frame data related to a plurality of detection times; the frame data includes frame label data of image data of all infrared cameras at the same detection time; the frame label data includes one or more sub-label data in a frame of image data; the sub-label data includes coordinate data of a first calibration point and an identification code of a corresponding optical camera;

[0019] Performing data preprocessing on the calibration data based on the basic parameters of the observed calibration rod, eliminating invalid data and retaining valid data;

[0020] The optical camera with the most valid data is used as the main camera, and the other optical cameras to be optimized are set as slave cameras; the coordinate system of the main camera is used as the world coordinate system, and the external parameters of the main camera are set as unit external parameters, so that multiple slave cameras are optimized in pairs with the main camera respectively, and the calculated external parameters and calculated internal parameters of all slave cameras are obtained.

[0021] Preferably, the data preprocessing includes:

[0022] Perform interference light source detection on all the frame mark data in turn, construct a corresponding detection area, and remove the sub-mark data located in the detection area;

[0023] Eliminate the frame mark data whose number of sub-mark data is not equal to the number of first mark points;

[0024] Performing morphological detection on the sub-label data in all the frame label data in turn, and removing the frame label data that fails the morphological detection;

[0025] Performing static detection on a plurality of the frame mark data in the same frame data, and removing duplicate frame mark data;

[0026] The frame data whose number of frame mark data is less than a predetermined value is eliminated.

[0027] Preferably, the step of performing pairwise camera optimization on the slave cameras and the master camera comprises:

[0028] Acquire the original internal parameters of the master camera and the slave camera as the calculation internal parameters, and obtain the internal parameter matrix by initializing the original internal parameters of the optical camera;

[0029] Based on the coordinate data detected by the master camera and the slave camera for the same first calibration point, a basic matrix is ​​obtained by a random sampling consistency algorithm, and then the intrinsic matrix is ​​obtained by combining the intrinsic parameter matrices of the master camera and the slave camera;

[0030] Performing singular value decomposition on the essential matrix to obtain the calculated external parameters, where the calculated external parameters include a rotation matrix and a translation vector.

[0031] Preferably, the coordinate data is centroid coordinates; the calculation formula for the centroid coordinates is:

[0032]

[0033] where x is the x-axis coordinate of the centroid coordinates; y is the y-axis coordinate of the centroid coordinates; g(i,j) represents the pixel value at the coordinate (i,j) on the image, i represents the value on the x-axis of the two-dimensional coordinate of the pixel point, and j represents the value on the y-axis of the two-dimensional coordinate of the pixel point.

[0034] Preferably, after obtaining the calculated external parameters and the calculated internal parameters, it further includes:

[0035] Combining the calculated external parameters and the calculated internal parameters of the main camera, as well as the calculated external parameters and the calculated internal parameters of the slave camera, with the corresponding calibration data for iterative optimization. The loss function in the iterative optimization process is the reprojection error, and the optimized calculated internal parameters and the calculated external parameters of the main camera, as well as the optimized calculated external parameters and the calculated internal parameters of the slave camera, are obtained.

[0036] Preferably, the observation calibration rod includes a plurality of first calibration points, and the plurality of first calibration points are distributed in a predetermined form on the observation calibration rod;

[0037] After optimizing the calculated internal parameters and the calculated external parameters of multiple optical cameras, it further includes:

[0038] Based on the calculated internal parameters and the calculated external parameters of each optical camera and the corresponding first detection data, constructing three-dimensional coordinate data of the plurality of first calibration points on the observation calibration rod;

[0039] If the three-dimensional coordinate data of the plurality of first calibration points corresponding to any one of the optical cameras does not conform to the predetermined form, then re-optimize this optical camera pairwise with the main camera, and recalculate the calculated internal parameters and the calculated external parameters of this optical camera.

[0040] Preferably, the observation calibration rod is a one-dimensional calibration rod, having a plurality of first calibration points, and the plurality of first calibration points are on the same straight line, and the distances between the first calibration points are not equal.

[0041] Preferably, the set calibration rod is an L-shaped calibration rod, having a plurality of second calibration points, which are respectively arranged on the two right-angle arms of the L-shaped calibration rod.

[0042] Preferably, the second detection data includes two-dimensional coordinate data of the plurality of second calibration points detected by one of the optical cameras;

[0043] Based on the computed intrinsic parameters, computed extrinsic parameters, and pose information of each of the multiple optical cameras, and combining the second detection data, the simulated set data is obtained by reconstructing the set calibration rod based on the epipolar geometry principle, and the offset data between the simulated set data and the basic data of the set calibration rod is obtained. Specifically, it includes:

[0044] Based on the pose information of each optical camera and the second detection data, a two-dimensional point matching result is obtained through the epipolar geometry principle;

[0045] Based on the computed intrinsic parameters, computed extrinsic parameters of the multiple optical cameras, and the two-dimensional point matching result, three-dimensional reconstruction is performed through triangulation calculation to obtain the three-dimensional coordinate data of multiple second calibration points on the set calibration rod;

[0046] The three-dimensional coordinate data is optimized through clustering processing and reprojection error;

[0047] According to the plane data of the set calibration rod reconstructed in three dimensions, the short side is used as the x-axis of the coordinate system, the long side is used as the y-axis of the coordinate system, the normal vector perpendicular to this plane is obtained and used as the z-axis of the coordinate system, and then the reconstructed pose information of the set calibration rod is calculated, and further the offset data between the reconstructed pose information and the basic data of the set calibration rod is obtained.

[0048] Preferably, based on the offset data, the computed intrinsic parameters and computed extrinsic parameters of the multiple optical cameras are optimized to obtain the calibrated intrinsic parameters and calibrated extrinsic parameters of the multiple optical cameras. Specifically, it includes:

[0049] Based on the offset data, the rotation and translation relationships between the coordinate system of the master camera in the pairwise camera optimization and the real-world coordinate system are obtained, and then the calibrated intrinsic parameters and calibrated extrinsic parameters of the master camera are obtained;

[0050] Based on the calibrated intrinsic parameters and calibrated extrinsic parameters of the master camera, the calibrated intrinsic parameters and calibrated extrinsic parameters of other slave cameras are obtained through pairwise camera optimization.

[0051] On the other hand, the present invention provides a multi-camera calibration system, including:

[0052] A primary processing module for obtaining first detection data of a plurality of optical cameras for an observed calibration rod waved in the view space, and then based on the first detection data, obtaining the computed intrinsic parameters and computed extrinsic parameters of the plurality of optical cameras through pairwise camera optimization;

[0053] The secondary processing module is used to obtain the second detection data of each optical camera for the set calibration rod in the real-world coordinate system; based on the calculation internal parameters, calculation external parameters and pose information of each of the multiple optical cameras, combined with the second detection data, the set calibration rod is reconstructed based on the epipolar geometry principle to obtain simulated set data, and the offset data between the simulated set data and the basic data of the set calibration rod is obtained; based on the offset data, the calculation internal parameters and calculation external parameters of the multiple optical cameras are optimized to obtain the calibrated internal parameters and calibrated external parameters of the multiple optical cameras.

[0054] On the other hand, the present invention provides an electronic device, including:

[0055] A memory storing a computer program;

[0056] A processor, when executing the computer program, implements the multi-camera calibration method described above.

[0057] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the multi-camera calibration method described above is implemented.

[0058] Compared with the prior art, a multi-camera calibration method, system, device and storage medium provided by the present invention have the following beneficial effects:

[0059] Based on the detection of the waved observation calibration rod to obtain the first detection data, and then after obtaining the calculation internal parameters and calculation external parameters of all the optical cameras through pairwise camera optimization, through ground alignment, the second detection data of the static set calibration rod is detected, the set calibration rod is reconstructed based on the epipolar geometry principle, and then the calculation internal parameters and calculation external parameters are optimized to obtain the final calibrated internal parameters and calibrated external parameters, so that the calibrated data of the internal parameters and external parameters of the optical camera are more adaptable to the real-world coordinate system and have higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flowchart of the multi-camera calibration method provided by the present invention.

[0061] Figure 2 is a schematic diagram of the observation calibration rod provided by the present invention.

[0062] Figure 3 is a flowchart of an embodiment of the multi-camera calibration method provided by the present invention.

[0063] Figure 4 is a schematic diagram of the set calibration rod provided by the present invention.

[0064] Figure 5It is a structural block diagram of an implementation manner of the multi-camera system provided by the present invention. Specific implementation manner

[0065] To make the objectives, technical solutions and effects of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0066] Those skilled in the art should understand that the foregoing general description and the following detailed description are exemplary and illustrative specific embodiments of the present invention and are not intended to limit the present invention.

[0067] As used herein, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process or method including a list of steps includes not only those steps but also other steps that are not explicitly listed or are inherent to such process or method. Throughout the specification, the occurrences of the phrases "in one embodiment", "in another embodiment" and similar language may or may not all refer to the same embodiment.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0069] Please participate in Figures 1 - 5 The present invention provides a multi-camera calibration method, which is applied to a multi-camera system. The multi-camera system includes a plurality of optical cameras and a management device respectively connected to the plurality of infrared cameras for data connection. The management device includes a computer, a server, a smart mobile device, and a dedicated smart processing device. The smart processing device may be an FPGA (Field-Programmable Gate Array, field programmable logic gate array controller). The optical camera is preferably an infrared camera.

[0070] In some embodiments, data transmission between the infrared camera and the management device is performed by wired or wireless means. The wired means includes USB data transmission, optical fiber data transmission, etc., and the wireless means includes WiFi communication, 4G communication, 5G communication, etc. There is a communication interface with the corresponding communication means between the infrared camera and the management device, which will not be elaborated herein.

[0071] It can be understood that in addition to the above structures and connection relationships, the infrared camera and the management device may further include other structures or modules with additional functions, or combine certain components, or arrange different components.

[0072] The present invention provides a multi-camera calibration method, including:

[0073] S1. Obtain the first detection data of a plurality of optical cameras for the observed calibration rod waved in the view space, and then obtain the calculated internal parameters and calculated external parameters of the plurality of optical cameras through pairwise camera optimization based on the first detection data; in this embodiment, the pairwise camera optimization adopts the conventional operation process for obtaining the calculated internal parameters and calculated external parameters in the art. Of course, it cannot be limited to the conventional operations before this application. In some embodiments, the main purpose of the first step is to obtain the calculated internal parameters and calculated external parameters of all optical cameras obtained by the first calculation through pairwise camera optimization.

[0074] Further, in some embodiments, the main purpose is to obtain the calculated internal parameters and calculated external parameters with the coordinate system of the main camera in the pairwise camera optimization as the reference system.

[0075] In some embodiments, the observed calibration rod is a two-dimensional calibration rod or a one-dimensional calibration rod, adopting the conventional form in the art. Further, the optimization scheme in the art is to use a one-dimensional calibration rod as the observed calibration rod.

[0076] S2. Obtain the second detection data of each optical camera for the set calibration rod in the real-world coordinate system;

[0077] S3. Based on the calculated internal parameters, calculated external parameters of the plurality of optical cameras and the pose information of each optical camera, combine the second detection data to reconstruct the set calibration rod based on the epipolar geometry principle to obtain simulated set data, and obtain the offset data between the simulated set data and the basic data of the set calibration rod;

[0078] S4. Optimize the calculated internal parameters and calculated external parameters of the plurality of optical cameras based on the offset data to obtain the calibrated internal parameters and calibrated external parameters of the plurality of optical cameras.

[0079] Based on the detection of the waved observed calibration rod to obtain the first detection data, and then through pairwise camera optimization to obtain the calculated internal parameters and calculated external parameters of all optical cameras. After that, through ground alignment, detect the static set calibration rod to obtain the second detection data, reconstruct the set calibration rod based on the epipolar geometry principle, and then optimize the calculated internal parameters and calculated external parameters to obtain the final calibrated internal parameters and calibrated external parameters, making the calibration data of the internal parameters and external parameters of the optical camera more adaptable to the real-world coordinate system and having higher accuracy.

[0080] Further, as a preferred solution, the observed calibration rod has a plurality of first marking points;

[0081] Obtaining the first detection data of a plurality of optical cameras for the observed calibration rod waved in the view space, and then obtaining the calculated internal parameters and calculated external parameters of the plurality of optical cameras based on the first detection data specifically includes:

[0082] S11. Obtain the image data of multiple first calibration points on the waved observation calibration rod collected by multiple infrared cameras; perform image preprocessing on the image data to obtain calibration data; the calibration data includes multiple frame data related to detection time; the frame data is the frame label data of the image data of all infrared cameras at the same detection time; the frame label data includes one or more sub-label data in a frame of image data; the sub-label data includes the coordinate data of a first calibration point and the identification code of the corresponding optical camera; the identification code can be the unique identification code of the optical camera or the serial number of the optical camera.

[0083] Data acquisition step; within the effective vision space range of the multi-camera system, continuously wave a one-dimensional calibration rod, and multiple infrared cameras in the multi-camera system acquire the image data of the one-dimensional calibration rod within the calibration time period and transmit the image data to the management device.

[0084] There are multiple marking points on the one-dimensional calibration rod. In some embodiments, the marking points are reflective balls and the number is 3. In some embodiments, the one-dimensional calibration rod is as Figure 2 shown, the 3 marking points are on the same straight line, and the distances between adjacent two marking points are not equal. For example, Figure 2 the distance between the first two first marking points from left to right is 175 mm, and the distance between the last two first marking points is 325 mm.

[0085] In this embodiment, after the management device receives the image data transmitted by all infrared cameras, it performs image preprocessing on all the image data at the same moment to obtain the frame data related to the same moment; the frame data includes multiple frame label data corresponding to multiple infrared cameras respectively; the frame label data includes all sub-label data related to a frame of image data; the sub-label data includes the coordinate data of a light spot and the device serial number of the corresponding infrared camera.

[0086] On the FPGA, mainly perform preprocessing on the collected image data and calculate the centroid of the light spots (reflective balls) in the image. The image data preprocessing is mainly to prevent the influence of the centroid calculated from the light spots blocked by occlusion or affected by noise on the actual camera calibration.

[0087] Preferably, the observation calibration rod is a one-dimensional calibration object with 3 infrared reflective balls on the rod, and the distances between the reflective balls are not equal. Obtain the serial number of each camera, wave the one-dimensional calibration rod in the multi-camera view space, and fully collect the coordinate data of the reflective balls on the one-dimensional calibration rod corresponding to each infrared camera at the same moment.

[0088] Specifically, the image preprocessing includes:

[0089] S111. Obtain the coordinate data of each light spot in each frame of image data;

[0090] S112. Associate the coordinate data of each light spot with the device serial number of the corresponding infrared camera to obtain sub-label data;

[0091] S113. Aggregate all the sub-label data in the same frame of image data to obtain frame-label data;

[0092] S114. Aggregate the frame-label data of all infrared cameras at the same moment to obtain frame data;

[0093] S115. Integrate all the frame data within the calibration time period to obtain a calibration data set.

[0094] That is, in order to better save the data for calibration, these data need to be saved in a fixed standard data format. In this step, the data format takes synchronization as the first priority. Save the same frame-label data into a vector, denoted as frame data Frame; each frame-label data contains a corresponding frame of image data. Due to the different poses of the optical cameras, the shooting angles for observing the calibration rod are different, which may result in not being able to capture multiple first identification points on the calibration rod simultaneously. In addition, the one-dimensional calibration rod swings within the view spaces of all optical cameras, that is, any optical camera can capture the calibration rod, and there is no situation where it cannot be captured. Therefore, there may be one or more first identification point images on a frame of image data, including one or more sub-label data on this frame of image. The sub-label data includes the identification code Camera_id of the optical camera for indexing and the coordinate data Points of the corresponding first identification point; Combine all the frame data Frame together to form the calibration data Frames for calibration.

[0095] In some embodiments, after the management device obtains the image data transmitted by the infrared camera, it actively obtains the device serial number of the corresponding infrared camera according to the data interface for associated storage.

[0096] In some embodiments, when the infrared camera transmits the image data to the management device, it synchronously transmits its own device serial number to facilitate the associated storage of the coordinate data.

[0097] In some embodiments, the coordinate data is the centroid coordinates of the reflective sphere, and the calculation formula for the centroid coordinates is:

[0098]

[0099] Among them, x is the x-axis coordinate of the centroid coordinate; y is the y-axis coordinate of the centroid coordinate; g(i,j) represents the pixel value with coordinates (i,j) on the image, i represents the value of the pixel point on the x-axis of the two-dimensional coordinate, and j represents the value of the pixel point on the y-axis of the two-dimensional coordinate.

[0100] S12, performing data preprocessing on the calibration data based on the basic parameters of the observed calibration rod, eliminating invalid data and retaining valid data;

[0101] In order to obtain better calibration results, the data needs to be preprocessed. Some erroneous, repeated and invalid data are discarded, and accurate valid data at different spatial locations are retained. Further, in some embodiments, the data preprocessing includes:

[0102] S121. Perform interference light source detection on all the frame mark data in turn, construct a corresponding detection area, and remove the sub-mark data located in the detection area; this step mainly performs interference light source detection on the spatial field of the multi-camera system, constructs a corresponding Mask area; and removes the coordinate data located in the Mask area. In the spatial field of the multi-camera system, there may be some interference light sources, causing some cameras to output some interfering center of mass coordinates. In order to eliminate the interference of these noise light sources, the interference light source detection is performed in this step, and the detected area is called the Mask area. The Mask area is composed of a rectangular block composed of the upper left corner point and the lower right corner point. The center of mass data of all cameras in each frame are traversed, and the center of mass coordinates located in the rectangular block are removed. Of course, the identification code associated with the center of mass coordinates is deleted synchronously.

[0103] S122, remove the frame mark data whose number of sub-mark data is not equal to the number of the first calibration points; for the data group at the same time, if the one-dimensional calibration rod is waved randomly, so that the centroid data recorded by the camera at some times does not meet the geometric rules of the one-dimensional calibration rod, it is necessary to remove these erroneous data. First, for each frame of data Frame in the calibration data Frames, check the centroid data Points under all identification codes Carmera_id. If the number of first calibration points on the one-dimensional calibration rod is 3, if the data dimension of the centroid data is not equal to 3, it means that the data obtained by the camera in this frame does not constitute a one-dimensional calibration rod, or there may be some interfering noise points. For stability considerations, the frame mark data whose centroid data dimension is not equal to 3 in a frame of data Frame can be removed.

[0104] S123. Morphologically detect the sub-label data in all the frame label data in sequence, and eliminate the frame label data that fails the morphological detection. Specifically, if the observation calibration rod is a one-dimensional calibration rod and conforms to the morphology in the foregoing embodiments, in this step, for the frame label data with 3 centroid coordinates detected, it is necessary to determine that these 3 points are the 3 first calibration points shown on the one-dimensional calibration rod. Therefore, in this step, it is necessary to perform a line detection on the coordinate data of these 3 existing points to determine that these three points are on a straight line, which conforms to the spatial geometric meaning of the one-dimensional calibration rod. In a frame data Frame, eliminate the frame label data of the optical camera that fails the line detection, and retain the data that passes the line detection for data screening in the next step. If the observation calibration is a two-dimensional calibration rod (such as a T-shaped calibration rod), the specific morphology of the two-dimensional calibration rod needs to be satisfied, which will not be elaborated here.

[0105] S124. Perform static detection on multiple frame label data in the same frame data, and eliminate the duplicate frame label data. Specifically, a set of excellent optimization data should not only be accurate and effective, but also diverse, conforming to the general nature of optimization and avoiding the optimization falling into a local optimal solution. Therefore, after the above steps of screening the validity of the original data, diverse data should also be selected, that is, the data selected for calibration calculation optimization should be distributed in all directions of the multi-camera field of view space, rather than being limited to one place only. For this reason, while ensuring the uniformity of the human sweep field, a static detection method is also applied. Static detection is to compare whether there is a movement in the coordinate data of the current frame label data relative to the coordinate data of the adjacent frame label data. If there is a movement, the current frame label data is retained; if there is no movement, that is, there is no deviation in the centroid within a certain threshold, the current frame label data is eliminated.

[0106] S125. Eliminate the frame data with the number of frame label data less than a predetermined value. Further screen the data according to the principle of binocular camera calibration. Since the camera is calibrated based on binocular geometry, in the same frame data Frame, at least two cameras need to observe the real coordinate data of the one-dimensional calibration rod simultaneously for this frame data to be used for subsequent camera calibration. To make the data more rigorous, such an operation is performed in this step. Check that the number of frame label data in each frame data Frame is greater than or equal to 3, which means that the one-dimensional calibration rod can be observed by 3 or more optical cameras simultaneously, and record the frame data Frame into the final valid data group Frames; if not satisfied, eliminate this group of frame data Frame.

[0107] S13. Take the optical camera with the most valid data as the master camera, and set the other optical cameras to be optimized as slave cameras; use the coordinate system of the master camera as the world coordinate system, and set the external parameters of the master camera as the unit external parameters, so that each of the multiple slave cameras is optimized with the master camera pairwise to obtain the calculated external parameters and calculated internal parameters of all slave cameras. The calibration of a multi-camera system needs to calculate the external parameters of the cameras, that is, the rotation and translation of the cameras relative to a certain coordinate system. Therefore, in a multi-camera calibration system, a master camera needs to be determined, and the rotation and translation of other cameras (i.e., slave cameras) relative to this master camera need to be calculated. That is, the external parameters of the slave cameras are calculated with the coordinate system of the master camera as the reference system.

[0108] Specifically, binocular calibration calculates the relative pose between camera pairs, which is the pose of one camera relative to another camera. Therefore, a master camera needs to be determined. Based on this master camera, the poses of other cameras relative to this master camera can be calculated, and then the relative spatial poses of all cameras can be obtained. In multi-camera calibration, the selection of the master camera (also known as the reference camera or benchmark camera) is very crucial because it will affect the accuracy and stability of the entire calibration result. For better calibration accuracy and stability, a camera with a wide viewing angle coverage and a large overlapping area with other cameras should be selected as the master camera, that is, a camera with the most valid observation data and a large amount of data simultaneously observed with other cameras should be selected as the master camera. Set the rotation matrix of the master camera as the identity matrix and the translation vector as the zero vector, that is, set the coordinate system of the selected master camera as the world coordinate system, and the poses of other cameras are all relative to the master camera.

[0109] It can be understood that "the most valid data" can be understood as the largest number of retained identification codes or the largest number of frame label data among the remaining data.

[0110] Since the initial camera internal parameters are set according to the factory-built-in parameters of the camera, they are not very accurate in an actual multi-camera system and are prone to falling into the problem of local optimum in subsequent global optimization. Therefore, pairwise optimization between cameras is required during the calibration process to obtain relatively accurate internal and external parameters of the optical camera. This process first determines a main camera and regards the camera coordinate system of the main camera as the world coordinate system, that is, the external parameters of the main camera, with the rotation matrix set as the identity matrix and the translation vector set as the zero vector. Then, based on the corresponding matching point data of the collected camera pairs, the fundamental matrix F and the essential matrix E of the camera pair (main camera and slave camera) are calculated using the Random Sample Consensus (RANSAC) algorithm, and the relative pose of the camera pair, that is, the rotation matrix and translation vector of the slave camera relative to the main camera, is solved using the Singular Value Decomposition (SVD). According to the calculated relative pose and triangulation, the 3D point coordinates of the first calibration point in space are obtained, and the validity of the 3D point coordinates is verified based on the depth information. At the same time, according to the distance between two points on the one-dimensional calibration rod, the absolute scale ratio between the camera space coordinate system and the real space coordinate system is calculated. Finally, the internal and external parameters of the camera are optimized through the reprojection error.

[0111] Further, as an optimal solution, the pairwise camera optimization of the slave camera and the main camera specifically includes:

[0112] S131. Obtain the original internal parameters of the main camera and the slave camera as the calculated internal parameters, and initialize the original internal parameters of the optical camera to obtain the internal parameter matrix;

[0113] First, the initial internal parameters of the optical camera system need to be set. For optical camera calibration, it is mainly to optimize the internal and external parameters of the camera in the actual space. Among them, the internal parameters of the camera mainly include the focal length f, the coordinates of the principal point (cx, cy), and the distortion parameters; the external parameters of the camera mainly include the rotation matrix and the translation vector.

[0114] Since the image sensors commonly used in modern optical cameras are not always completely symmetric and may have different physical sizes in the x and y directions, the focal length f is defined as fx and fy. To efficiently optimize the internal and external parameters of the optical camera, the internal parameters of the optical camera system need to be initialized.

[0115] For the focal lengths fx and fy of each optical camera, an initial focal length parameter is given according to the focal length specification and pixel size of the optical camera at the factory. Among them, fx represents the focal length in the horizontal direction, and fy represents the focal length in the vertical direction. The calculation formula is as follows:

[0116] fx = f / ux;

[0117] fy = f / uy;

[0118] Among them, ux and uy represent the pixel size, with the unit of um / pixel.

[0119] For the principal point parameters cx and cy of each camera, according to the width W and height H of the imaging, calculate the principal point parameters. The calculation formula is as follows:

[0120] cx = W / 2;

[0121] cy = H / 2;

[0122] Generally speaking, the above internal parameters can form an internal parameter matrix K, expressed as K = [fx, 0, cx; 0, fy, cy; 0, 0, 1].

[0123] S132. Obtain the fundamental matrix through the Random Sample Consensus (RANSAC) algorithm based on the coordinate data detected by the main camera and the slave camera for the same first calibration point, and then combine the internal parameter matrices of the main camera and the slave camera to obtain the essential matrix.

[0124] The calculation formula is expressed as follows:

[0125] Denote each pair of points as (x1, x2), where x1 and x2 are the centroid coordinates of the same light sphere projected onto the main camera and the slave camera at the same moment. According to the point pair relationship, calculate the fundamental matrix F:

[0126] x T 2 * F * x1 = 0;

[0127] According to the known internal parameter matrix K1 of the main camera and the internal parameter matrix K2 of camera A, calculate their essential matrix E:

[0128] E = K T 2 * F * K1;

[0129] S133. Perform singular value decomposition on the essential matrix to obtain the calculated external parameters, and the calculated external parameters include the rotation matrix and the translation vector.

[0130] After determining the main camera, it is respectively matched with multiple slave cameras to form camera pairs, and then based on the camera pair matching data point set, the fundamental matrix and essential matrix of the corresponding camera pair are calculated. The camera pair matching data point set includes the point pair data of multiple first calibration points, and the point pair data has identity and uniqueness. Therefore, before calculating the fundamental matrix, it is necessary to rearrange and match the two-dimensional point data of the same frame of data of the camera pair to ensure that the matched two-dimensional points are the projection points of the same three-dimensional point on the main camera and the slave camera. Then, in order to exclude the interference of some noise points on the calculation of the fundamental matrix, the RANSAC algorithm is used in this step to calculate and optimize the fundamental matrix F of the camera pair. Then, according to the internal parameter matrix K1 of the main camera and the internal parameter matrix K2 of the slave camera, use K T 2 *F*K 1 to calculate the essential matrix E of the camera pair. Finally, by decomposing the essential matrix E through the SVD decomposition method, the rotation information R and translation information T of the target camera can be obtained. Specifically, the essential matrix is a 3*3 matrix and can be decomposed into the form of UΣV T . Among them, U and V are orthogonal matrices, and Σ is a diagonal matrix. Define a rotation base matrix W = [0, -1, 0; 1, 0, 0; 0, 0, 1], and calculate the possible rotations R1 = UWV T and R2 = UW T V T , and calculate the rotation matrix and translation vector between the camera pairs that conform to the real scene through the point pair geometric relationship.

[0131] Furthermore, as a preferred solution, after obtaining the calculated external parameters and calculated internal parameters, it further includes:

[0132] Combining the calculated external parameters and calculated internal parameters of the main camera, as well as the calculated external parameters and calculated internal parameters of the slave camera with the corresponding calibration data for iterative optimization. The loss function in the iterative optimization process is the reprojection error, and the optimized calculated internal parameters and calculated external parameters of the main camera, as well as the optimized calculated external parameters and calculated internal parameters of the slave camera are obtained.

[0133] Specifically, given the initial internal parameters of the main camera and the external parameter data of the secondary camera calculated, the coordinate representation of the corresponding 2D image points of the camera in 3D space can be calculated by triangulation. First, the 2D coordinate points in the image coordinate system are converted from the image coordinate system to the normalized points in the camera coordinate system through de-distortion and internal parameter conversion. Then, through the rotation matrix and translation vector, the points are converted to the points in the world coordinate system, that is, the points in the main camera coordinate system. Then, check whether the depth information of the generated 3D points is reasonable, that is, it should be greater than 0; if not, discard it, and if so, save it as the valid 3D point set of the current frame. It should be noted that the number of valid 3D point sets in one frame must be equal to 3. With 3 3D points corresponding to the reflective spheres on the 1D calibration rod, the ratio scale of the spatial distance in the camera coordinate system to the actual spatial distance can be calculated according to the distance between the 3D points and the fixed reflective spheres on the 1D calibration rod pairwise. The value of the translation vector is converted from the size of the camera coordinate system to the actual spatial scale, that is, T = T * scale.

[0134] Finally, according to the reprojection error and the geometric property constraints of 1D calibration, the focal length, distortion coefficient, and external parameters of the camera are optimized. Specifically, first, the reconstructed rod is represented in the way of centering and vector normalization. That is, originally there are three 3D points representing the rod, which are represented by a center point coordinate mid_point and a normalized vector line representing the direction of the rod, and these are incorporated into the optimized parameter list. Optimizing these two parameters instead of optimizing the three 3D point coordinates can improve the optimization speed. Calculate the 3D coordinate points on the 1D calibration rod with these two vectors respectively, and according to the projection matrix, calculate the reprojection error of these three points on the camera, denoted as error. The specific optimization process is as follows:

[0135] According to mid_point and line_normlinze, where the short side point short_point and long side point long_point of the 1D calibration rod can be calculated according to the geometric properties of the rod:

[0136] short_point = mid_point - short_side_distance * line_normlinze;

[0137] long_point = mid_point + long_side_distance * line_normlinze;

[0138] In the above formula, short_side_distance and long_side_distance represent the short side distance and long side distance of the 1D calibration rod respectively.

[0139] Due to certain errors in the internal parameters of the initially set camera pair, the calculated rotation matrix and translation vector will also have errors. Therefore, in order to obtain more accurate internal and external camera parameters, it is necessary to iteratively optimize the parameters of the camera pair according to the reprojection error of the 3D point coordinates reconstructed in three dimensions on the camera pair. Before optimizing the parameters of the camera pair, the system needs to perform a depth detection on the point pair data, and only record the data with qualified depth information into the optimized data. The specific process of this optimization is as follows:

[0140] First, according to the rotation matrix R and translation vector T between the camera pair obtained by solving, the projection matrices of the two cameras can be obtained, denoted as P1 and P2 respectively. The basic expression of the projection matrix is:

[0141] P = K[R|T];

[0142] Among them, K represents the internal parameter matrix of the camera; [R|T] represents the external parameter matrix of the camera.

[0143] Secondly, according to the projection matrices P1 and P2 of the camera pair, the coordinates of the 3D points in the corresponding space are calculated, and the formula is expressed as follows:

[0144] x1 = P1*X;

[0145] x2 = P2*X;

[0146] Then, according to the calculated 3D point coordinates, verify the depth information to check whether it is a valid 3D point. If it is a valid 3D point, project it onto the image coordinate system, calculate the deviation between its observed coordinates, and use the cumulative error as the loss function to optimize the internal and external parameters of the camera. The specific explanation process of this process is as follows:

[0147] According to the rotation matrix R and translation vector T, convert the world coordinate p’ to the camera coordinate system:

[0148] p = R*p’ + T = {x, y, z};

[0149] Normalize the coordinate p in the camera coordinate system to obtain the coordinate pn = {xn, yn, 1} = {x / z, y / z, 1} on the normalized camera plane;

[0150] Apply the distortion model of the camera to the normalized coordinate pn to obtain the undistorted normalized coordinate pn’ = {xn’, yn’}:

[0151] Radial distortion, the radial distortion parameters are k1, k2, k3:

[0152] xn’ = xn*(1 + k1*r^2 + k2*r^4 + k3*r^6);

[0153] yn'=yn*(1+k1*r^2+k2*r^4+k3*r^6);

[0154] Tangential distortion, the tangential distortion parameters are p1 and p2:

[0155] xn'=xn'+[2*p1*xn*yn+p2*(r^2+2*xn^2)];

[0156] yn'=yn'+[p1*(r^2+2*yn^2)+2*p2*xn*yn];

[0157] Map the dedistorted normalized coordinates to the image coordinate system, denoted as pi = (u, v):

[0158] u=fx*xn'+cx;

[0159] v = fy*yn'+cy;

[0160] Finally, calculate the error between the observed coordinates po = (x, y) under the corresponding camera and the reprojected coordinates pi, denoted as error:

[0161]

[0162] During the iteration process, when the value of the loss function is less than the set error, the iteration is terminated, and finally the intrinsic parameters and extrinsic parameters of all cameras are finally optimized between the two cameras. Compared with the calculated intrinsic parameters initially set and the calculated extrinsic parameters preliminarily calculated, the accuracy of the camera parameters is further improved.

[0163] After repeated optimization between two cameras, we can finally get relatively accurate calculation of internal and external parameters for each camera. Moreover, according to the relative posture between two camera pairs, we can transform the posture of all cameras into a coordinate system with the camera coordinate system of the main camera as the world coordinate system, so as to unify the coordinate system of the camera posture.

[0164] Further, as a preferred embodiment, the observation calibration rod includes multiple first calibration points, and the multiple first calibration points are distributed on the observation calibration rod in a predetermined form; specifically, the predetermined form is determined according to the form of the observation calibration rod, if it is one-dimensional, it is in a straight line form, and if it is a two-dimensional calibration rod, such as a T-shaped calibration rod, it is in a T-shaped form.

[0165] After the binocular calibration of the camera pairs, the internal and external parameter data of each optical camera are obtained. In order to ensure the smooth progress of subsequent global optimization, the calibration parameters of the cameras are verified in this step to ensure that each camera has been calibrated and the calibration results conform to the geometric properties of the one-dimensional calibration rod. Therefore, after obtaining the calculated internal and external parameters of multiple optical cameras, the following steps are also included:

[0166] First, verify the internal and external parameters of all calculated optical camera pairs to check the camera status. Through the depth-first search algorithm, recursively traverse the correspondence between the binocular geometry of the camera pairs and the camera pair parameters to ensure that each optical camera has been binocularly calibrated. Only when all optical cameras have been binocularly calibrated can the coordinate systems of all optical cameras be unified into one coordinate system through the connection between the optical cameras. Then, check whether the internal parameter matrix and the external parameter matrix of each camera pair correspond one by one to ensure that each optical camera has the internal and external parameter matrices after binocular calibration optimization and they correspond to each other.

[0167] Based on the calculated internal and external parameters of each optical camera and the corresponding first detection data, construct the three-dimensional coordinate data of multiple first calibration points on the observed calibration rod;

[0168] If the three-dimensional coordinate data of multiple first calibration points corresponding to any one of the optical cameras do not conform to the predetermined form, then re-use this optical camera as a slave camera to perform pairwise optimization with the master camera, and recalculate the calculated internal and external parameters of this optical camera.

[0169] According to the information recorded in the binocular geometry of the camera pairs, check the internal and external parameters of the optimized optical cameras to see if the geometric relationship between the three three-dimensional points representing the one-dimensional calibration rod after triangulation satisfies the properties of the one-dimensional calibration rod itself. In the specific process, according to the pose information between the cameras and the optimized internal parameter information recorded in the binocular geometry of the camera pairs, triangulate the observed two-dimensional coordinate points into three-dimensional coordinate points, and check whether the three-dimensional coordinate points of different cameras satisfy the pose transformation between the cameras and whether the three-dimensional coordinate points satisfy the geometric properties of the one-dimensional calibration rod. If it is satisfied, the data is retained; if it is not satisfied and the number of three-dimensional reconstruction points of the camera pair is less than three, then there is an error in the parameters of this camera pair in this case, and the binocular geometry information of this group of camera pairs is excluded and the internal and external parameters of the slave camera are recalculated.

[0170] Furthermore, in some embodiments, after obtaining the calculated internal and external parameters of all cameras after relatively accurate pairwise camera optimization, in order to make the internal and external parameters of the optical cameras more suitable for the overall system, all camera Bundle Adjustment optimizations need to be performed in this process. The core purpose of the BA optimization is to optimize the parameters of all cameras and the three-dimensional point coordinates to minimize the total reprojection error.

[0171] Through the steps of the above embodiments, more accurate camera parameters can be obtained than the initial settings, and connectivity exists between the camera pairs in the multi-camera system, meeting the sufficient requirements for the global optimization of the multi-camera system. However, the above optimization is only for the optimization between pairwise camera pairs. For a multi-camera system with mutual influence, it is necessary to optimize all cameras jointly, comprehensively, and hierarchically so that the reconstruction of three-dimensional points is more accurate and stable in the multi-camera system.

[0172] The process of this step is similar to that of step three. The difference is that step three is a local optimization of the multi-camera system, while this step is a global optimization of the entire multi-camera system, which is a joint overall optimization of the poses, internal parameters, and 3D points of all cameras in the multi-camera system. Its goal is to minimize the reprojection error so that the optimization result has global consistency. At the same time, this step embeds a hierarchical optimization strategy while performing global optimization, aiming to reduce the risk of local optimality, establish constraint relationships in the optimization process, and improve the computational efficiency and convergence performance of the optimization. For the hierarchical optimization strategy, this step first optimizes the pose of the camera and then optimizes the internal parameters of the camera. Because the pose parameters of the camera are easier to optimize and are not easily affected by interference points during the optimization process. When optimizing the internal parameters, there is an accurate estimate of the external parameters first, which can avoid the error accumulation of the internal parameters and reduce the risk of falling into local optimality. Adjusting the order of external parameter optimization and internal parameter optimization helps to reduce the propagation of errors, making the final optimization result more accurate and stable. Optimizing the external parameters first can quickly adjust the relative positions and directions between cameras during the initial optimization, and the internal parameter optimization can then reduce unnecessary calculations and convergence time based on these initial optimization results. Optimizing the external parameters first can reduce the interference of external pose estimation errors on the internal parameter optimization. Especially when the quality of the calibration data is not high, this hierarchical optimization can effectively improve the accuracy of the final result.

[0173] Furthermore, as a preferred solution, the observation calibration rod is a one-dimensional calibration rod with multiple first calibration points. The multiple first calibration points are on the same straight line, and the distances between the first calibration points are not equal. According to the optical infrared camera, image data of the reflective spheres on the one-dimensional calibration object is collected. The shape of the one-dimensional calibration rod is as Figure 2 shown. There are three marked points on the rod. The three marked points are on a straight line, and the distances between any two of them are not equal. During the data collection process, within the effective space range of the multi-camera system, the one-dimensional calibration rod is continuously waved, and the optical camera will automatically collect the image data of the reflective spheres and implement image data processing on the Field-Programmable Gate Array (FPGA);

[0174] Furthermore, please refer to Figure 4, As a preferred solution, the set calibration rod is an L-shaped calibration rod, which has a plurality of second calibration points respectively arranged on two right-angle arms of the L-shaped calibration rod.

[0175] Through Bundle Adjustment (BA) optimization and hierarchical optimization, the internal and external calculation parameters of the optical camera are optimized. However, the external calculation parameters of the camera are relative to the coordinate system of the main camera. In the above embodiments, the camera coordinate system of the main camera is simply regarded as the world coordinate system, but this is not the case in the actual space. Therefore, it is necessary to set up a practical world coordinate system, calculate the pose of each optical camera relative to the set world coordinate system, so that the three-dimensional points reconstructed in three dimensions are located in the set world coordinate system.

[0176] Furthermore, as a preferred solution, the second detection data includes two-dimensional coordinate data of a plurality of the second calibration points detected by one of the optical cameras;

[0177] Reconstructing the set calibration rod by using the internal and external calculation parameters of a plurality of the optical cameras, the pose information of each optical camera, and combining the second detection data based on the epipolar geometry principle to obtain simulated set data, and obtaining the offset data between the simulated set data and the basic data of the set calibration rod specifically includes:

[0178] S31. Obtain a two-dimensional point matching result based on the internal and external calculation parameters of a plurality of the optical cameras, the pose information of each optical camera, and the second detection data through the epipolar geometry principle.

[0179] It is necessary to reconstruct the second calibration points on the L-shaped calibration rod. First, it is necessary for a plurality of optical cameras to obtain the second detection data for the L-shaped calibration rod based on the optimized external and internal calculation parameters and the determined pose state, and then obtain the two-dimensional point matching result of a plurality of second calibration points. Since the rod is an L-shaped rod, the two-dimensional observation coordinates in different camera image coordinate systems cannot be matched one by one under different camera perspectives. Therefore, it is necessary to use the camera parameters optimized in the previous steps to obtain the relationship between the matching points. First, calculate the rays formed by the two-dimensional observation points on different camera image coordinate systems and the camera optical centers, and save them into cam_rays. Then, apply the principle of camera epipolar geometry. If the two-dimensional observation coordinates of two cameras represent the same spatial point, then the two rays cam_ray1 and cam_ray2 formed by the camera optical centers and the corresponding two-dimensional observation coordinates, and the straight line formed by the connection of the optical centers of the two cameras form a plane. Therefore, match the rays that can form a plane, and finally obtain the two-dimensional point matching result according to the ray matching result.

[0180] S32. Perform 3D reconstruction through triangulation calculation based on the 2D point matching result to obtain the 3D coordinate data of multiple second calibration points on the set calibration rod.

[0181] S33. Optimize the 3D coordinate data through clustering processing and reprojection error;

[0182] Specifically, according to the 2D point matching result obtained in S31, calculate the 3D coordinates of the second calibration points on the L-shaped calibration rod through triangulation calculation. Since the 2D points in the image may be too close, resulting in the cam_rays matching result exceeding the original quantity. Therefore, it is necessary to perform clustering processing on the reconstructed 3D point set and reduce the dimension of the 3D point set calculated by other smaller error matches within a certain range. Then, adjust the 3D points obtained by clustering based on the saved cam_rays information. Finally, use the reprojection error of multiple views to further optimize the coordinates of the 3D points, making the 3D coordinates balance the consistency of all cameras as much as possible. Specifically, the method of reprojection error here is basically the same as the application method of reprojection error in the foregoing embodiments, and will not be elaborated here.

[0183] S34. According to the plane data of the set calibration rod reconstructed in 3D, take the short side as the x-axis of the coordinate system and the long side as the y-axis of the coordinate system, find the normal vector perpendicular to this plane as the z-axis of the coordinate system, and then calculate the reconstructed pose information of the set calibration rod, and further obtain the offset data between the reconstructed pose information and the basic data of the set calibration rod. The basic data of the set calibration rod is the original pose information of the set calibration rod.

[0184] Further, as a preferred solution, it is necessary to set a set calibration rod related to the world coordinate system. This world coordinate system is set by an L-shaped rod including the x-axis and the y-axis, as Figure 3 . Specifically, the pose data of this set calibration rod is fixed, that is, it will not move easily, and the basic data is relatively fixed and has been measured and stored.

[0185] Further, as a preferred solution, optimize the calculated internal parameters and calculated external parameters of multiple optical cameras based on the offset data to obtain the calibrated internal parameters and calibrated external parameters of multiple optical cameras, specifically including:

[0186] S41. Obtain the rotation and translation relationship between the coordinate system of the master camera in the pairwise camera optimization and the real world coordinate system based on the offset data, and then obtain the calibrated internal parameters and calibrated external parameters of the master camera; that is, in this step, first obtain the calibrated internal parameters and calibrated external parameters of one of the multiple optical cameras, and then, through the rotation and translation relationship between multiple optical cameras, it is convenient to determine the calibrated internal parameters and calibrated external parameters of all optical cameras.

[0187] S42. Obtain the calibrated internal parameters and calibrated external parameters of other slave cameras through pairwise camera optimization based on the calibrated internal parameters and calibrated external parameters of the master camera.

[0188] In the multi-camera system calibration implemented by the calibration method provided by the present invention, through methods such as multi-view geometry, global hierarchical optimization, and clustering, a relatively simple and accurate multi-camera calibration method is realized.

[0189] 1. First, through the preliminary detection of the waved one-dimensional calibration rod, the multi-camera system is calibrated. The calibration object is easy to carry, the calibration data is not easily blocked, and the calibration logic is relatively simple and convenient for users.

[0190] 2. High-precision and high-efficiency calibration results depend on high-quality and diverse calibration data. In data processing, excluding abnormal data and retaining accurate data can improve the efficiency of multi-camera calibration optimization calculation and ensure the accuracy of multi-camera calibration.

[0191] 3. Use multi-view geometry and global hierarchical optimization methods to optimize the camera parameters step by step according to the logic, and integrate the geometric constraints of the one-dimensional calibration rod in the multi-camera view. This not only speeds up the speed and efficiency of multi-camera calibration, but also can improve the overall accuracy of multi-camera calibration and the general applicability of multi-camera calibration results.

[0192] 4. The ground alignment module uses the principle of epipolar geometry. Without forcing the matching point pair relationship by error magnitude, the method of clustering is used to reduce the dimension and fit the similar point pairs, and a three-dimensional point set and point correspondence matching relationship with more general practicality and conforming to the global characteristics of the multi-camera system can be obtained. The relationship between the camera coordinate system and the world coordinate system can be calculated more accurately, and then the calibrated internal parameters and calibrated external parameters that finally conform to the real world coordinate system can be calculated.

[0193] Correspondingly, please refer to Figure 5 , the present invention also provides a multi-camera calibration system, including:

[0194] A primary processing module, configured to obtain the first detection data of a waved observation calibration rod in the view space by multiple optical cameras, and then obtain the calculated internal parameters and calculated external parameters of the multiple optical cameras through pairwise camera optimization based on the first detection data;

[0195] The secondary processing module is used to obtain the second detection data of each optical camera for the set calibration rod in the real-world coordinate system; based on the calculation internal parameters, calculation external parameters and pose information of each of the plurality of optical cameras, and combining the second detection data, the set calibration rod is reconstructed based on the epipolar geometry principle to obtain simulated set data, and the offset data between the simulated set data and the basic data of the set calibration rod is obtained; based on the offset data, the calculation internal parameters and the calculation external parameters of the plurality of optical cameras are optimized to obtain the calibrated internal parameters and calibrated external parameters of the plurality of optical cameras.

[0196] Correspondingly, the present invention also provides an electronic device, including:

[0197] A memory storing a computer program;

[0198] A processor, when executing the computer program, implements the multi-camera calibration method according to any one of the embodiments.

[0199] Correspondingly, the present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the multi-camera calibration method according to any one of the embodiments is implemented.

[0200] More specific examples of the computer-readable storage medium may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0201] It can be understood that for those of ordinary skill in the art, equivalent substitutions or changes can be made according to the technical solutions and inventive concepts of the present invention, and all such changes or substitutions should fall within the protection scope of the appended claims of the present invention.

Claims

1. A multi-camera calibration method, characterized in that: include: Acquire first detection data of a plurality of optical cameras for an observation calibration rod waving in a view space, and then obtain calculation intrinsic parameters and calculation extrinsic parameters of the plurality of optical cameras through pairwise camera optimization based on the first detection data; Acquire second detection data of each optical camera for a set calibration pole in a real-world coordinate system; Reconstructing the set calibration rod based on the epipolar geometry principle by using the calculated internal parameters and external parameters of the plurality of optical cameras and the position and posture information of each optical camera in combination with the second detection data to obtain simulated set data, and obtaining offset data between the simulated set data and the basic data of the set calibration rod; The calculated internal parameters and the calculated external parameters of the multiple optical cameras are optimized based on the offset data to obtain the calibrated internal parameters and the calibrated external parameters of the multiple optical cameras.

2. The multi-camera calibration method according to claim 1, characterized in that: The observation calibration rod has a plurality of first marking points; Acquiring first detection data of a plurality of optical cameras for an observation calibration rod waving in a view space, and then obtaining calculation internal parameters and calculation external parameters of the plurality of optical cameras based on the first detection data specifically includes: Acquire image data of a plurality of first calibration points on a swinging observation calibration rod collected by a plurality of infrared cameras; perform image preprocessing on the image data to obtain calibration data; the calibration data includes frame data related to a plurality of detection times; the frame data includes frame label data of image data of all infrared cameras at the same detection time; the frame label data includes one or more sub-label data in a frame of image data; the sub-label data includes coordinate data of a first calibration point and an identification code of a corresponding optical camera; Performing data preprocessing on the calibration data based on the basic parameters of the observed calibration rod, eliminating invalid data and retaining valid data; The optical camera with the most valid data is used as the main camera, and the other optical cameras to be optimized are set as slave cameras; the coordinate system of the main camera is used as the world coordinate system, and the external parameters of the main camera are set as unit external parameters, so that multiple slave cameras are optimized in pairs with the main camera respectively, and the calculated external parameters and calculated internal parameters of all slave cameras are obtained.

3. The multi-camera calibration method according to claim 2, characterized in that: The data preprocessing includes: Perform interference light source detection on all the frame mark data in turn, construct a corresponding detection area, and remove the sub-mark data located in the detection area; Eliminate the frame mark data whose number of sub-mark data is not equal to the number of first mark points; Performing morphological detection on the sub-label data in all the frame label data in turn, and removing the frame label data that fails the morphological detection; Performing static detection on a plurality of the frame mark data in the same frame data, and removing duplicate frame mark data; The frame data whose number of frame mark data is less than a predetermined value is eliminated.

4. The multi-camera calibration method according to claim 2, characterized in that: The step of performing two-by-two camera optimization on the slave cameras and the master camera comprises: Acquire the original internal parameters of the master camera and the slave camera as the calculation internal parameters, and obtain the internal parameter matrix by initializing the original internal parameters of the optical camera; Based on the coordinate data detected by the master camera and the slave camera for the same first calibration point, a basic matrix is ​​obtained by a random sampling consistency algorithm, and then the intrinsic matrix is ​​obtained by combining the intrinsic parameter matrices of the master camera and the slave camera; The essential matrix is ​​subjected to singular value decomposition to obtain calculation extrinsic parameters, wherein the calculation extrinsic parameters include a rotation matrix and a translation vector.

5. The multi-camera calibration method according to claim 2, characterized in that: The coordinate data is the centroid coordinates; the calculation formula of the centroid coordinates is: Among them, x is the x-axis coordinate of the centroid coordinate; y is the y-axis coordinate of the centroid coordinate; g(i,j) represents the pixel value with coordinates (i,j) on the image, i represents the value of the pixel point on the x-axis of the two-dimensional coordinate, and j represents the value of the pixel point on the y-axis of the two-dimensional coordinate.

6. The multi-camera calibration method according to claim 2, characterized in that: After obtaining the calculation external parameters and the calculation internal parameters, the method further includes: The calculated extrinsic parameters and the calculated internal parameters of the main camera, as well as the calculated extrinsic parameters and the calculated internal parameters of the slave camera are iteratively optimized in combination with corresponding calibration data, wherein the loss function in the iterative optimization process is the reprojection error, and the optimized calculated internal parameters and the calculated extrinsic parameters of the main camera, as well as the optimized calculated extrinsic parameters and the calculated internal parameters of the slave camera are obtained.

7. The multi-camera calibration method according to claim 6, characterized in that: The observation calibration rod comprises a plurality of first calibration points, and the plurality of first calibration points are distributed on the observation calibration rod in a predetermined form; After optimizing the calculation internal and external parameters of multiple optical cameras, it also includes: Based on the calculated internal parameters and calculated external parameters of each of the optical cameras and the corresponding first detection data, construct the three-dimensional coordinate data of a plurality of the first calibration points on the observation calibration rod; If the three-dimensional coordinate data of the plurality of first calibration points corresponding to any of the optical cameras do not conform to the predetermined form, the optical camera is re-used as a slave camera to perform pairwise optimization with the master camera, and the calculation internal parameters and calculation external parameters of the optical camera are recalculated.

8. The multi-camera calibration method according to claim 1, characterized in that: The observation calibration rod is a one-dimensional calibration rod having a plurality of first calibration points. The plurality of first calibration points are located on the same straight line, and the distances between the first calibration points are not equal.

9. The multi-camera calibration method according to claim 1, characterized in that: The set calibration rod is an L-shaped calibration rod having a plurality of second calibration points, which are respectively arranged on two right-angle arms of the L-shaped calibration rod.

10. The multi-camera calibration method according to claim 9, characterized in that: The second detection data includes two-dimensional coordinate data of a plurality of the second calibration points detected by the optical camera; The step of reconstructing the set calibration rod based on the epipolar geometry principle by calculating the internal parameters and external parameters of the plurality of optical cameras and the position information of each optical camera in combination with the second detection data to obtain the simulated set data, and obtaining the offset data between the simulated set data and the basic data of the set calibration rod, specifically includes: Obtaining a two-dimensional point matching result based on the position information of each optical camera and the second detection data by using the epipolar geometry principle; Based on the calculated internal parameters, calculated external parameters and the two-dimensional point matching results of the plurality of optical cameras, three-dimensional reconstruction is performed through triangulation calculation to obtain three-dimensional coordinate data of the plurality of second calibration points on the set calibration rod; Optimizing the three-dimensional coordinate data through clustering processing and reprojection errors; According to the plane data of the set calibration rod obtained by three-dimensional reconstruction, the short side is regarded as the x-axis of the coordinate system, and the long side is regarded as the y-axis of the coordinate system. The normal vector perpendicular to the plane is calculated and regarded as the z-axis of the coordinate system. Then, the reconstructed pose information of the set calibration rod is calculated, and then the offset data between the reconstructed pose information and the basic data of the set calibration rod is obtained.

11. The multi-camera calibration method according to claim 10, characterized in that: The calculation internal parameters and the calculation external parameters of the multiple optical cameras are optimized based on the offset data to obtain the calibration internal parameters and the calibration external parameters of the multiple optical cameras, specifically including: Based on the offset data, a rotation and translation relationship between a coordinate system of a main camera in the pairwise camera optimization and a real-world coordinate system is obtained, thereby obtaining a calibration intrinsic parameter and a calibration extrinsic parameter of the main camera; Based on the calibrated internal parameters and the calibrated external parameters of the master camera, the calibrated internal parameters and the calibrated external parameters of other slave cameras are obtained through pairwise camera optimization.

12. A multi-camera calibration system, characterized in that: include: A primary processing module, used to obtain first detection data of a plurality of optical cameras for an observation calibration rod waving in a view space, and then obtain calculation intrinsic parameters and calculation extrinsic parameters of the plurality of optical cameras through pairwise camera optimization based on the first detection data; The secondary processing module is used to obtain second detection data of each optical camera for a set calibration rod in a real-world coordinate system; reconstruct the set calibration rod based on the polar geometry principle through the calculated internal parameters and external parameters of the multiple optical cameras and the pose information of each optical camera in combination with the second detection data to obtain simulated setting data, and obtain offset data between the simulated setting data and the basic data of the set calibration rod; optimize the calculated internal parameters and the calculated external parameters of the multiple optical cameras based on the offset data to obtain the calibration internal parameters and calibration external parameters of the multiple optical cameras.

13. An electronic device, characterized in that: include: a memory storing a computer program; The processor, when executing the computer program, implements the multi-camera calibration method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the multi-camera calibration method described in any one of claims 1-11 is implemented.

Images