Multi-camera calibration system and method without overlapping view

Abstract

The invention discloses a multi-camera calibration system and method without overlapping visual fields, and the method comprises the steps: enabling a tracker composed of a plurality of receivers to cooperate with a base station as a conversion bridge, and achieving the coordinate system conversion between a plurality of trackers, the coordinate system conversion between cameras and trackers, and the coordinate system conversion between cameras. And multi-camera calibration without public view is realized. The main advantages / differences are as follows: only the internal reference of one camera needs to be calibrated before calibration of the same type of camera is started, and the internal reference of the camera does not need to be measured in the actual calibration process. Electric signal processing replaces an image processing mode, signal acquisition is fast, a tedious image acquisition process is not needed, and calibration time is remarkably shortened. Calibration precision is guaranteed through the appearance structure, and environment changes are adapted. A calibration board is not needed, and the device is simple and portable and can be rapidly deployed in a new environment. The number of cameras, installation positions and angles, installation intervals and the like can be designed according to actual requirements, and influences on calibration feasibility do not need to be considered in advance like visual calibration.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a multi-camera calibration system and method without overlapping fields of view. Background Art

[0002] Computer vision is based on the physical environment information obtained by the camera. It can identify, track and segment people and objects in the scene with high precision, serve the needs of environmental perception, scene understanding, intelligent interaction, etc., create huge economic and social benefits, and is a typical feature of the new generation of information technology. Among them, three-dimensional (3D) cameras have spatial perception capabilities and can obtain relative depth information of objects in the scene while perceiving color images. Therefore, they are widely used in three-dimensional scenes such as three-dimensional reconstruction, object segmentation, and spatial tracking. Its typical use method is to superimpose the 3D physical scene information obtained by multiple cameras according to their actual spatial positions and postures, supplement the field of view of a single camera from different directions, and expand the perceptible scene range through data splicing.

[0003] In this process, one of the key steps is to obtain the relative position and attitude information between cameras, that is, to obtain the position and attitude of the local coordinate system of all cameras in a unified global coordinate system. The more commonly used method is binocular positioning. By setting identifiable markers in the overlapping field of view between each two cameras, the relative position and attitude relationship between the camera and the marker is obtained based on the image processing method, thereby calculating the relative position and attitude relationship between each two cameras, and obtaining the relative position and attitude relationship between all cameras through coordinate system transfer. However, the premise of using this method is that there must be overlapping fields of view between cameras. It is not only inefficient, but also fails in scenes with large-scale non-overlapping fields of view, such as discrete distribution of cameras, long distances, and large viewing angle deviations.

[0004] After searching, the existing multi-camera calibration solutions without overlapping field of view can be roughly divided into the following three technical routes:

[0005] The first method is to put multiple calibration plates into the same coordinate system to obtain the transformation relationship between the coordinate systems and realize the transfer of coordinate systems between cameras with non-overlapping fields of view, as described in reference 1, using the calibration method of multiple small calibration plates. First, the Zhang Zhengyou method is used to calibrate the intrinsic parameters of each camera; then, the relative position relationship of the small calibration plates is measured using a total station, and the small calibration plates are unified into the same coordinate system, integrated into a large calibration plate, and the pose relationship of each camera relative to the integrated calibration plate is calculated respectively; finally, the coordinate system is unified to obtain the extrinsic parameters between multiple cameras.

[0006] The second method is to move the cameras so that they can all capture the same target to obtain the transformation relationship between the coordinate systems, thereby realizing the transfer of coordinate systems between cameras with non-overlapping fields of view. As described in Reference 2, a plane calibration plate and a two-axis turntable are fixed, and multiple cameras are fixed on the two-axis turntable. A world coordinate system is established with the upper left corner of the target as the origin. Multiple images are obtained at multiple positions using a single camera through the rotation of the turntable itself, and the relationship between the world coordinate system and the turntable coordinate system is determined; secondly, the turntable is rotated to make the target enter the field of view of each camera respectively to complete the external parameter calibration of the single camera; finally, the relative position relationship between each camera is solved based on the relationship between the world coordinate system and the turntable coordinate system, and the relationship between the world coordinate system and the coordinate system of each camera.

[0007] The third method is to introduce an auxiliary camera to shoot all calibration plates to obtain the transformation relationship between the coordinate systems and realize the transfer of coordinate systems between cameras with non-overlapping fields of view. The principle is as described in document 3, and the rotation matrix and translation vector between each camera to be calibrated and the auxiliary camera are obtained respectively.

[0008] Through the analysis of the non-overlapping field of view multi-camera calibration technology in the existing technical routes, the inventors found that the three routes in the existing technology require each camera to take a photo of the calibration plate or target separately, and then use the photos to determine the intrinsic and extrinsic parameters of each camera by means of, for example, the Zhang Zhengyou calibration method, and then use the extrinsic parameters of each camera to construct a transformation matrix to obtain the coordinate system transformation relationship between each camera, that is, to achieve the transfer of coordinate systems between cameras with non-overlapping fields of view (for a detailed explanation of the specific formula principle, please refer to Chapter 3 of Reference 4 on multi-camera calibration). However, when the camera constructs the transformation matrix between the coordinate systems of multiple cameras using the intrinsic and extrinsic parameters of each camera, there are the following deficiencies:

[0009] 1. Before performing multi-camera calibration, each camera must be calibrated separately and the internal parameters of each camera must be recorded. Only then can these cameras be combined into a new system to perform multi-camera calibration without a common field of view.

[0010] 2. Due to processing reasons, lens distortion includes radial distortion, tangential distortion, centrifugal distortion and thin prism distortion. In the calibration process of a single camera, based on the complexity of the algorithm, only the radial distortion of the lens is generally considered, while other distortion parameters are ignored. For a single camera, the distortion parameters have little effect on subsequent 3D reconstruction, object segmentation, spatial tracking and other tasks. However, in multi-camera calibration, the distortion error of the camera farthest from the camera of the reference coordinate system will be accumulated and amplified in the coordinate system transfer operation. For example, if the camera numbered 1 is used as the reference camera and there are 40 cameras, the coordinate system conversion process of the 40th camera to the reference camera needs to rely on the parameters of 38 cameras from No. 2 to No. 39 for coordinate transfer, resulting in the distortion error of the 40th camera being accumulated and amplified. At the same time, if these 40 cameras use different calibration plates for separate calibration. It may also lead to poor consistency of parameters obtained from different camera calibrations. These poorly consistent parameters will bring uncertainty in the coordinate system transfer in multi-camera calibration. For example, if the calibration accuracy of camera No. 5 is poor, then the cameras that need to be converted into the reference coordinate system through the parameters of camera No. 5 will have the accuracy of the tutorial. In general, when the intrinsic and extrinsic parameters of the camera are used as the parameters of the coordinate transfer matrix, the errors carried in the calibration process may be propagated to the multi-camera calibration system.

[0011] 3. The above method must use a calibration board, which has shortcomings in portability and ease of use, and has clear constraints on the camera installation position and direction.

[0012] Reference 1, Wang Anran, Hao Xiangyang, Cheng Chuanqi, Jia Kaikai. A multi-camera external parameter calibration method using multiple small calibration plates[J]. Surveying and Mapping and Spatial Geographic Information, 2019, 42(06): 222-225+229.

[0013] Reference 2, Lu Yanan, Wan Zijing, Wang Xiangjun. A method for solving the position relationship of cameras without a common field of view [J]. Applied Optics, 2017, 38(03): 400-405.

[0014] Reference 3, Patent Publication Number, CN116188591A, Patent Name, Multi-camera global calibration method and device, electronic device.

[0015] Reference 4, Li Wenjing. Research on multi-camera calibration and 3D reconstruction technology[D]. Xi’an University of Technology, 2022.DOI:10.27398 / d.cnki.gxalu.2022.000869. Summary of the invention

[0016] The present invention provides a multi-camera calibration system and method with non-overlapping fields of view, which can sequentially obtain the position and attitude parameters of the local coordinate system of each camera in a common unified coordinate system for a large range of non-overlapping fields of view, such as when multiple cameras are discretely distributed, far away, and have large viewing angle deviations. It is also possible to eliminate the need to calibrate the intrinsic parameters of all cameras separately before calibrating the multiple cameras, and reduce the risk of propagation of the intrinsic parameter errors of some cameras in the multi-camera system.

[0017] The multi-camera calibration described in the present invention is camera extrinsic calibration, that is, obtaining the position and attitude description of the local coordinate systems of all cameras in a unified reference coordinate system. The description is usually a 4×4 position and attitude matrix, including 3 position parameters and 9 rotation attitude parameters.

[0018] To achieve the above objectives, in a first aspect, the present invention provides a multi-camera calibration system with non-overlapping fields of view, the system comprising:

[0019] At least two cameras to be calibrated;

[0020] At least two trackers, each tracker can be detachably rigidly connected to each camera, and after connection, the tracker and each camera have the same relative position and posture relationship, and the tracker is provided with a plurality of signal receiving devices in different directions on the outer surface, and is used in conjunction with a base station that transmits signals outward, and is used to obtain the position and posture information of the tracker's local coordinate system relative to the base station coordinate system according to the distribution of the intensity of the received signal, and form a corresponding position and posture matrix;

[0021] A base station, wherein the signal emitted by the base station is dispersedly transmitted outward from a point, and is used to establish a real-time tracking relationship with the tracker in the observable area based on its own coordinate system within the signal coverage angle and range; and

[0022] A combination, comprising a planar marker and a tracker whose relative positions and postures are known, is used to obtain, through coordinate system transformation, the position and posture relationship between the camera coordinate system and the local coordinate system of the tracker installed thereon after the tracker is installed on the camera.

[0023] Compared with the prior art, it is necessary to establish a conversion matrix for transferring coordinate systems between adjacent cameras in sequence through the internal and external parameters obtained after each camera is calibrated separately, which causes the inconvenience of calibrating each camera separately in advance, and the calibration errors of each camera may be caused to propagate widely in the multi-camera system. A multi-camera calibration system without overlapping fields of view provided in the present invention uses a tracker composed of multiple photoelectric receivers and a base station as a conversion bridge to realize the coordinate system conversion between multiple trackers, as well as the coordinate system conversion between cameras and trackers, and the coordinate system conversion between cameras, and finally realizes multi-camera calibration without a common field of view, without using the matrix composed of the internal and external parameters of multiple cameras as a conversion bridge, and can eliminate the need to calibrate the internal and external parameters of all cameras separately before multi-camera calibration, which is convenient for changing the position and posture of some cameras during the multi-camera calibration process, and reduces the risk of propagation of internal and external parameter errors of some cameras in the multi-camera system.

[0024] As a further improvement, the plane marker is a rectangular plane marker that can be identified through image positioning, and the tracker is placed on the plane marker. The relative position relationship between the tracker coordinate system and the plane marker coordinate system can be obtained through physical measurement.

[0025] Since the difference between the two in the vertical direction of the plane marker is the height of the tracker base, and the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the plane marker coordinate system, the coordinate system conversion between the two is convenient and quick without the need for complicated calculations.

[0026] As a further improvement, it also includes a tracker base and a rigid intermediate connector, the tracker is fixedly connected to the tracker base, the rigid intermediate connector is adapted to the shape of each camera and is fixedly connected thereto, each camera has a rigid connector connected thereto as a whole, and the tracker can be rigidly mounted to the intermediate connector corresponding to each camera through the tracker base.

[0027] Taking into account that camera manufacturers will manufacture cameras into various shapes for commercial reasons, in order to make the tracker more convenient, tight and accurate to connect with the camera, a rigid middleware is specially designed to connect the camera and tracker.

[0028] In a second aspect, the present invention provides a multi-camera calibration method without overlapping fields of view, the method being applied to the above-mentioned system, the method comprising:

[0029] Camera and tracker calibration step: based on the assembly, determining a first transformation relationship between a camera coordinate system of each camera and a tracker coordinate system mounted thereon;

[0030] Local tracker positioning step: adjust the position and observation angle of the base station so that at least two cameras are within the observable range of the base station. After installing trackers for all cameras in the observable area, obtain the position and attitude parameters of all trackers in the observable area relative to the base station, and calculate the transformation relationship between the tracker coordinate system of each tracker and the coordinate system of the known object in this observation based on the observed position and attitude parameters of all trackers relative to the base station; wherein the coordinate system of the known object is a local coordinate system established for the base station or the tracker itself; the transformation relationship between the coordinate system of the known object and the global coordinate system is known; when the local tracker positioning step is performed for the first time, the known object is the base station at the time of this observation, and when the local tracker positioning step is performed subsequently, it is the first known tracker in the observation range.

[0031] Global tracker expansion positioning step: repeating the local tracker positioning step multiple times, adjusting the position and observation angle of the base station each time, so that the new observable area covers at least one known tracker except the first positioning, until the transformation relationship between the tracker coordinate system of all trackers and the tracker coordinate system of the known tracker in each observation is obtained;

[0032] The step of global unification of the camera coordinate system: based on the first transformation relationship and the transformation relationship between the coordinate system of the tracker installed on each camera and the global coordinate system, a second transformation relationship between the camera coordinate system of each camera and the global coordinate system is determined.

[0033] Compared with the prior art, it is necessary to establish the conversion matrix of the coordinate system of each adjacent camera in sequence by using the internal and external parameters obtained after each camera is calibrated separately, which causes the inconvenience of calibrating each camera in advance, and the calibration errors of each camera may be widely propagated in the multi-camera system. A multi-camera calibration method without overlapping field of view provided in the present invention is to use a tracker composed of multiple photoelectric receivers and a base station as a conversion bridge to realize the coordinate system conversion between multiple trackers, as well as the coordinate system conversion between cameras and trackers, and the coordinate system conversion between cameras, and finally realize the calibration of multiple cameras without common field of view, without using the matrix composed of the internal and external parameters of multiple cameras as a conversion bridge, and can be used without the need to calibrate the internal and external parameters of all cameras separately before the multi-camera calibration, which is convenient for changing the position and posture of some cameras during the multi-camera calibration process, and reducing the risk of propagation of the internal and external parameter errors of some cameras in the multi-camera system. In addition, in this method, each new tracker is to perform coordinate conversion with the known tracker, and the base station only provides the transfer function for the coordinate system conversion between the two, so there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the difficulty of implementation.

[0034] As a further step, the camera and tracker calibration step specifically includes:

[0035] Acquire a third transformation relationship between a plane marker coordinate system of the plane marker in the assembly and a tracker coordinate system of the tracker;

[0036] Using an image of the plane marker captured by any one of the multiple cameras, calibrate the corresponding camera to obtain a fourth transformation relationship between a camera coordinate system of the camera and a plane marker coordinate system of the plane marker;

[0037] Obtaining, through the base station, a fifth transformation relationship between a tracker coordinate system of the tracker on the assembly and a tracker coordinate system of the tracker on the camera;

[0038] Based on the third transformation relationship, the fourth transformation relationship and the fifth transformation relationship, a first transformation relationship between the camera coordinate system of each camera and the tracker coordinate system installed thereon is calculated.

[0039] Specifically, with the aid of the assembly, a first transformation relationship between the camera coordinate system and the tracker coordinate system installed thereon can be quickly established, and the operation is simple and quick.

[0040] As a further improvement, the third transformation relationship between the plane marker coordinate system of the plane marker in the assembly and the tracker coordinate system of the tracker is obtained, specifically including:

[0041] The third transformation relationship between the plane marker coordinate system of the plane marker in the assembly and the tracker coordinate system of the tracker is obtained by measuring the height of the tracker base;

[0042] Among them, the plane marker is a rectangular plane marker that can be identified by image positioning, and the tracker is installed on the tracker base, and the whole consisting of the tracker and the tracker base is placed at the center of the plane marker, wherein the plane marker defines the origin of its own local coordinate system at its center, and the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the plane marker coordinate system, and the difference between the two in the vertical direction of the plane marker is a height distance of the tracker base.

[0043] Since the difference between the two in the vertical direction of the plane marker is the height of the tracker base, and the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the plane marker coordinate system, the coordinate system conversion between the two is convenient and quick without the need for complicated calculations.

[0044] As a more general implementation method, the third transformation relationship between the plane marker coordinate system of the plane marker in the assembly and the tracker coordinate system of the tracker is obtained, specifically including:

[0045] Obtaining a third change relationship by measuring the relative position and posture between the coordinate system of the planar marker and the tracker coordinate system of the tracker;

[0046] Among them, the planar marker is any two-dimensional picture that can be positioned through image recognition, and the two coordinate axes of the planar marker coordinate system are set on the picture plane, and the corresponding coordinate origin and the two coordinate axis directions are marked for positioning the tracker in the combination; the tracker in the combination is set on the planar marker.

[0047] In this implementation, the two-dimensional image can be of any shape, and the tracker in the assembly can be set at any position on the plane marker, as long as the third transformation relationship can be obtained based on the actual spatial position measurement. Therefore, this implementation is more versatile.

[0048] As a further improvement, the method further includes:

[0049] When the number of trackers is less than the number of all cameras that need to be calibrated, after the base station changes its position and angle, the trackers on the cameras that are not in the observable area are removed and the removed trackers that are not in the observable area are installed on the cameras in the current observable area.

[0050] The detachable design greatly reduces the number of trackers required.

[0051] As a further improvement, the method further includes:

[0052] When the cameras are far apart and the observable area of ​​the base station cannot cover the trackers of at least two cameras, one or more new relay trackers are set in the middle of the two cameras, so that the base station can change its position and observation angle so that the observable area of ​​the base station can cover the tracker installed on the camera and at least one relay tracker, or can simultaneously cover at least two relay trackers located in the middle of the two cameras;

[0053] Through multiple position transfers of the relay tracker, according to the local tracker positioning step and the global tracker expansion positioning step, the transformation relationship between the tracker coordinate system of all trackers and the tracker coordinate system of the known tracker in each observation is obtained.

[0054] By adopting a relay tracker solution, the applicability of the number and range of cameras in this method can be greatly increased.

[0055] As a further improvement, the method further includes:

[0056] Set the local coordinate system of one of the cameras as the reference, and transform the position and attitude parameters of all cameras into relative position and attitude parameters relative to the local coordinate system of the camera through coordinate transformation.

[0057] As a further improvement, the method further includes:

[0058] An additional reference coordinate system is introduced, and the position and attitude parameters of all cameras are transformed into relative position and attitude parameters relative to the reference coordinate system through coordinate transformation.

[0059] Since the tracker in the present invention can quickly and easily obtain its position and posture information through the base station, it is easier to align with the introduced additional reference coordinate system compared to the solution using internal and external parameters of the camera in the prior art.

[0060] In summary, the multi-camera calibration system and method with non-overlapping fields of view provided by the present invention has at least the following main advantages / differences:

[0061] 1. For cameras of the same model, only one camera needs to be calibrated before calibration. There is no need to measure the camera's internal parameters during the actual calibration process.

[0062] 2. Electrical signal processing replaces image processing, signal acquisition is fast, no cumbersome image acquisition process is required, and calibration time is significantly shortened.

[0063] 3. The appearance structure ensures calibration accuracy and adapts to environmental changes; no calibration board is required, the equipment is simple and portable, and can be quickly deployed in a new environment.

[0064] 4. The number of cameras, installation positions and angles, installation intervals, etc. can be designed according to actual needs, without having to consider the impact on calibration feasibility like visual calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of the installation assembly of a tracker, a tracker base, a rigid intermediate connector, and a camera in one embodiment;

[0066] Figure 2 A schematic diagram of an embodiment in which a tracker and its base are mechanically mounted on a rigid intermediate connector;

[0067] Figure 3 A schematic diagram of an embodiment in which a tracker and its base are mounted on a rigid intermediate connector by magnetic attraction;

[0068] Figure 4 A schematic diagram of a scene in which a camera and a tracker mounted thereon are calibrated with a coordinate system by means of a combination of a planar marker and the tracker in one embodiment;

[0069] Figure 5 A flowchart of coordinate system calibration of a camera and a tracker mounted thereon in one embodiment;

[0070] Figure 6 A schematic diagram of positioning a single local tracker in one embodiment;

[0071] Figure 7 A flowchart of a single local tracker positioning in one embodiment;

[0072] Figure 8 A flowchart of global tracker extension and camera coordinate system global unification in one embodiment;

[0073] Fig. 9A A schematic diagram of a global tracker expansion in one embodiment;

[0074] Fig. 9B A schematic diagram of a global tracker expansion in one embodiment;

[0075] Fig. 9C A schematic diagram of a global tracker expansion in one embodiment;

[0076] Fig.9D A schematic diagram of a global tracker expansion in one embodiment;

[0077] Fig.9E A schematic diagram of a global tracker expansion in one embodiment;

[0078] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0079] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.

[0080] The multi-camera spatial positioning method with a large range and non-overlapping field of view described in the present invention has four steps, including installation and positioning of the tracker on the camera, calibration of the tracker coordinate system and the camera coordinate system, local tracker positioning, global tracker expansion and global unification of the camera coordinate system.

[0081] The step 1 is to install and position the tracker on the camera.

[0082] The tracker is provided with a plurality of signal receiving devices in different directions on its outer surface. When used in conjunction with a base station that transmits signals outward, the position and attitude information of the local coordinate system of the tracker relative to the base station coordinate system can be obtained according to the distribution of the intensity of the received signal, thereby forming a corresponding position and attitude matrix. The bottom can be a regular plane structure such as a regular rectangle or circle, and can be provided with regular positioning structures such as square holes, round holes, V-shaped grooves, and threaded holes. The center position of the bottom is the origin of the coordinate system during spatial tracking. Two of the three coordinate axes are located on the bottom surface, and the third axis is perpendicular to the bottom surface, forming a local coordinate system on the tracker. The position and attitude information obtained during spatial tracking is the pose matrix of the coordinate system relative to the base station coordinate system.

[0083] The signal emitted by the base station is dispersedly transmitted from one point to the outside, and within the signal coverage angle and range, the base station uses its own coordinate system as a reference to establish a real-time tracking relationship with the tracker in the observable area. After adjusting the observation position and observation angle, the base station re-establishes the reference coordinate system at its location and establishes a tracking relationship with the tracker in the new observable area.

[0084] The tracker is provided with a special base in a regular rectangular structure, which is precisely combined with the tracker through the positioning and connection structure at the bottom of the tracker. The rectangular edges of the base are parallel to the coordinate axis of the tracker. After the connection, the two are no longer separated and are connected to the outside as a whole.

[0085] The tracker base may be provided with positioning holes, positioning pins and other structures. Except for the parts connected to the tracker, the other structures are not restricted, and the positioning and connection structures may be specially designed according to the external connection method.

[0086] The positioning of the tracker and the base structure on the camera can be done with the help of a rigid intermediate connector with a special shape. The rigid intermediate connector is first positioned on the outer surface of the camera, and then the tracker and the base are positioned and connected to the rigid intermediate connector as a whole, so that the tracker and the camera have a fixed relative position relationship.

[0087] The rigid intermediate connecting piece has a structure adapted to the camera shape and is pre-fixed on each camera by plane, end face contact positioning and hole axis constraint positioning.

[0088] The positioning of the tracker and the base as a whole on the rigid intermediate connector can be mechanically performed by end face positioning and screw tightening. The positioning method can also be magnetic, with positioning pins and magnets provided on the tracker base, positioning holes and opposite magnets provided at corresponding positions of the rigid intermediate connector, and the positioning is performed by hole-shaft matching and the connection is performed by magnet adsorption.

[0089] The tracker and the base are integrally connected and fixed on the rigid middle connector, and are a detachable structure, which can be installed when the camera position needs to be obtained, and can be disassembled after the position is obtained.

[0090] The second step is to calibrate the tracker coordinate system and the camera coordinate system.

[0091] The camera coordinate system is a reference coordinate system when the camera collects spatial depth information (a 3D camera is composed of a depth sensor (such as a laser scanner or a structured light scanner, which collects depth images) and a visible light sensor (i.e., a camera, which is used to collect color images). The depth image and the color image are first spatially aligned (i.e., the coordinate system is converted), and then the depth information collected by the depth sensor can be converted to the camera coordinate system). It is also a local coordinate system when 3D physical scenes are stitched, and is defined on the color image sensor (i.e., the visible light image sensor) of the camera.

[0092] The calibration of the tracker coordinate system and the camera coordinate system is to obtain the relative position and posture relationship between the tracker coordinate system and the camera coordinate system after the tracker and the base are integrally positioned and installed on the rigid intermediate connector. The specific method is to set a plane marker and tracker combination whose relative position and posture are known, and the base station simultaneously obtains the coordinates of the tracker on the combination and the tracker on the camera, and obtains the position and posture of the tracker installed on the camera relative to the local coordinate system of the plane marker through coordinate system conversion. After the camera obtains its position and posture relative to the local coordinate system of the plane marker through image recognition, the position and posture of the camera coordinate system relative to the local coordinate system of the tracker installed thereon can be obtained through coordinate system change.

[0093] The planar marker and tracker assembly comprises a rectangular planar marker that can be located and identified by color image and a tracker and base assembly placed at the center of the marker. The rectangular planar marker defines its own local coordinate system at its center, and the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the marker local coordinate system. The two differ only by the height of the tracker base in the vertical direction of the planar marker. The relative transformation matrix between the two coordinate systems can be obtained by measuring the height of the tracker base.

[0094] The plane marker may be a regular chessboard, an ellipse, or any image with a complex texture, and the relative position relationship between the plane marker and the camera may be obtained through a feature point recognition method.

[0095] The relative position and attitude relationship between the tracker coordinate system and the camera coordinate system is consistent and universal when the shapes of the tracker base, the rigid intermediate connector and the camera do not change. When the same model of tracker and its base, the rigid intermediate connector and the camera are used for connection, the relative position and attitude relationship between the tracker coordinate system and the camera coordinate system does not change, and the position and attitude parameters of one coordinate system can be used to calculate the position and attitude parameters of the other coordinate system.

[0096] The step three is local tracker positioning.

[0097] The local tracker positioning is to use the base station to locate the trackers within its observable area once and obtain the coordinates of all trackers. The specific method is to adjust the position and observation angle of the base station so that at least two cameras are within its observable range, install trackers for all cameras in the observable area, and directly obtain the position and attitude parameters of all trackers in the observable area, that is, the local coordinates relative to the base station coordinate system. Subsequently, the local coordinates of all observed trackers are converted into relative coordinates relative to the coordinate system of the known object in this observation, and further calculate the coordinates of all trackers observed in this observation in the global reference coordinate system based on the global coordinates of the known object, and all trackers observed in this observation are recorded as new known trackers.

[0098] The known object is an object whose global coordinates have been obtained before a local positioning. When the local positioning is performed for the first time, it is the base station during the observation. When the local observation is performed later, it is the first known tracker within the observation range. In each local positioning, the local coordinates of the known object in the observation are obtained. When the base station is used as the known object, the position and rotation parameters of its local coordinates and global coordinates are both 0, and the position and attitude matrices are unit matrices.

[0099] The coordinate system of the known object is a local coordinate system established by the base station or the tracker itself.

[0100] The global reference coordinate system is the coordinate system of the base station when the local positioning is performed for the first time. The position and attitude parameters of the tracker observed this time are the global coordinates of the tracker, and there is no need to perform relative coordinate transformation.

[0101] The step 4 is to globally unify the global tracker extension and the camera coordinate system.

[0102] The global tracker expansion is to adjust the observation position and observation angle of the base station in turn to change its observable area, and gradually expand until all trackers are covered. The specific method is to repeat the local tracker positioning method multiple times, and adjust the position and observation angle of the base station each time, except for the first positioning, so that the new observable area covers at least one known tracker, until the position and attitude parameters of all trackers are obtained.

[0103] The global tracker expansion process can be performed using more than one base station at the same time, or the number of cameras that can be located by the base station at a time can be increased by increasing the number of trackers.

[0104] The local tracker positioning and global tracker expansion require at least one base station and at least two trackers regardless of the number of cameras to be positioned.

[0105] After the tracker becomes a known tracker, the tracker and the base can be removed as a whole from the rigid intermediate connecting member.

[0106] After the base station changes the observation position and the observation angle, the removed tracker that is not in the observable area can be installed on the camera in the current observable area.

[0107] When the number of trackers is insufficient to be installed on all cameras covered by a local positioning operation, the process can be split into multiple local positioning processes, with trackers installed on only a small number of cameras each time for positioning.

[0108] When the cameras are far away and the observable area of ​​the base station cannot cover the trackers of at least two cameras, one or more new relay trackers are set in the middle of the two cameras, so that the base station can change its position and observation angle so that its observable area covers the tracker and at least one relay tracker installed on the camera when it is near the camera, and covers at least two relay trackers at the same time when it is in the middle of the two cameras. Through multiple position transmissions of the relay tracker, according to the local tracker positioning and global tracker expansion method, the coordinate relationship of the trackers on multiple long-distance cameras in the same coordinate system is established.

[0109] It can be found that the transfer between the coordinate systems of each camera in the present invention is not the conversion matrix constructed by the internal and external parameters obtained by calibrating each camera separately, but the conversion matrix constructed by the tracker and base station fixed on the camera. Therefore, it is not necessary to calibrate the internal and external parameters of all cameras separately before multi-camera calibration, which is convenient for changing the position and posture of some cameras during the multi-camera calibration process, and reduces the risk of the internal and external parameter errors of some cameras propagating in the multi-camera system. In addition, in this method, each new tracker performs coordinate conversion with the known tracker, and the base station only provides a transfer function for the coordinate system conversion between the two, so there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the difficulty of implementation.

[0110] On the other hand, assuming that there are 40 cameras that need to be calibrated without field of view, when the tracker is fixed to camera No. 30, the positions and postures of the following 10 cameras No. 31-40 can be adjusted. There is no need to re-calibrate individually after adjustment before continuing the calibration of the multiple cameras as in the prior art. Therefore, the solution of the present invention is more efficient in multi-camera calibration.

[0111] When the number of cameras is small or the distance is close, the positioning of the trackers on all cameras can be completed through a local positioning of the base station. The local tracker positioning process becomes a global positioning process, and the global tracker expansion is no longer performed.

[0112] According to the calibration results of the tracker coordinate system and the camera coordinate system, the position and attitude of the camera coordinate system relative to the local coordinate system of the tracker installed thereon is a fixed constant, and the position and attitude information of each camera in the global coordinate system can be obtained through the global coordinates of the tracker installed on the camera.

[0113] The method can further set the local coordinate system of one of the cameras as a reference, and through coordinate transformation, change the position and attitude parameters of all cameras to relative coordinates relative to the local coordinate system of the camera, and the position and angle values ​​of the camera become 0, and the position and attitude matrices are unit matrices.

[0114] The method may also introduce an additional reference coordinate system, change the positions and postures of all cameras into relative values ​​under the reference coordinate system, and perform an overall transformation on the collected spatial position data.

[0115] This embodiment is aimed at the spatial positioning of multiple 3D space cameras in a large range without overlapping fields of view, but is also applicable to the spatial positioning of more than two ordinary 2D plane cameras, and the spatial positioning of more than two 3D cameras in close distance with overlapping fields of view.

[0116] Before implementing this embodiment, the installation positions of multiple 3D cameras have been determined according to the requirements, and there is no need to set overlapping field of view areas between cameras in advance when designing the installation positions. According to this embodiment, the intervals between cameras can be set to be infinite and face different observation angles.

[0117] When implementing this embodiment, the required operations are generally divided into four stages, namely, installation and positioning of the tracker on the camera, calibration of the tracker coordinate system and the camera coordinate system, local tracker positioning, and global tracker expansion and global unification of the camera coordinate system.

[0118] The first implementation phase is the installation and positioning of the tracker on the camera.

[0119] refer to Figure 1 The tracker 102 and the tracker base 104 are installed and combined into a tracker and base whole 106, the rigid intermediate connecting piece 108 and the camera 110 are installed and combined into a camera and connecting piece whole 112, which is further combined into an overall structure 114 including the tracker and the camera. The whole is used in conjunction with the tracking base station 116 to obtain the spatial position and posture of the overall structure 114 relative to the base station 116.

[0120] Tracker 102 has multiple signal receiving devices in different directions on its outer surface. Base station 116 actively transmits specific signals to a certain area. When tracker 102 is within the observable range of base station 116, each signal receiving device calculates the position and attitude information of the local coordinate system of tracker 102 relative to the coordinate system of base station 116 in real time based on the distribution of signal perception strength, forming a corresponding position and attitude matrix. After adjusting the observation position and observation angle, base station 116 will re-establish the reference coordinate system at the new position and establish a tracking relationship with the tracker in the new observable area.

[0121] In some embodiments, the tracker 102 can also actively transmit signals in multiple directions and the base station 116 can sense the signal strength and distribution to obtain the relative positions of the two. The signal receiving device on the tracker 102 can also be set to multiple markers that can independently track the XYZ spatial coordinates. The base station 116 sequentially obtains the XYZ spatial coordinates of at least three markers on the tracker 102, and then obtains the spatial position and posture of the tracker 102 after combining. The electromagnetic tracking technology composed of the tracker and the base station belongs to the conventional technical means in this field and will not be repeated here. The specific implementation can refer to the following patent documents: CN107452036A-A global optimal optical tracker posture calculation method, CN102374847B-Workspace six-degree-of-freedom posture dynamic measurement device and method. It should be noted that the present invention uses optical positioning technology, which has higher accuracy than UWB (ultra-wideband technology) and can reach sub-millimeter level. Among them, in the prior art, such as CN102374847B-Workspace Six-DOF Position and Attitude Dynamic Measurement Equipment and Method, it is necessary to calculate the rotation angle of the transmitter (i.e., the base station) in order to settle the position and attitude of each tracker. In the present invention, each tracker that newly enters the current observation range is subjected to coordinate transformation with the known tracker, and the base station only provides a transit role for the coordinate system transformation between the two. Therefore, there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the difficulty of implementation.

[0122] In this embodiment, the bottom of the tracker 102 is a regular rectangular structure, the center of the bottom is the origin of the coordinate system during space tracking, two of the three coordinate axes are located on the bottom surface, and the third axis is perpendicular to the bottom surface, forming the local coordinate system of the tracker 102. The position and posture information obtained during space tracking is the pose matrix of the coordinate system relative to the coordinate system of the base station 116. The dedicated base 104 corresponding to the bottom structure of the tracker 102 is a regular rectangular parallelepiped structure, and the edges are parallel to the coordinate axes of the tracker 102. The two are accurately combined on the contact surface through positioning and connection structures to form a tracker and base as a whole 106, which will not be split thereafter and connected to the outside through the base 104 as a whole.

[0123] In this embodiment, a rigid intermediate connector 108 is used to position the overall structure 106 of the tracker 102 and the base 104 on the camera 110. The rigid intermediate connector 108 has a structure that matches the appearance of the camera 110, and is pre-fixed on the camera 1120 by means of side plane and end face contact positioning to form a camera and connector overall 112, which does not affect the installation and normal use of the camera 110 on the external support structure.

[0124] The tracking device and the base assembly 106 are positioned on the rigid intermediate connection 108 by using Figure 2The mechanical method of end face positioning and screw tightening is shown. In this method, the tracker and base assembly 106 and the rigid intermediate connector 108 are provided with four pairs of corresponding positioning holes, and the positioning holes of the rigid intermediate connector 108 are provided with threads. After the two are aligned, the four bolts 202 are tightened. This method can realize the rapid installation and removal of the tracker and base assembly 106 on the camera and connector assembly 112. This positioning can also be adopted Figure 3 The magnetic attraction method shown in the figure. In this method, three positioning pins 302 and an annular magnet 306 are set on the tracker base, and three positioning holes 304 and an annular magnet 308 are correspondingly set on the rigid intermediate connecting member 108. The positioning holes 304 and the positioning pins 302 are used to position the two, and the adsorption force between the magnets 306 and 308 is used to connect and fix the two.

[0125] In this embodiment, the tracking device and the base assembly 106 may be positioned on the rigid intermediate connector 108 using either of two methods, both of which have the same positioning effect and do not affect the camera positioning method and the final effect.

[0126] The tracker and base assembly 106 is only installed on the camera and connector assembly 112 when the position of the camera 110 needs to be obtained, and is disassembled after the position of the camera 110 is obtained. The tracker base 106 and the rigid intermediate connector 108 need to have high processing accuracy and can be freely combined between different instances.

[0127] In some embodiments, the structural design of the tracker base 104 and the rigid intermediate connector 108 is not restricted except for the areas in direct contact with the tracker 102 and the camera 110. The matching appearance and positioning structures such as square holes, round holes, V-shaped grooves, and connection and force-bearing structures such as threaded holes can be designed according to requirements.

[0128] The second implementation phase is the calibration of the tracker coordinate system and the camera coordinate system.

[0129] After the tracker and camera structure 114 is assembled, the tracker 102 and the camera 110 have a fixed relative position relationship. Figure 4 The overall structure 114 of the tracker 102-1 and the camera 110, and the combination of a planar marker 402 and the tracker 106 with a base are placed in the observable area of ​​the same base station 116. The overall structure 114 of the tracker and the camera needs to adjust the corresponding posture so that the camera 110 can capture the image of the planar marker 402 at a close distance and try to be located in the center of the image.

[0130] refer to Figure 5, the calibration of the coordinate system of the tracker 102-1 and the coordinate system of the camera 110 may include steps 502 to 520. Steps 502 to 508 generate a coordinate transformation relationship between the tracker 102-2 and the planar marker 402, steps 510 to 512 generate a coordinate transformation relationship between the tracker 102-1 and the tracker 102-2 on the overall structure 114 of the tracker and camera, and steps 514 to 516 generate a coordinate transformation relationship between the camera 110 and the planar marker 402.

[0131] Step 502 is to place the assembly 106 of the base 104B and the tracker 102-2 at the center of the planar marker 402 so that it is within the observable range of the base station 116 and the planar marker 402 is at the center of the 2D image of the camera 110 and occupies most of the image area. Step 504 is to measure the height of the tracker base 104B using a high-precision measuring tool, i.e., the distance from the connection surface of the tracker base 104B and the tracker 102-2 to the contact surface of the tracker base 104B and the planar marker 402. Step 506 is to adjust the position of the base 104B on the planar marker 402 so that the coordinate axes of the coordinate system of the tracker 102-2 and the coordinate axes of the local coordinate system of the marker 402 are parallel to each other, and the difference between the two is only the height distance of the tracker base 104B in the vertical direction of the planar marker 402, which is the origin distance between the coordinate system of the tracker 102-2 and the coordinate system of the planar marker 402. In step 508, since the coordinate system of tracker 102-2 and the coordinate system of planar marker 402 differ only in the height distance of tracker base 104B in one coordinate axis direction, the transformation matrix between the two coordinate systems can directly change a corresponding translation coordinate into the height distance of tracker base 104B based on the unit matrix.

[0132] Step 510 is to simultaneously observe the tracker 102-1 and the tracker 102-2 on the overall structure 114 of the tracker and the camera through the base station 116 to obtain their respective spatial positions, and then in step 512, obtain the relative transformation matrix between the two trackers 102-1 and 102-2 through coordinate system transformation.

[0133] In another embodiment, in order to avoid the mechanical error of measuring the height of the tracker base 104B from affecting the conversion matrix between the plane marker coordinate system and the tracker coordinate system thereon, a camera calibration method can be used to achieve this. Specifically, the coordinates of each signal receiving device on the tracker in the tracker coordinate system are known, and the camera can be used to simultaneously capture the plane marker and the tracker thereon, and then the camera can be used to obtain the conversion relationship between the tracker coordinate system and the plane marker coordinate system through image feature point matching technology, and then the coordinate system is transferred to obtain the conversion matrix between the plane marker coordinate system and the tracker coordinate system thereon.

[0134] In step 514, the 2D plane camera of the camera 110 captures the corresponding image, and simultaneously records the resolution of the image and its own internal parameters, and saves them as an electronic file. In step 516, the camera 110 extracts the pixel coordinates of the marker point on the image of the plane marker 402 through image recognition, and obtains its position and posture relative to the coordinate system of the plane marker 402 through the mapping relationship between the 2D and 3D positions, that is, the relative transformation matrix between the coordinate system of the camera 110 and the coordinate system of the plane marker 402.

[0135] In step 518, the coordinate transformation matrix between the tracker 102-2 and the plane marker 402, the coordinate transformation matrix between the tracker 102-1 and the tracker 102-2, and the coordinate transformation matrix between the camera 110 and the plane marker 402 are integrated to calculate the coordinate transformation matrix between the tracker 102-1 and the camera 110 on the overall structure 114 of the tracker and the camera. Step 520 is to store the obtained coordinate transformation matrix between the tracker 102-1 and the camera 110 as a known constant. When the coordinate position of any object between the tracker 102-1 and the camera 110 is obtained, the coordinate position of the other object can be calculated.

[0136] In this embodiment, the plane marker 402 is composed of black and white squares of equal size arranged according to a certain rule, but in some embodiments, the plane marker 402 can also be a regular circle, ellipse, or any image with complex texture, and the relative position relationship with the camera can be obtained through feature point recognition method.

[0137] In this embodiment, the relative position and posture relationship between the coordinate system of the tracker 102 and the coordinate system of the camera 110 is consistent and universal when the shapes of the tracker base 104, the rigid intermediate connector 108 and the camera 110 do not change. That is, after multiple trackers 102 and multiple cameras 110 are combined through the tracker base 104 and the rigid intermediate connector 108, the relative position between the selected tracker 102 and the camera 110 remains unchanged, and the coordinate transformation matrix remains unchanged.

[0138] In some embodiments, when the same model of tracker 102 and its base 104, rigid intermediate connector 108 and camera 110 are connected, the relative position and attitude relationship between the tracker 102 coordinate system and the camera 110 coordinate system does not change, and the position and attitude parameters of one coordinate system can be calculated from the position and attitude parameters of the other coordinate system.

[0139] Implementation phase three is local tracker positioning.

[0140] In this embodiment, each time the base station 116 is placed, the tracker 102 installed on the camera 110 in the observable area is observed and positioned once, which is a local tracker positioning process. The local tracker positioning process only locates and obtains the position of the tracker 102, and does not calculate the position of the camera 110. The calculation is only performed after the corresponding trackers on all cameras have completed positioning.

[0141] refer to Figure 6 , adjust the position and observation angle of the base station 116 so that at least two cameras 110 are within its observable range, install the tracker and the base assembly 106 for all cameras 110 in the observable area to obtain the overall structure 114 including the tracker and the camera, and directly obtain the position and posture parameters of all trackers 102 in the observable area, that is, the local coordinates relative to the base station 116 coordinate system. The base station 116 then sends these position coordinates to the computer for storage and processing.

[0142] refer to Figure 7 , the operation and algorithm flow of a single local tracker positioning may include steps 702 to 722. Step 702 is to adjust the position and observation angle of the base station 116 so that at least two cameras 110 are within its observable range, and install the tracker and the base unit 106 for all cameras 110. Step 704 is to simultaneously obtain the coordinates of each tracker 102 relative to the base station 116 coordinate system, and in step 706, send the coordinate data to the computer in real time for storage and processing. Step 708 is to determine the positioning progress. Steps 710 to 714 are performed when the local positioning is performed for the first time, and steps 716 to 720 are performed when the local positioning is not performed for the first time.

[0143] At step 710, the first local positioning is started. No tracker 102 has obtained the global coordinates yet, and the global coordinate system has not been defined. The base station 116 can be set as a known object and become the reference coordinate system for this observation. At step 712, the coordinate system of the base station 116 is defined as the global coordinate system, and at step 714, the coordinates of all trackers 102 in this observation are set as global coordinates.

[0144] In step 716, at least one local positioning has been performed and a global coordinate system has been established. Some trackers 102 have obtained global coordinates, and the coordinate system of the first known tracker 102 is selected as the reference coordinate system for this observation. In step 718, the relative position and posture of all other trackers 102 observed in this observation relative to the reference coordinate system can be obtained through coordinate transformation, and multiple coordinate transformation matrices can be obtained. In step 720, based on the obtained coordinates of other trackers 102 relative to the known tracker 102 coordinate system and the global coordinates of the known tracker 102, the global coordinates of all trackers 102 observed in this observation in the global reference coordinate system can be calculated.

[0145] Step 722 is to obtain the global coordinates of all trackers 102 observed by the base station 116 during the current positioning, record all trackers 102 observed this time as new known trackers 102, record the data and complete the local tracker positioning process.

[0146] In some embodiments, the base station 116 may simultaneously obtain the coordinates of multiple known trackers 102 during one observation, and may select one of the trackers 102 with known coordinates as a reference object for this positioning.

[0147] In some embodiments, a relay tracker is arranged between cameras that are far apart. In this case, the position of the base station 116 can be adjusted so that its observation range covers the relay tracker. The relay tracker only serves as a common positioning point for two local positioning.

[0148] The fourth implementation phase is the global tracker expansion and global unification of the camera coordinate system.

[0149] In this embodiment, the observation position and observation angle of the base station 116 are adjusted in sequence to change its observable area. In addition to the first positioning, the new observable area covers at least one known tracker. After multiple local tracker positioning processes, the observation area can be gradually expanded until it covers the trackers 102 on all cameras. This process is a global tracker expansion process.

[0150] refer to Figure 8 , the global tracker extension and camera coordinate system global unification includes steps 802 to 810. After the base station position is updated according to step 702 so that its observation range covers at least two cameras, the tracker outside the current observation area is removed in step 802 and installed on the camera in the current observation area or as a relay tracker according to step 804.

[0151] After performing a local tracker positioning according to the process at 700, it is determined in step 806 whether the positioning of the trackers on all cameras has been completed. When the positioning of the trackers on all cameras has not been completed, steps 802 to 806 and 700 are repeated. When the positioning of the trackers on all cameras has been completed, the coordinates of all cameras in a unified global coordinate system are calculated in step 808 based on the relative conversion relationship between the camera 110 and the tracker 102 installed thereon stored in step 520. In step 810, a certain camera or an additional coordinate system can be set as a reference, and the coordinates of all cameras can be converted to the new reference coordinate system, and the relative position relationship between the cameras remains unchanged.

[0152] Reference FIG. 9A to FIG. 9E , cameras 110A, 110B, 110C, 110D, 110E, and 110F are the cameras to be positioned. The distance between the cameras is far and there is no overlapping field of view. Three trackers 102A, 102B, and 102C are used. Multiple local tracker positioning is performed through the global tracker expansion process, and the position coordinates of the corresponding trackers on all cameras can be obtained.

[0153] refer to Fig. 9A , the first local tracker positioning is performed on cameras 110A, 110B, and 110C. Base station 116 is located at position 1, and its observation range covers cameras 110A, 110B, and 110C. Trackers 102A, 102B, and 102C are installed on the cameras accordingly, and the position coordinates of trackers 102A, 102B, and 102C can be obtained at one time. In this local tracker positioning, the coordinate system of base station 116 is the global coordinate system, and each tracker does not need to perform global coordinate system conversion, and the trackers on cameras 110A, 110B, and 110C are marked as known objects.

[0154] refer to Fig. 9B , a local tracker positioning is performed on cameras 110C and 110D. Base station 116 is located at position 2, and its observation range covers cameras 110C and 110D. Tracker 102A on camera 110A is disassembled and reinstalled on camera 110D, and the position coordinates of trackers 102C and 102A can be obtained at one time. In this local tracker positioning, tracker 102C is a known object. After calculating the relative position coordinates of tracker 102A relative to tracker 102C, the global coordinates of tracker 102A are calculated based on the global coordinates of tracker 102C, and the tracker on camera 110D is marked as a known object.

[0155] refer to Fig. 9C, since the camera 110D area is far away from the camera 110E and 110F area, the tracker 102B on the camera 110B is disassembled and placed near the camera 110D, and the tracker 102C on the camera 110C is disassembled and placed near the cameras 110E and 110F, and the trackers 102B and 102C are set as relay trackers, and the trackers 102A and 102B are locally positioned. The base station 116 is located at position 3, and the observation range covers the tracker 102A on the camera 110D and the relay tracker 102B, and the position coordinates of the trackers 102A and 102B can be obtained at one time. In this local tracker positioning, tracker 102A is a known object. After calculating the relative position coordinates of tracker relay 102B relative to tracker 102A, the global coordinates of relay tracker 102B are calculated based on the global coordinates of tracker 102A, and relay tracker 102B is marked as a known object.

[0156] refer to Fig.9D , the first local positioning of the relay trackers 102B and 102C is performed. The base station 116 is located at position 4, and the observation range covers the relay trackers 102B and 102C, and their position coordinates can be obtained at one time. In this local tracker positioning, the relay tracker 102B is a known object. After calculating the relative position coordinates of the relay tracker 102C relative to the relay tracker 102B, the global coordinates of the relay tracker 102C are calculated based on the global coordinates of the relay tracker 102B, and the relay tracker 102C is marked as a known object.

[0157] refer to Fig.9E , a local tracker positioning is performed on the relay tracker 102C, cameras 110E, and 110F. The base station 116 is located at position 5, and the observation range covers cameras 110E, 110F, and the relay tracker 102C. The relay tracker 102B is disassembled and reinstalled on the camera 110E, and the relay tracker 102A is disassembled and reinstalled on the camera 110F, and the position coordinates of the trackers 102C, 102B, and 102A can be obtained at one time. In this local tracker positioning, the relay tracker 102C is a known object. After calculating the relative position coordinates of the trackers 102B and 102A relative to the relay tracker 102C, the global coordinates of the trackers 102B and 102A are calculated based on the global coordinates of the relay tracker 102C, and the trackers on the cameras 110E and 110F are marked as known objects.

[0158] After calculating the positions of the trackers 102 installed on the cameras 110A, 110B, 110C, 110D, 110E, and 110F, based on the relative position transformation matrix between the camera 110 and the tracker 102 installed thereon, the coordinates of the cameras 110A, 110B, 110C, 110D, 110E, and 110F relative to the coordinate system of the base station at position 1 can be calculated, thereby completing the spatial positioning of multiple cameras at long distances and without overlapping fields of view.

[0159] In some embodiments, the local coordinate system of one of the cameras can be further set as the reference, and the position and attitude parameters of all cameras can be changed to relative coordinates relative to the local coordinate system of the camera through coordinate transformation, and the position and angle values ​​of the camera become 0, and the position and attitude matrix is ​​the unit matrix. It is also possible to introduce an additional reference coordinate system, change the position and attitude of all cameras to relative values ​​under the reference coordinate system, and transform the collected spatial position data as a whole.

[0160] In this embodiment, a base station 116 and three trackers 102 are used for camera positioning. After the tracker 102 becomes a known tracker, the tracker and base assembly 106 can be removed from the rigid intermediate connector and installed on a new camera 110 that needs to be observed or at a relay position. In some embodiments, regardless of the number of cameras to be positioned, at least one base station and at least two trackers are required. The global tracker expansion process can be performed using more than one base station at the same time, and the number of cameras that can be positioned by the base station at a time can also be increased by increasing the number of trackers, thereby speeding up the positioning process.

[0161] In some embodiments, when the number of trackers is insufficient to be installed on all cameras covered by a local positioning, the local positioning process can be split into multiple local positioning processes, with trackers installed on only a small number of cameras for positioning each time.

[0162] In this embodiment, when the distance between the cameras 110 is too far so that the observable area of ​​the base station 116 cannot cover the trackers 102 of at least two cameras 110, one or more new relay trackers 102 are set in the middle of the two cameras 110, so that the base station 116 can change its position and observation angle so that its observable area covers the tracker 102 installed on the camera 110 and at least one relay tracker 102 when it is near the camera 110, and covers at least two relay trackers 102 at the same time when it is in the middle of the two cameras 110. Through multiple position transmissions of the relay trackers 102, the coordinate relationship of the trackers 102 on multiple long-distance cameras 110 in the same coordinate system can be established.

[0163] In some embodiments, when the number of cameras is small or the distance is close, the positioning of the trackers on all cameras can be completed through a local positioning of the base station. The local tracker positioning process becomes a global positioning process, and the global tracker expansion and coordinate system are no longer performed.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A multi-camera calibration system with non-overlapping fields of view, It is characterized in that The system comprises: At least two cameras to be calibrated; At least two trackers, each tracker can be detachably rigidly connected to each camera, and after connection, the tracker and each camera have the same relative position and posture relationship, and the tracker is provided with a plurality of signal receiving devices in different directions on the outer surface, and is used in conjunction with a base station that transmits signals outward, and is used to obtain the position and posture information of the tracker's local coordinate system relative to the base station coordinate system according to the distribution of the intensity of the received signal, and form a corresponding position and posture matrix; A base station, wherein the signal emitted by the base station is dispersedly transmitted outward from a point, and is used to establish a real-time tracking relationship with the tracker in the observable area based on its own coordinate system within the signal coverage angle and range; and A combination, comprising a planar marker and a tracker whose relative positions and postures are known, is used to obtain, through coordinate system transformation, the position and posture relationship between the camera coordinate system and the local coordinate system of the tracker installed thereon after the tracker is installed on the camera.

2. The system according to claim 1, It is characterized in that The planar marker is a planar marker that can be identified through image positioning. The tracker is placed on the planar marker, and the relative position relationship between the tracker coordinate system and the planar marker coordinate system can be obtained through physical measurement.

3. The system according to claim 1, It is characterized in that It also includes a tracker base and a rigid intermediate connector, the tracker is fixedly connected to the tracker base, the rigid intermediate connector is adapted to the shape of each camera and is fixedly connected thereto, each camera has a rigid connector connected thereto as a whole, and the tracker can be rigidly mounted to the intermediate connector corresponding to each camera through the tracker base.

4. A multi-camera calibration method without overlapping fields of view, It is characterized in that The method is applied to the system described in claim 1, and the method comprises: Camera and tracker calibration step: based on the assembly, determining a first transformation relationship between a camera coordinate system of each camera and a tracker coordinate system mounted thereon; Local tracker positioning step: adjust the position and observation angle of the base station so that at least two cameras are within the observable range of the base station, install trackers for all cameras in the observable area, obtain the position and attitude parameters of all trackers in the observable area relative to the base station, and calculate the transformation relationship between the tracker coordinate system of each tracker and the coordinate system of the known object in this observation based on the observed position and attitude parameters of all trackers relative to the base station; wherein the coordinate system of the known object is a local coordinate system established for the base station or the tracker itself; the transformation relationship between the coordinate system of the known object and the global coordinate system is known; when the local tracker positioning step is performed for the first time, the known object is the base station at the time of the observation, and when the local tracker positioning step is performed subsequently, it is the first known tracker within the observation range; Global tracker expansion positioning step: repeating the local tracker positioning step multiple times, adjusting the position and observation angle of the base station each time, so that the new observable area covers at least one known tracker except the first positioning, until the transformation relationship between the tracker coordinate system of all trackers and the tracker coordinate system of the known tracker in each observation is obtained; The step of global unification of the camera coordinate system: based on the first transformation relationship and the transformation relationship between the coordinate system of the tracker installed on each camera and the global coordinate system, a second transformation relationship between the camera coordinate system of each camera and the global coordinate system is determined.

5. The method according to claim 4, It is characterized in that The camera and tracker calibration steps specifically include: Acquire a third transformation relationship between a plane marker coordinate system of the plane marker in the assembly and a tracker coordinate system of the tracker; Using an image of the plane marker captured by any one of the multiple cameras, calibrate the corresponding camera to obtain a fourth transformation relationship between a camera coordinate system of the camera and a plane marker coordinate system of the plane marker; Obtaining, through the base station, a fifth transformation relationship between a tracker coordinate system of the tracker on the assembly and a tracker coordinate system of the tracker on the camera; Based on the third transformation relationship, the fourth transformation relationship and the fifth transformation relationship, a first transformation relationship between the camera coordinate system of each camera and the tracker coordinate system installed thereon is calculated.

6. The method according to claim 5, It is characterized in that The step of acquiring the third transformation relationship between the plane marker coordinate system of the plane marker in the assembly and the tracker coordinate system of the tracker specifically includes: Obtaining a third change relationship by measuring the relative position and posture between the coordinate system of the planar marker and the tracker coordinate system of the tracker; Among them, the planar marker is any two-dimensional picture that can be positioned through image recognition, and the two coordinate axes of the planar marker coordinate system are set on the picture plane, and the corresponding coordinate origin and the two coordinate axis directions are marked for positioning the tracker in the combination; the tracker in the combination is set on the planar marker.

7. The method according to claim 4, It is characterized in that The method further comprises: When the number of trackers is less than the number of all cameras that need to be calibrated, after the base station changes its position and angle, the trackers on the cameras that are not in the observable area are removed and the removed trackers that are not in the observable area are installed on the cameras in the current observable area.

8. The method according to claim 4, It is characterized in that The method further comprises: When the cameras are far apart and the observable area of ​​the base station cannot cover the trackers of at least two cameras, one or more new relay trackers are set in the middle of the two cameras, so that the base station can change its position and observation angle so that the observable area of ​​the base station can cover the tracker installed on the camera and at least one relay tracker, or can simultaneously cover at least two relay trackers located in the middle of the two cameras; Through multiple position transfers of the relay tracker, according to the local tracker positioning step and the global tracker expansion positioning step, the transformation relationship between the tracker coordinate system of all trackers and the tracker coordinate system of the known tracker in each observation is obtained.

9. The method according to claim 4, It is characterized in that The method further comprises: Set the local coordinate system of one of the cameras as the reference, and transform the position and attitude parameters of all cameras into relative position and attitude parameters relative to the local coordinate system of the camera through coordinate transformation.

10. The method according to claim 4, It is characterized in that The method further comprises: An additional reference coordinate system is introduced, and the position and attitude parameters of all cameras are transformed into relative position and attitude parameters relative to the reference coordinate system through coordinate transformation.

Images