A multi-camera calibration system and method with non-overlapping fields of view

Abstract

The present invention discloses a multi-camera calibration system and method without overlapping fields of view. This method uses a tracker composed of multiple receivers in cooperation with a base station as a conversion bridge to achieve coordinate system conversion between multiple trackers, as well as coordinate system conversion between a camera and a tracker, and coordinate system conversion between cameras, so as to realize multi-camera calibration without a common field of view. Main advantages / differences: For cameras of the same model, only the internal parameters of one camera need to be calibrated before the start of calibration, and there is no need to measure the internal parameters of the camera during the actual calibration process. The electrical signal processing replaces the image processing method, with fast signal acquisition and no need for a cumbersome image acquisition process, significantly shortening the calibration time. The calibration accuracy is ensured by the external structure and it can adapt to environmental changes; there is no need for a calibration board, the equipment is simple and portable, and it can be quickly deployed in a new environment. The number of cameras, installation positions and angles, installation intervals, etc. can be designed according to actual needs, without the need to pre-consider the impact on the feasibility of calibration as in visual calibration.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and particularly to a multi-camera calibration system and method with non-overlapping fields of view. Background Art

[0002] Computer vision is based on the physical environment information obtained by cameras, and can perform high-precision recognition, tracking and segmentation of people and objects in a scene, serving requirements such as environmental perception, scene understanding, and intelligent interaction, creating huge economic and social benefits, and being a typical feature of the new generation of information technology. Among them, three-dimensional (3D) cameras, due to their spatial perception ability, can obtain the relative depth information of objects in the scene while perceiving color images, and thus are widely used in three-dimensional scenes such as three-dimensional reconstruction, object segmentation, and spatial tracking. Its typical usage method is to superimpose the 3D physical scene information obtained by multiple cameras according to their respective actual spatial positions and postures, supplement the field of view of a single camera from different directions, and expand the perceivable scene range through data stitching.

[0003] In this process, one of the key steps is to obtain the relative position and attitude information between cameras, that is, to obtain the positions and postures of the local coordinate systems of all cameras in a unified global coordinate system. The currently more commonly used method is binocular calibration. By setting recognizable markers in the overlapping field of view area between every two cameras, the relative position and attitude relationship between the camera and the marker are respectively obtained based on image processing methods, so as to calculate the relative position and attitude relationship between every two cameras, and the relative position and attitude relationship between all cameras is obtained through coordinate transformation. However, the premise of this method is that there must be an overlapping field of view between cameras, which is not only inefficient but also fails in scenarios with a large range of non-overlapping fields of view such as discrete camera distribution, long distance, and large viewing angle deviation.

[0004] After retrieval, the non-overlapping field of view multi-camera calibration schemes in the prior art can be roughly divided into the following three technical routes:

[0005] The first one is to unify multiple calibration plates into the same coordinate system to obtain the transformation relationship between coordinate systems and realize the coordinate transformation between non-overlapping field of view cameras. As described in Reference 1, a calibration method using multiple small calibration plates is used. First, the internal parameters of each camera are calibrated using the Zhang Zhengyou method; then, the relative position relationship of the small calibration plates is measured using a total station, and the small calibration plates are unified into the same coordinate system and integrated into a large calibration plate, and the pose relationship between each camera and the integrated calibration plate is calculated respectively; finally, the external parameters between multiple cameras are obtained by unifying the coordinate system.

[0006] Second, move the camera so that it can capture the same target, so as to obtain the transformation relationship between coordinate systems, thereby realizing the transfer of coordinate systems between cameras with non-overlapping fields of view. As described in Reference 2, fix the planar calibration board and the two-axis turntable, fix multiple cameras on the two-axis turntable, establish a world coordinate system with the upper left corner of the target as the origin, and use a single camera to obtain multiple images at multiple positions by rotating the turntable itself to determine the relationship between the world coordinate system and the turntable coordinate system; secondly, rotate the turntable so that the target enters the field of view of each camera respectively to complete the external parameter calibration of the single camera; finally, solve the relative position relationship between each camera according to the relationship between the world coordinate system and the turntable coordinate system and the relationship between the world coordinate system and each camera coordinate system.

[0007] Third, introduce an auxiliary camera to capture all calibration boards to obtain the transformation relationship between coordinate systems and realize the transfer of coordinate systems between cameras with non-overlapping fields of view. Its principle is as described in Document 3, and the rotation matrix and translation vector between each of the cameras to be calibrated and the auxiliary camera are obtained respectively.

[0008] Through the analysis of the multi-camera calibration technology with non-overlapping fields of view in the existing technical routes, the inventor found that all these three routes in the existing technology require each camera to separately take pictures of the calibration board or target first, and then use methods such as Zhang Zhengyou calibration method to determine the internal and external parameters of each camera by using the photos. After that, the external parameters of each camera can be used to construct a transformation matrix to obtain the coordinate system transformation relationship between each camera, that is, to realize the transfer of coordinate systems between cameras with non-overlapping fields of view (for the specific formula principle description, please refer to Chapter 3 of Reference 4 on multi-camera calibration). However, there are the following deficiencies in constructing the transformation matrix between multi-camera coordinate systems through the internal and external parameters of each camera:

[0009] 1. It is required that each camera must be calibrated separately before multi-camera calibration to record the internal parameters of each camera, and then these cameras can be combined into a new system for multi-camera calibration without a common field of view.

[0010] 2. Due to processing reasons, lens distortion includes radial distortion, tangential distortion, decentering distortion, and thin prism distortion, etc. During the calibration process of a single camera, considering the algorithm complexity, generally only the radial distortion of the lens is considered, while other distortion parameters are ignored. For a single camera, the distortion parameters have little impact on subsequent tasks such as 3D reconstruction, object segmentation, and spatial tracking. However, in multi-camera calibration, the distortion error of the camera farthest from the reference coordinate system in the coordinate system transfer operation will be accumulated and amplified. For example, when the camera numbered 1 is used as the reference camera and there are 40 cameras, the coordinate transformation of the 40th camera to the reference camera depends on the parameters of a total of 38 cameras numbered from 2 to 39 for coordinate transfer, resulting in the accumulation and amplification of the distortion error of the 40th camera. At the same time, if these 40 cameras are separately calibrated using different calibration boards. It may also lead to poor consistency of the parameters obtained from different camera calibrations, and these inconsistent parameters will bring uncertainties during the coordinate transfer in multi-camera calibration. For example, when the calibration accuracy of the 5th camera is poor, then all subsequent cameras that need to convert their parameters to the reference coordinate system camera will have the same poor accuracy. Generally speaking, when using the internal and external parameters of the camera as the parameters of the coordinate system transfer matrix, the errors carried during the calibration process may be propagated into the multi-camera calibration system.

[0011] 3. The above methods must use a calibration board, which has deficiencies in terms of portability, ease of use, etc., and has clear constraints on the installation position and direction of the camera.

[0012] Reference 1: Wang Anran, Hao Xiangyang, Cheng Chuanqi, Jia Kaikai. A Multi-Camera External Parameter Calibration Method Using Multiple Small Calibration Boards [J]. Geomatics & Spatial Information Technology, 2019, 42(06): 222 - 225 + 229.

[0013] Reference 2: Lu Yanan, Wan Zijin, Wang Xiangjun. A Method for Solving the Position Relationship of Cameras without Common Field of View [J]. Journal of Applied Optics, 2017, 38(03): 400 - 405.

[0014] Reference 3: Patent Publication Number, CN116188591A, Patent Name, Multi-Camera Global Calibration Method and Device, Electronic Equipment.

[0015] Reference 4: Li Wenjing. Research on Multi-Camera Calibration and 3D Reconstruction Technology [D]. Xi'an University of Technology, 2022. DOI: 10.27398 / d.cnki.gxalu.2022.000869. Summary of the Invention

[0016] The present invention provides a multi-camera calibration system and method without overlapping fields of view, which can obtain the position and attitude parameters of the local coordinate systems of multiple cameras in a common unified coordinate system in sequence for the situation of large-range non-overlapping fields of view with multiple cameras discretely distributed, far apart, and large view angle deviations, and can perform calibration without separately calibrating the internal parameters of all cameras individually before multi-camera calibration, and reduce the risk of the propagation of internal parameter errors of some of the cameras in the multi-camera system.

[0017] The multi-camera calibration described in the present invention is for external camera parameter calibration, that is, to obtain the position and attitude descriptions of the local coordinate systems of all cameras in a unified reference coordinate system, and this description is usually a 4×4 position and attitude matrix, including 3 position parameters and 9 rotation attitude parameters.

[0018] To achieve the above object, in a first aspect, the present invention provides a multi-camera calibration system without overlapping fields of view, and the system includes:

[0019] At least two cameras to be calibrated for external parameters;

[0020] At least two trackers, which can be detachably and rigidly connected to each camera, and after connection, the trackers and each camera have the same relative position and attitude relationship. The trackers are provided with a plurality of signal receiving devices in different directions on the outer surface, which are used in cooperation with a base station that emits signals outward, and are used to obtain the position and attitude information of the local coordinate system of the tracker relative to the base station coordinate system according to the distribution of the intensity of the received signals, and form a corresponding position and attitude matrix;

[0021] A base station, the signal emitted by the base station is dispersed and emitted outward from a point, and within the signal coverage angle and range, it is used to establish a real-time tracking relationship with the trackers within the observable area based on its own coordinate system; and

[0022] An assembly, the assembly includes a planar marker and a tracker with known relative positions and attitudes, and is used to obtain the position and attitude relationship between the camera coordinate system and the local coordinate system of the tracker mounted on it after installing the tracker on the camera through coordinate transformation.

[0023] Compared with the prior art, the internal and external parameters obtained by separately calibrating each camera are used to establish the transformation matrix for sequentially transmitting the coordinate systems between adjacent cameras, which causes the inconvenience of separately calibrating each camera in advance and the large-scale propagation of the calibration errors of each camera in the multi-camera system. A multi-camera calibration system with non-overlapping fields of view provided in the present invention uses a tracker composed of multiple photoelectric receivers and a base station as a conversion bridge to realize the coordinate system conversion between multiple trackers, the coordinate system conversion between the camera and the tracker, and the coordinate system conversion between cameras, and finally realizes the calibration of multi-cameras without a common field of view, without using the matrix composed of the internal and external parameters of multiple cameras as a conversion bridge, and can avoid separately calibrating the internal and external parameters of all cameras separately before multi-camera calibration, which is convenient for changing the position and attitude of some cameras during the multi-camera calibration process and reduces the risk of the propagation of the internal and external parameter errors of some cameras in the multi-camera system.

[0024] As a further improvement, the planar marker is a rectangular planar marker that can be identified by image positioning. The tracker is placed on the planar marker, and the relative position relationship between the tracker coordinate system and the planar marker coordinate system can be obtained through physical measurement.

[0025] Since there is a height difference of the tracker base in the vertical direction of the planar marker between the two, and the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the planar marker coordinate system, the coordinate system conversion between the two is convenient and fast without complex calculations.

[0026] As a further improvement, it further includes a tracker base and a rigid intermediate connector. The tracker is fixedly connected to the tracker base. The rigid intermediate connector is adapted to the outer shape of each camera and fixedly connected thereto. Each camera has a rigid connector connected thereto as a whole, and the tracker can be rigidly mounted on the corresponding intermediate connector of each camera through the tracker base.

[0027] Considering that camera manufacturers will manufacture the camera shapes into various models for commercial reasons, in order to make the tracker connect to the camera more conveniently, firmly and accurately, a rigid intermediate piece is specially designed to connect the camera and the tracker.

[0028] In a second aspect, the present invention provides a multi-camera calibration method for non-overlapping fields of view. The method is applied to the above system, and the method includes:

[0029] Camera and tracker calibration step: Based on the combination, determine the first transformation relationship between the camera coordinate system of each camera and the tracker coordinate system mounted thereon;

[0030] Local Tracker Positioning Steps: Adjust the position and observation angle of the base station so that at least two cameras are within the observable range of the base station. After installing trackers on all cameras within the observable area, obtain the position and attitude parameters of all trackers within the observable area relative to the base station, and calculate the transformation relationship between the tracker coordinate system of each tracker and the coordinate system of the known object in this observation based on the observed position and attitude parameters of all trackers relative to the base station; among them, the coordinate system of the known object is a local coordinate system established for the base station or the tracker itself; the transformation relationship between the coordinate system of the known object and the global coordinate system is known; when performing the local tracker positioning steps for the first time, the known object is the base station at the time of this observation, and the first known tracker within the observable range for subsequent local tracker positioning steps.

[0031] Global Tracker Extended Positioning Steps: Repeat the local tracker positioning steps multiple times. Each time, adjust the position and observation angle of the base station. Except for the first positioning, make the new observable area cover at least one known tracker until the transformation relationships between the tracker coordinate systems of all trackers and the tracker coordinate systems of the known trackers in each observation are obtained.

[0032] Camera Coordinate System Global Unification Steps: Based on the first transformation relationship and the transformation relationships between the tracker coordinate systems corresponding to each camera and the global coordinate system, determine the second transformation relationship between the camera coordinate system of each camera and the global coordinate system.

[0033] Compared with the prior art, it is necessary to obtain the internal and external parameters through separate calibration of each camera and then establish the transformation matrix for sequentially transmitting the coordinate systems between adjacent cameras, which causes the inconvenience of separately calibrating each camera in advance and the possible large-scale propagation of the calibration errors of each camera in the multi-camera system. A multi-camera calibration method without overlapping fields of view provided in the present invention uses a tracker composed of multiple photoelectric receivers in cooperation with a base station as a conversion bridge to achieve coordinate conversion between multiple trackers, coordinate conversion between the camera and the tracker, and coordinate conversion between cameras, and finally realizes multi-camera calibration without a common field of view, without using a matrix composed of the internal and external parameters of multiple cameras as a conversion bridge, can avoid separately calibrating the internal and external parameters of all cameras separately before multi-camera calibration, is convenient for changing the position and attitude of some cameras during multi-camera calibration, and reduces the risk of propagation of the internal and external parameter errors of some cameras in the multi-camera system. And in this method, each new tracker performs coordinate conversion with a known tracker, and the base station only provides a transfer function for the two to perform coordinate system conversion, so there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the implementation difficulty.

[0034] As a further step, the camera and tracker calibration steps specifically include:

[0035] Obtain the third transformation relationship between the planar marker coordinate system of the planar marker in the combined body and the tracker coordinate system of the tracker;

[0036] Calibrate the corresponding camera through the image of the planar marker captured by any one of multiple cameras, and obtain the fourth transformation relationship between the camera coordinate system of the camera and the planar marker coordinate system of the planar marker;

[0037] Obtain the fifth transformation relationship between the tracker coordinate system of the tracker on the combined body and the tracker coordinate system of the tracker on the camera through the base station;

[0038] Based on the third transformation relationship, the fourth transformation relationship, and the fifth transformation relationship, calculate the first transformation relationship between the camera coordinate system of each camera and the tracker coordinate system installed thereon.

[0039] Specifically, with the help of the combined body, the first transformation relationship between the camera coordinate system and the tracker coordinate system installed thereon can be quickly established, and the operation is simple and fast.

[0040] As a further improvement, the obtaining of the third transformation relationship between the planar marker coordinate system of the planar marker in the combined body and the tracker coordinate system of the tracker specifically includes:

[0041] Obtain the third transformation relationship between the planar marker coordinate system of the planar marker in the combined body and the tracker coordinate system of the tracker by measuring the height of the tracker base;

[0042] Wherein, the planar marker is a rectangular planar marker that can be identified by image positioning. The tracker is installed on the tracker base, and the whole formed by the tracker and the tracker base is placed at the center position of the planar marker. Wherein, the origin of the local coordinate system of the planar marker is defined at its center position, the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the planar marker coordinate system, and there is a height distance of the tracker base between them in the vertical direction of the planar marker.

[0043] Since there is a height distance of the tracker base between them in the vertical direction of the planar marker, and the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the planar marker coordinate system, the coordinate system conversion between them is convenient and fast, without complex calculations.

[0044] As a more general implementation method, the obtaining of the third transformation relationship between the planar marker coordinate system of the planar marker in the combined body and the tracker coordinate system of the tracker specifically includes:

[0045] Obtain the third variation relationship by measuring the relative position and attitude between the coordinate system of the planar marker and the tracker coordinate system of the tracker;

[0046] Wherein, the planar marker is any two-dimensional picture that can be located through image recognition. The two coordinate axes of the planar marker coordinate system are set on the picture plane, and the corresponding coordinate origin and the directions of the two coordinate axes are marked for locating the tracker in the combined body; the tracker in the combined body is set on the planar marker.

[0047] In this implementation, the two-dimensional picture can be of any shape, and the tracker in the combined body can also be set at any position on the planar marker, as long as the third transformation relationship can be obtained according to the measurement of the actual spatial position. Therefore, this implementation has stronger versatility.

[0048] As a further improvement, the method further includes:

[0049] When the number of trackers is less than the number of all cameras to be calibrated, after the base station changes its position and angle, remove the trackers on the cameras that are not in the observable area and install the removed trackers that are not in the observable area on the cameras in the current observable area.

[0050] Through the detachable design, the number of required trackers is greatly saved.

[0051] As a further improvement, the method further includes:

[0052] When the observable area of the base station cannot cover the trackers of at least two cameras due to the cameras being far apart, set one or more new relay trackers at the middle position between the two cameras, so that the base station can, by changing its position and observation angle, make the observable area of the base station cover the tracker installed on the camera and at least one relay tracker or be able to cover at least two relay trackers located at the middle position between the two cameras at the same time;

[0053] Through multiple position transmissions of the relay tracker, obtain the transformation relationship between the tracker coordinate system of all trackers and the tracker coordinate system of the known trackers in each observation according to the local tracker positioning steps and the global tracker extended positioning steps.

[0054] By adopting the scheme of the relay tracker, the applicable degree of the number and range of cameras in this method can be greatly increased.

[0055] As a further improvement, the method further includes:

[0056] Set the local coordinate system of one of the cameras as the reference, and through coordinate transformation, transform the position and attitude parameters of all cameras into relative position and attitude parameters with respect to the local coordinate system of this camera.

[0057] As a further improvement, the method further includes:

[0058] Introduce an additional reference coordinate system, and through coordinate transformation, transform the position and attitude parameters of all cameras into relative position and attitude parameters with respect to this reference coordinate system.

[0059] Since the tracker in the present invention can conveniently and quickly obtain its position and attitude information through the base station, compared with the prior art solution of using the internal and external parameters of the camera, it is easier to align with the introduced additional reference coordinate system.

[0060] Generally speaking, a multi-camera calibration system and method without overlapping fields of view provided by the present invention at least have the following main advantages / differences:

[0061] 1. For cameras of the same model, only the internal parameters of one camera need to be calibrated before the start of calibration, and there is no need to measure the internal parameters of the camera during the actual calibration process.

[0062] 2. The electrical signal processing replaces the image processing method. The signal acquisition is fast, there is no need for a cumbersome image acquisition process, and the calibration time is significantly shortened.

[0063] 3. The calibration accuracy is ensured by the external shape structure, and it can adapt to environmental changes; there is no need for a calibration board, the equipment is simple and portable, and it can be quickly deployed in a new environment.

[0064] 4. The number of cameras, the installation position and angle, the installation interval, etc. can be designed according to actual needs, and there is no need to consider the impact on the feasibility of calibration as in visual calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the installation combination of the tracker, the tracker base, the rigid intermediate connector and the camera in an embodiment;

[0066] Figure 2 It is a schematic diagram of the tracker and its base mechanically installed on the rigid intermediate connector in an embodiment;

[0067] Figure 3 It is a schematic diagram of the tracker and its base magnetically installed on the rigid intermediate connector in an embodiment;

[0068] Figure 4 It is a schematic diagram of the scene where the camera and the tracker installed thereon perform coordinate calibration by means of the combination of the planar marker and the tracker in an embodiment;

[0069] Figure 5 Flow chart for calibrating the coordinate system of a camera and a tracker mounted thereon in an embodiment

[0070] Figure 6 Schematic diagram of single - shot local tracker positioning in an embodiment

[0071] Figure 7 Flow chart for single - shot local tracker positioning in an embodiment

[0072] Figure 8 Flow chart for global tracker extension and global unification of the camera coordinate system in an embodiment

[0073] Figure 9A Schematic diagram of global tracker extension in an embodiment

[0074] Figure 9B Schematic diagram of global tracker extension in an embodiment

[0075] Figure 9C Schematic diagram of global tracker extension in an embodiment

[0076] Figure 9D Schematic diagram of global tracker extension in an embodiment

[0077] Figure 9E Schematic diagram of global tracker extension in an embodiment

[0078] The realization of the object, functional features and advantages of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0079] This part will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The role of the drawings is to supplement the description in the text part of the specification, enabling people to intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it should not be construed as a limitation on the protection scope of the present invention.

[0080] The method for multi - camera spatial positioning with a large - range non - overlapping field of view according to the present invention has four steps, including the installation and positioning of the tracker on the camera, the calibration of the tracker coordinate system and the camera coordinate system, the local tracker positioning, and the global tracker extension and global unification of the camera coordinate system.

[0081] The first step is the installation and positioning of the tracker on the camera.

[0082] The tracker is provided with a plurality of signal receiving devices on its outer surface in different directions. When used in cooperation with a base station that emits signals outward, the position and attitude information of the local coordinate system of the tracker relative to the base station coordinate system can be obtained according to the distribution of the intensity of the received signals, and a corresponding position and attitude matrix can be formed. Its bottom can be a regular planar structure such as a regular rectangle or a circle, and regular positioning structures such as square holes, round holes, V-shaped grooves, and threaded holes can be provided. The center position of its bottom is the origin of the coordinate system during spatial tracking. Two of the three coordinate axes are located on the bottom surface, and the third axis is perpendicular to the bottom surface, constituting the local coordinate system on the tracker. The position and attitude information obtained during spatial tracking is the pose matrix of this coordinate system relative to the base station coordinate system.

[0083] The signals emitted by the base station are emitted outward in a scattered manner from a point. Within the signal coverage angle and range, with its own coordinate system as the reference, a real-time tracking relationship is established for the trackers within the observable area. After adjusting the observation position and observation angle, the base station re-establishes a reference coordinate system at its location and establishes a tracking relationship for the trackers within the new observable area.

[0084] The tracker is provided with a special base, which is in a regular cuboid structure. Through the positioning and connection structures at the bottom of the tracker, it is precisely combined with the base. The edges of the cuboid of the base are parallel to the coordinate axes of the tracker. After the two are connected, they are no longer disassembled and are connected externally as a whole.

[0085] The base of the tracker can be provided with structures such as positioning holes and positioning pins. Except for the parts connected to the tracker, the rest of the structures are not restricted, and the positioning and connection structures can be specifically designed according to the external connection method.

[0086] The positioning of the overall structure of the tracker and the base on the camera can be achieved with the help of a rigid intermediate connector with a special shape. First, the rigid intermediate connector is positioned on the outer surface of the camera, and then the tracker and the base as a whole are positioned and connected to the rigid intermediate connector, so that the tracker and the camera have a fixed relative position relationship.

[0087] The rigid intermediate connector has a structure adapted to the shape of the camera and is installed and pre-fixed on each camera through planar, end-face contact positioning, and hole-shaft constraint positioning methods.

[0088] The positioning of the tracker and the base as a whole on the rigid intermediate connector can adopt mechanical methods such as end-face positioning and screwing tight for fixation. This positioning method can also adopt a magnetic attraction method. Positioning pins and magnets are provided on the base of the tracker, and positioning holes and opposite-pole magnets are provided at the corresponding positions on the rigid intermediate connector. Connection and fixation are achieved through hole-shaft cooperation positioning and magnetic attraction.

[0089] The overall connection and fixation of the tracker and the base on the rigid intermediate connector is a detachable structure, which is installed when the camera position needs to be obtained and disassembled after the position is obtained.

[0090] Step 2 is the calibration of the tracker coordinate system and the camera coordinate system.

[0091] The camera coordinate system is the reference coordinate system when the camera acquires spatial depth information (the 3D camera consists of a depth sensor (such as a laser scanner or a structured light scanner, due to acquiring depth images) and a visible light sensor (i.e., a camera, used to acquire color images). First, spatially align the depth image and the color image (i.e., coordinate system transformation), and then the depth information acquired by the depth sensor can be converted to the camera coordinate system), and it is also the local coordinate system during 3D physical scene stitching, defined on the color image sensor (i.e., the visible light image sensor) of the camera.

[0092] The calibration of the tracker coordinate system and the camera coordinate system is to obtain the relative position and attitude relationship between the tracker coordinate system and the camera coordinate system after the overall tracker and base are positioned and installed on the rigid intermediate connector. The specific method is to set up a planar marker and tracker combination with a known relative position and attitude. The base station simultaneously acquires the coordinates of the tracker on the combination and the tracker on the camera, and obtains the position and attitude of the tracker installed on the camera relative to the local coordinate system of the planar marker through coordinate system transformation. After the camera acquires its position and attitude relative to the local coordinate system of the planar marker through image recognition, the position and attitude of the camera coordinate system relative to the local coordinate system of the tracker installed on it can be obtained through coordinate system transformation.

[0093] The planar marker and tracker combination includes a rectangular planar marker that can be located and recognized through color images and an overall tracker and base placed at the center of the marker. The rectangular planar marker defines its own local coordinate system at its center position. The coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the marker local coordinate system, and the two only differ by the height of the tracker base in the direction perpendicular to the planar marker. The relative transformation matrix between the two coordinate systems can be obtained by measuring the height of the tracker base.

[0094] The planar marker can be a regular checkerboard, an ellipse, or any image with complex textures, and the relative position relationship with the camera can be obtained through feature point recognition methods.

[0095] The relative position and attitude relationship between the tracker coordinate system and the camera coordinate system is consistent and universal when the shapes of the tracker base, the rigid intermediate connector, and the camera do not change. When connecting trackers of the same model with their bases, rigid intermediate connectors, and cameras, the relative position and attitude relationship between the tracker coordinate system and the camera coordinate system do not change, and the position and attitude parameters of one coordinate system can be calculated from those of the other coordinate system.

[0096] Step 3 is local tracker positioning.

[0097] The local tracker positioning is to use the base station to perform a single positioning of the trackers within its observable area to obtain the coordinates of all trackers. The specific method is to adjust the position and observation angle of the base station so that at least two cameras are within their observable ranges, install trackers for all cameras within this observable area, and directly obtain the position and attitude parameters of all trackers within the observable area, that is, the local coordinates relative to the base station coordinate system. Subsequently, convert the local coordinates of all observed trackers into relative coordinates with respect to the coordinate system of the known object in this observation, and further calculate the coordinates of all observed trackers in the global reference coordinate system based on the global coordinates of this known object. Denote all observed trackers in this observation as new known trackers.

[0098] The known object is an object whose global coordinates have been obtained before a local positioning. It is the base station during the first local positioning observation, and the first known tracker within the observable range during subsequent local observations. The local coordinates of this known object in each local positioning observation are obtained. When the base station is used as the known object, both the position and rotation parameters of its local coordinates and global coordinates are 0, and the position and attitude matrix is the identity matrix.

[0099] The coordinate system of the known object is a local coordinate system established by the base station or the tracker itself.

[0100] The global reference coordinate system is the coordinate system of the base station during the first local positioning. The position and attitude parameters of the observed trackers in this observation are their global coordinates, and no relative coordinate transformation is required.

[0101] Step 4 is global tracker expansion and global unification of the camera coordinate system.

[0102] The global tracker extension is to sequentially adjust the observation position and observation angle of the base station to change its observable area, and gradually expand it until it covers all trackers. The specific method is to repeatedly perform the local tracker positioning method multiple times. Each time, adjust the position and observation angle of the base station. Except for the first positioning, make the new observable area cover at least one known tracker until the position and attitude parameters of all trackers are obtained.

[0103] The global tracker extension process can be carried out simultaneously using more than one base station, or by increasing the number of trackers and thus increasing the number of cameras that can be located by the base station each time.

[0104] For the local tracker positioning and global tracker extension, regardless of the number of cameras to be located, at least one base station and at least two trackers are required.

[0105] After the tracker becomes a known tracker, the tracker and the base can be removed from the rigid intermediate connecting member as a whole.

[0106] After the base station changes the observation position and observation angle, the removed trackers that are not in the observable area can be installed on the cameras in the current observable area.

[0107] When the number of trackers is not enough to be installed on all cameras covered by a single local positioning, it can be split into multiple local positioning processes, and only a small number of cameras are installed with trackers for positioning each time.

[0108] When the distance between the cameras is so far that the observable area of the base station cannot cover the trackers of at least two cameras, one or more new relay trackers are set at the middle position between the two cameras, so that the base station can change its position and observation angle to make its observable area cover the tracker installed on the camera and at least one relay tracker when it is near the position of the camera, and cover at least two relay trackers simultaneously when it is at the middle position between the two cameras. Through multiple position transmissions of the relay trackers, according to the local tracker positioning and global tracker extension methods, the coordinate relationships of the trackers on multiple long-distance cameras in the same coordinate system are established.

[0109] It can be found that in the present invention, the transformation matrix for the transfer between camera coordinate systems is not constructed by the internal and external parameters obtained through the individual calibration of each camera, but by the tracker fixed on the camera and the base station. Therefore, it is not necessary to separately calibrate the internal and external parameters of all cameras individually before multi-camera calibration, which facilitates the modification of the position and attitude of some cameras during the multi-camera calibration process, and reduces the risk of the internal and external parameter errors of some cameras spreading in the multi-camera system. And in this method, each new tracker performs coordinate transformation with a known tracker, and the base station only provides a transfer function for the coordinate transformation between the two. Therefore, there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the implementation difficulty.

[0110] On the other hand, assuming that 40 cameras need to be calibrated for multi-camera without field of view, when the tracker is fixed to the 30th camera, the positions and attitudes of the subsequent 10 cameras numbered 31-40 can be adjusted. Unlike the prior art, after adjustment, there is no need to re-separately calibrate before continuing the multi-camera calibration. Therefore, the solution of the present invention is more efficient in multi-camera calibration.

[0111] When the number of cameras is small or the distances are close, and the local positioning of the base station can complete the positioning of the trackers on all cameras in one time, the local tracker positioning process becomes a global positioning process, and the global tracker expansion no longer proceeds.

[0112] According to the calibration results of the tracker coordinate system and the camera coordinate system, the position and attitude of the camera coordinate system relative to the local coordinate system of the tracker installed thereon are a fixed constant. The position and attitude information of each camera in the global coordinate system can be correspondingly obtained through the global coordinates of the tracker correspondingly installed on the camera.

[0113] The method can further set the local coordinate system of one of the cameras as a reference. Through coordinate transformation, the position and attitude parameters of all cameras are changed to relative coordinates relative to the local coordinate system of this camera, and the position and angle values of this camera both become 0, and the position and attitude matrix is the identity matrix.

[0114] The method can also introduce an additional reference coordinate system, change the positions and attitudes of all cameras to relative values in this reference coordinate system, and perform an overall transformation on the collected spatial position data.

[0115] This embodiment is directed to the spatial positioning of multiple 3D space cameras in a large range without overlapping fields of view, but it is also applicable to the spatial positioning of two or more ordinary 2D plane cameras, and the spatial positioning of two or more 3D cameras at close range with overlapping fields of view.

[0116] Before implementing with reference to this embodiment, the installation positions of multiple 3D cameras have been determined according to requirements, and there is no need to pre-set overlapping field of view areas between cameras when designing the installation positions. According to this embodiment, the interval between cameras can be set infinitely far and they can be oriented towards different observation angles.

[0117] When implementing according to this embodiment, the required operations are generally divided into four stages, namely the installation and positioning of the tracker on the camera, the calibration of the tracker coordinate system and the camera coordinate system, the local tracker positioning, and the global tracker expansion and the global unification of the camera coordinate system.

[0118] The first stage of implementation is the installation and positioning of the tracker on the camera.

[0119] Reference Figure 1 , the tracker 102 and the tracker base 104 are installed and combined into the overall tracker and base 106, the rigid intermediate connector 108 and the camera 110 are installed and combined into the overall camera and connector 112, and further combined into the overall structure 114 including the tracker and the camera. This overall structure is used in cooperation with the tracking base station 116 to obtain the spatial position and attitude of the overall structure 114 relative to the base station 116.

[0120] The tracker 102 has a plurality of signal receiving devices in different directions on its outer surface. The base station 116 actively emits specific signals within a certain area. When the tracker 102 is within the observable range of the base station 116, each signal receiving device calculates in real time the position and attitude information of the local coordinate system of the tracker 102 relative to the coordinate system of the base station 116 according to the distribution of the signal sensing intensity, and forms the corresponding position and attitude matrix. After the base station 116 adjusts the observation position and observation angle, it will re-establish the reference coordinate system at the new position and establish a tracking relationship with the trackers within the new observable area.

[0121] In some embodiments, the tracker 102 can also actively emit signals in multiple directions outward and the base station 116 can sense the signal intensity and distribution to obtain their relative positions. The signal receiving devices on the tracker 102 can also be set as multiple marker points that can independently track the XYZ space coordinates. The base station 116 sequentially obtains the XYZ space coordinates of at least three marker points on the tracker 102, and combines them to obtain the spatial position and attitude of the tracker 102. The electromagnetic tracking technology composed of the tracker and the base station belongs to the conventional technical means in this field and will not be elaborated here. The specific implementation can refer to the following patent documents: CN107452036A - A method for calculating the pose of an optical tracker with global optimality, CN102374847B - A device and method for dynamically measuring the pose of six degrees of freedom in the working space. It should be noted that the present invention adopts optical positioning technology, which has higher accuracy than UWB (Ultra-Wideband technology) and can reach the sub-millimeter level. Among them, in the prior art, for example, in CN102374847B - A device and method for dynamically measuring the pose of six degrees of freedom in the working space, it is necessary to calculate the rotation angle of the transmitter (i.e., the base station) to calculate the positions and postures of each tracker. However, in the present invention, each newly entered tracker in the current observation range performs coordinate transformation with a known tracker, and the base station only provides a transfer function for the two to perform coordinate system transformation. Therefore, there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the implementation difficulty.

[0122] In this embodiment, the bottom of the tracker 102 is a regular rectangular structure, and the center position of the bottom is the origin of the coordinate system during spatial tracking. Two of the three coordinate axes are located on the bottom surface, and the third axis is perpendicular to the bottom surface, forming the local coordinate system of the tracker 102. The position and attitude information obtained during spatial tracking is the pose matrix of this coordinate system relative to the coordinate system of the base station 116. The special base 104 corresponding to the bottom structure of the tracker 102 is in the shape of a regular cuboid, and the edges are parallel to the coordinate axes of the tracker 102. The two are precisely combined through positioning and connection structures on the contact surface to form the overall tracker and base 106, which will no longer be disassembled afterwards and is externally connected through the base 104 as a whole.

[0123] In this embodiment, the rigid intermediate connector 108 is used to position the overall structure 106 of the tracker 102 and the base 104 on the camera 110. The rigid intermediate connector 108 has a structure adapted to the outer shape of the camera 110 and is installed on the camera 1120 in advance by means of side plane and end face contact positioning to form the overall camera and connector 112, without affecting the installation and normal use of the camera 110 on the external support structure.

[0124] The positioning of the overall tracker and base 106 on the rigid intermediate connector 108 adopts Figure 2The mechanical method of end face positioning and screw tightening and fixing is shown. In this method, the tracker and the overall base 106 and the rigid intermediate connector 108 are provided with four pairs of corresponding positioning holes. Threads are provided on the positioning holes of the rigid intermediate connector 108. After the two are aligned, 4 bolts 202 are tightened. Through this method, the tracker and the overall base 106 can be quickly installed and disassembled on the camera and the overall connector 112. This positioning can also use Figure 3 The magnetic attraction method shown. In this method, three positioning pins 302 and a ring magnet 306 are provided on the tracker base, and three positioning holes 304 and a ring magnet 308 are correspondingly provided on the rigid intermediate connector 108. The two are positioned through the cooperation of the positioning holes 304 and the positioning pins 302, and the two are connected and fixed through the adsorption force between the magnets 306 and 308.

[0125] In this embodiment, both methods can be used for the positioning of the tracker and the overall base 106 on the rigid intermediate connector 108. The two have the same positioning effect and do not affect the camera positioning method and the final effect.

[0126] The tracker and the overall base 106 are only installed on the camera and the overall connector 112 when the position of the camera 110 needs to be obtained, and are disassembled after the position of the camera 110 is obtained. The tracker base 106 and the rigid intermediate connector 108 need to have high machining accuracy and can be freely combined between different instances.

[0127] In some embodiments, the structures of the tracker base 104 and the rigid intermediate connector 108 are not limited except for the areas in direct contact with the tracker 102 and the camera 110. The appearance and positioning structures such as square holes, round holes, V-shaped grooves, etc. and connection and stress structures such as threaded holes can be designed according to requirements.

[0128] The second implementation stage is the calibration of the tracker coordinate system and the camera coordinate system.

[0129] After the overall structure 114 of the tracker and the camera is assembled and installed, the tracker 102 and the camera 110 have a fixed relative position relationship. Refer to Figure 4 , the overall structure 114 of the tracker 102-1 and the camera 110, and the combination of a planar marker 402 and the tracker 106 with a base are placed in the observable area of the same base station 116. The overall structure 114 of the tracker and the camera needs to adjust the corresponding posture so that the camera 110 can collect the image of the planar marker 402 at a close distance and is preferably located at the center of the image.

[0130] Refer to Figure 5, the calibration of the coordinate system of the tracker 102-1 and the coordinate system of the camera 110 may include steps 502 to 520. Among them, steps 502 to 508 generate the coordinate conversion relationship between the tracker 102-2 and the planar marker 402, steps 510 to 512 generate the coordinate conversion relationship between the tracker 102-1 and the tracker 102-2 on the overall structure 114 of the tracker and the camera, and steps 514 to 516 generate the coordinate conversion relationship between the camera 110 and the planar marker 402.

[0131] Step 502 is to place the combination 106 of the base 104B and the tracker 102-2 at the central position of the planar marker 402, so that it is within the observable range of the base station 116, and the planar marker 402 is at the central position of the 2D image of the camera 110 and occupies most of the image area. In step 504, a high-precision measuring tool is used to measure the height of the tracker base 104B, that is, the distance between the connection surface of the tracker base 104B and the tracker 102-2 and the contact surface of the tracker base 104B and the planar marker 402. In step 506, the position of the base 104B on the planar marker 402 is adjusted so that the coordinate axes of the coordinate system of the tracker 102-2 are parallel to the coordinate axes of the local coordinate system of the marker 402, and the two only differ by the height distance of the tracker base 104B in the vertical direction of the planar marker 402, and this distance is the origin distance between the coordinate system of the tracker 102-2 and the coordinate system of the planar marker 402. In step 508, since the coordinate system of the tracker 102-2 and the coordinate system of the planar marker 402 only differ by the height distance of the tracker base 104B in one coordinate axis direction, the conversion matrix between the two coordinate systems can directly change the corresponding translation coordinate to the height distance of the tracker base 104B on the basis of the unit matrix.

[0132] Step 510 is to simultaneously observe the spatial positions of the tracker 102-1 and the tracker 102-2 on the overall structure 114 of the tracker and the camera through the base station 116, and then in step 512, the relative conversion matrix between the two trackers 102-1 and 102-2 is obtained through coordinate transformation.

[0133] In another embodiment, to avoid the influence of mechanical errors in measuring the height of the tracker base 104B on the transformation matrix between the planar marker coordinate system and the tracker coordinate system thereon, a camera calibration method can be adopted. Specifically, the coordinates of each signal receiving device on the tracker in the tracker coordinate system are known. Then, the camera can be used to simultaneously capture the planar marker and the tracker thereon. Subsequently, through image feature point matching technology, the transformation relationships between the camera and the tracker coordinate system, and between the camera and the planar marker coordinate system can be obtained respectively. Then, by transferring the coordinate systems, the transformation matrix between the planar marker coordinate system and the tracker coordinate system thereon can be obtained.

[0134] In step 514, the 2D planar camera of the camera 110 captures the corresponding image, and simultaneously records the resolution of the image and its own internal parameters, and saves them as an electronic file. Step 516 is that the camera 110 extracts the pixel coordinates of the marker points on the image of the planar marker 402 through image recognition, and obtains its position and orientation relative to the coordinate system of the planar marker 402 through the mapping relationship between 2D and 3D positions, that is, the relative transformation matrix between the coordinate system of the camera 110 and the coordinate system of the planar marker 402.

[0135] In step 518, based on the coordinate transformation matrix between the tracker 102-2 and the planar marker 402, the coordinate transformation matrix between the tracker 102-1 and the tracker 102-2, and the coordinate transformation matrix between the camera 110 and the planar marker 402 that have been obtained, calculate the coordinate transformation matrix between the tracker 102-1 and the camera 110 on the overall structure 114 of the tracker and the camera. Step 520 is to store the obtained coordinate transformation matrix between the tracker 102-1 and the camera 110 as a known constant. When the coordinate position of any one of the tracker 102-1 or the camera 110 is obtained, the coordinate position of the other object can be calculated.

[0136] In this embodiment, the planar marker 402 is composed of equally sized black and white squares arranged according to certain rules. However, in some embodiments, the planar marker 402 can also be a regular circle, ellipse, or any image with complex textures, and the relative position relationship with the camera can be obtained through the feature point recognition method.

[0137] In this embodiment, the relative position and orientation relationship between the coordinate system of the tracker 102 and the coordinate system of the camera 110 is consistent and universal when the shapes of the tracker base 104, the rigid intermediate connector 108, and the camera 110 do not change. That is, after multiple trackers 102 and multiple cameras 110 are combined through the tracker base 104 and the rigid intermediate connector 108, the relative position between the selected tracker 102 and the camera 110 remains unchanged, and the coordinate transformation matrix remains unchanged.

[0138] In some embodiments, when using trackers 102 of the same model and their bases 104, rigid intermediate connectors 108, and cameras 110 for connection, the relative position and attitude relationship between the coordinate system of the tracker 102 and the coordinate system of the camera 110 do not change. The position and attitude parameters of one coordinate system can be calculated from the position and attitude parameters of the other coordinate system.

[0139] The third implementation stage is local tracker positioning.

[0140] In this embodiment, each time a base station 116 is placed, an observation and positioning of the tracker 102 installed on the camera 110 within the observable area is performed, which is a local tracker positioning process. This local tracker positioning process only locates and obtains the position of the tracker 102, does not calculate the position of the camera 110, and only calculates after all the trackers corresponding to all the cameras are completed.

[0141] Reference Figure 6 , adjust the position and observation angle of the base station 116 so that at least two cameras 110 are within its observable range. For all the cameras 110, install the tracker and the base unit 106 as a whole to obtain the overall structure 114 including the tracker and the camera, and directly obtain the position and attitude parameters of all the trackers 102 within the observable area, that is, the local coordinates relative to the coordinate system of the base station 116. Subsequently, the base station 116 sends these position coordinates to a computer for storage and processing.

[0142] Reference Figure 7 , the operation and algorithm flow of a single local tracker positioning may include steps 702 to 722. Step 702 is to adjust the position and observation angle of the base station 116 so that at least two cameras 110 are within its observable range, and install the tracker and the base unit 106 as a whole for all the cameras 110. Step 704 is to simultaneously obtain the coordinates of each tracker 102 relative to the coordinate system of the base station 116, and in step 706, send the coordinate data to a computer for storage and processing in real time. Step 708 is to judge the positioning progress. When performing local positioning for the first time, steps 710 to 714 are performed, and when not performing local positioning for the first time, steps 716 to 720 are performed.

[0143] In step 710, start the first local positioning. No tracker 102 has obtained the global coordinates yet, and no global coordinate system has been defined; at this time, the base station 116 can be set as a known object to become the reference coordinate system for this observation. In step 712, define the coordinate system of the base station 116 as the global coordinate system, and in step 714, set the coordinates of all the trackers 102 observed this time as the global coordinates.

[0144] In step 716, at least one local positioning has been performed and a global coordinate system has been established. Some of the trackers 102 have obtained global coordinates. The coordinate system of the first known tracker 102 is selected as the reference coordinate system for this observation. In step 718, through coordinate transformation, the relative position and attitude of all other trackers 102 in this observation relative to the reference coordinate system can be obtained, and multiple coordinate transformation matrices are obtained. In step 720, based on the coordinates of the other trackers 102 relative to the coordinate system of the known tracker 102 and the global coordinates of the known tracker 102, the global coordinates of all the trackers 102 observed in this observation in the global reference coordinate system can be calculated.

[0145] Step 722 is after obtaining the global coordinates of all the trackers 102 observed in the current positioning of the base station 116, marking all the trackers 102 observed in this observation as new known trackers 102, recording data and completing the current local tracker positioning process.

[0146] In some embodiments, the base station 116 may obtain the coordinates of multiple known trackers 102 simultaneously in one observation, and one of the trackers 102 with known coordinates can be selected as the reference object for this positioning.

[0147] In some embodiments, relay trackers are arranged between cameras that are relatively far apart. At this time, the position of the base station 116 can be adjusted so that its observation range covers the relay trackers, and the relay trackers only serve as common positioning points during two local positionings.

[0148] The fourth implementation stage is global tracker expansion and global unification of the camera coordinate system.

[0149] In this embodiment, the observation position and observation angle of the base station 116 are adjusted in sequence to change its observable area. Except for the first positioning, the new observable area covers at least one known tracker. After performing multiple local tracker positioning processes, the observable area can be gradually expanded until it covers all the trackers 102 on all the cameras. This process is the global tracker expansion process.

[0150] Reference Figure 8 , global tracker expansion and global unification of the camera coordinate system include steps 802 to 810. After updating the position of the base station according to step 702 so that its observation range covers at least two cameras, in step 802, the trackers outside the current observation area are removed and installed on the cameras within the current observation area or used as relay trackers according to step 804.

[0151] After performing a local tracker positioning according to the process at 700, it is determined at step 806 whether the positioning of the trackers on all cameras has been completed. When the positioning of the trackers on all cameras has not been completed, steps 802 to 806 and 700 are continuously repeated. When the positioning of the trackers on all cameras has been completed, based on the relative transformation relationship between camera 110 and the tracker 102 installed thereon stored in step 520, the coordinates of all cameras in a unified global coordinate system are calculated at step 808. At step 810, a certain camera or an additional coordinate system can be set as a reference, and the coordinates of all cameras are transformed into a new reference coordinate system, while the relative position relationship between the cameras remains unchanged.

[0152] Refer to Figures 9A to 9E , cameras 110A, 110B, 110C, 110D, 110E, 110F are cameras to be positioned. The cameras are far apart and have no overlapping fields of view. By using three trackers 102A, 102B, 102C and performing multiple local tracker positionings through the global tracker extension process, the position coordinates of the corresponding trackers on all cameras can be obtained.

[0153] Refer to Figure 9A , a first local tracker positioning is performed on cameras 110A, 110B, 110C. The base station 116 is located at position 1, and its observation range covers cameras 110A, 110B, 110C. Trackers 102A, 102B, 102C are correspondingly installed on the cameras, and the position coordinates of trackers 102A, 102B, 102C can be obtained at one time. In this local tracker positioning, the coordinate system of the base station 116 is the global coordinate system, and no global coordinate system transformation is required for each tracker. The trackers on cameras 110A, 110B, 110C are marked as known objects.

[0154] Refer to Figure 9B , a local tracker positioning is performed on cameras 110C, 110D. The base station 116 is located at position 2, and its observation range covers cameras 110C, 110D. The tracker 102A on camera 110A is disassembled and reinstalled on camera 110D, and the position coordinates of trackers 102C, 102A can be obtained at one time. In this local tracker positioning, tracker 102C is a known object. After calculating the relative position coordinates of tracker 102A relative to tracker 102C, based on the global coordinates of tracker 102C, the global coordinates of tracker 102A are calculated, and the tracker on camera 110D is marked as a known object.

[0155] Refer to Figure 9C, since the areas of cameras 110D are far from those of cameras 110E and 110F, the tracker 102B on camera 110B is disassembled and placed near camera 110D, and the tracker 102C on camera 110C is disassembled and placed near cameras 110E and 110F. The trackers 102B and 102C are set as relay trackers, and a local positioning is performed on trackers 102A and 102B. The base station 116 is located at position 3, and its observation range covers the tracker 102A and the relay tracker 102B on camera 110D, and the position coordinates of trackers 102A and 102B can be obtained at one time. In this local tracker positioning, the tracker 102A is a known object. After calculating the relative position coordinates of the relay tracker 102B with respect to the tracker 102A, based on the global coordinates of the tracker 102A, the global coordinates of the relay tracker 102B are calculated, and the relay tracker 102B is marked as a known object.

[0156] Reference Figure 9D , a first local positioning is performed on the relay trackers 102B and 102C. The base station 116 is located at position 4, and its observation range covers the relay trackers 102B and 102C, and their position coordinates can be obtained at one time. In this local tracker positioning, the relay tracker 102B is a known object. After calculating the relative position coordinates of the relay tracker 102C with respect to the relay tracker 102B, based on the global coordinates of the relay tracker 102B, the global coordinates of the relay tracker 102C are calculated, and the relay tracker 102C is marked as a known object.

[0157] Reference Figure 9E , a local tracker positioning is performed on the relay tracker 102C, cameras 110E and 110F. The base station 116 is located at position 5, and its observation range covers cameras 110E, 110F and the relay tracker 102C. The relay tracker 102B is disassembled and reinstalled on camera 110E, and the relay tracker 102A is disassembled and reinstalled on camera 110F. The position coordinates of trackers 102C, 102B and 102A can be obtained at one time. In this local tracker positioning, the relay tracker 102C is a known object. After calculating the relative position coordinates of trackers 102B and 102A with respect to the relay tracker 102C, based on the global coordinates of the relay tracker 102C, the global coordinates of trackers 102B and 102A are calculated, and the trackers on cameras 110E and 110F are marked as known objects.

[0158] After calculating the positions of the trackers 102 correspondingly installed on cameras 110A, 110B, 110C, 110D, 110E, and 110F, based on the relative position transformation matrix between the camera 110 and the tracker 102 installed thereon, the coordinates of cameras 110A, 110B, 110C, 110D, 110E, and 110F relative to the coordinate system of the base station at position 1 can be calculated, completing the spatial positioning of multiple cameras with long distances and non-overlapping fields of view.

[0159] In some embodiments, the local coordinate system of one of the cameras can be further set as a reference. Through coordinate transformation, the position and attitude parameters of all cameras are changed to relative coordinates with respect to the local coordinate system of this camera, and the position and angle values of this camera both become 0, and the position and attitude matrix is the identity matrix. Additionally, an additional reference coordinate system can be introduced to change the positions and attitudes of all cameras to relative values in this reference coordinate system, and an overall transformation of the collected spatial position data is performed.

[0160] In this embodiment, one base station 116 and three trackers 102 are used for camera positioning. After the tracker 102 becomes a known tracker, the tracker and the base 106 as a whole can be removed from the rigid intermediate connector and installed on a new camera 110 to be observed or at a relay position. In some embodiments, regardless of the number of cameras to be positioned, at least one base station and at least two trackers are required. The global tracker expansion process can be carried out using more than one base station simultaneously, or the number of trackers can be increased to increase the number of cameras that can be positioned by the base station each time, thereby accelerating the positioning process.

[0161] In some embodiments, when the number of trackers is insufficient to be installed on all cameras covered by a single local positioning, it can be split into multiple local positioning processes, and only a small number of cameras are installed with trackers for positioning each time.

[0162] In this embodiment, when the distance between cameras 110 is so far that the observable area of the base station 116 cannot cover the trackers 102 of at least two cameras 110, one or more new relay trackers 102 are set at the intermediate position between the two cameras 110, so that the base station 116 can change its position and observation angle, and its observable area can cover the tracker 102 installed on this camera and at least one relay tracker 102 when it is near the position where the camera 110 is located, and can cover at least two relay trackers 102 at the intermediate position between the two cameras 110. Through multiple position transmissions of the relay tracker 102, the coordinate relationships of the trackers 102 on multiple long-distance cameras 110 in the same coordinate system can be established.

[0163] In some embodiments, when the number of cameras is small or the cameras are close to each other, and the positioning of the trackers on all cameras can be completed through a single local positioning by the base station, the local tracker positioning process becomes the global positioning process, and the global tracker expansion and coordinate system unification are no longer carried out.

[0164] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

Claims

1. A multi-camera calibration system with no overlapping field of view, characterized in that: The system comprises: At least two cameras to be calibrated; At least two trackers, each tracker can be detachably rigidly connected to each camera, and after connection, the tracker and each camera have the same relative position and posture relationship, and the tracker is provided with a plurality of signal receiving devices in different directions on the outer surface, and is used in conjunction with a base station that transmits signals outward, and is used to obtain the position and posture information of the tracker's local coordinate system relative to the base station coordinate system according to the distribution of the intensity of the received signal, and form a corresponding position and posture matrix; A base station, wherein the signal emitted by the base station is dispersedly transmitted outward from a point, and is used to establish a real-time tracking relationship with the tracker in the observable area based on its own coordinate system within the signal coverage angle and range; and A combination, the combination includes a planar marker with known relative position and posture and one of the at least two trackers, which is used to obtain the position and posture relationship between the camera coordinate system and the local coordinate system of the tracker installed thereon after the tracker is installed on the camera through coordinate system transformation; wherein the planar marker is any two-dimensional image that can be positioned through image recognition, and the two coordinate axes of the planar marker coordinate system are set on the image plane, and the corresponding coordinate origin and the two coordinate axis directions are marked for positioning the tracker in the combination; the tracker in the combination is set on the planar marker.

2. The system according to claim 1, characterized in that The planar marker is a planar marker that can be identified through image positioning. The tracker is placed on the planar marker, and the relative position relationship between the tracker coordinate system and the planar marker coordinate system can be obtained through physical measurement.

3. The system according to claim 1, characterized in that It also includes a tracker base and a rigid intermediate connector, the tracker is fixedly connected to the tracker base, the rigid intermediate connector is adapted to the shape of each camera and is fixedly connected thereto, each camera has a rigid connector connected thereto as a whole, and the tracker can be rigidly mounted to the intermediate connector corresponding to each camera through the tracker base.

4. A multi-camera calibration method without overlapping fields of view, characterized in that: The method is applied to the system described in claim 1, and the method comprises: Camera and tracker calibration step: based on the assembly, determining a first transformation relationship between a camera coordinate system of each camera and a tracker coordinate system mounted thereon; Local tracker positioning step: adjust the position and observation angle of the base station so that at least two cameras are within the observable range of the base station, install trackers for all cameras in the observable area, obtain the position and attitude parameters of all trackers in the observable area relative to the base station, and calculate the transformation relationship between the tracker coordinate system of each tracker and the coordinate system of the known object in this observation based on the observed position and attitude parameters of all trackers relative to the base station; wherein the coordinate system of the known object is a local coordinate system established for the base station or the tracker itself; the transformation relationship between the coordinate system of the known object and the global coordinate system is known; when the local tracker positioning step is performed for the first time, the known object is the base station at the time of the observation, and when the local tracker positioning step is performed subsequently, it is the first known tracker within the observation range; Global tracker expansion positioning step: repeating the local tracker positioning step multiple times, adjusting the position and observation angle of the base station each time, so that the new observable area covers at least one known tracker except the first positioning, until the transformation relationship between the tracker coordinate system of all trackers and the tracker coordinate system of the known tracker in each observation is obtained; The step of global unification of the camera coordinate system: based on the first transformation relationship and the transformation relationship between the coordinate system of the tracker installed on each camera and the global coordinate system, a second transformation relationship between the camera coordinate system of each camera and the global coordinate system is determined.

5. The method according to claim 4, characterized in that The camera and tracker calibration steps specifically include: Acquire a third transformation relationship between a plane marker coordinate system of the plane marker in the assembly and a tracker coordinate system of the tracker; Using an image of the plane marker captured by any one of the multiple cameras, calibrate the corresponding camera to obtain a fourth transformation relationship between a camera coordinate system of the camera and a plane marker coordinate system of the plane marker; Obtaining, through the base station, a fifth transformation relationship between a tracker coordinate system of the tracker on the assembly and a tracker coordinate system of the tracker on the camera; Based on the third transformation relationship, the fourth transformation relationship and the fifth transformation relationship, a first transformation relationship between the camera coordinate system of each camera and the tracker coordinate system installed thereon is calculated.

6. The method according to claim 5, characterized in that The step of acquiring the third transformation relationship between the plane marker coordinate system of the plane marker in the assembly and the tracker coordinate system of the tracker specifically includes: The third change relationship is obtained by measuring the relative position and posture between the coordinate system of the planar marker and the tracker coordinate system of the tracker.

7. The method according to claim 4, characterized in that The method further comprises: When the number of trackers is less than the number of all cameras that need to be calibrated, after the base station changes its position and angle, the trackers on the cameras that are not in the observable area are removed and the removed trackers that are not in the observable area are installed on the cameras in the current observable area.

8. The method according to claim 4, characterized in that The method further comprises: When the cameras are far apart and the observable area of ​​the base station cannot cover the trackers of at least two cameras, one or more new relay trackers are set in the middle of the two cameras, so that the base station can change its position and observation angle so that the observable area of ​​the base station can cover the tracker installed on the camera and at least one relay tracker, or can simultaneously cover at least two relay trackers located in the middle of the two cameras; Through multiple position transfers of the relay tracker, according to the local tracker positioning step and the global tracker expansion positioning step, the transformation relationship between the tracker coordinate system of all trackers and the tracker coordinate system of the known tracker in each observation is obtained.

9. The method according to claim 4, characterized in that The method further comprises: Set the local coordinate system of one of the cameras as the reference, and transform the position and attitude parameters of all cameras into relative position and attitude parameters relative to the local coordinate system of the camera through coordinate transformation.

10. The method according to claim 4, characterized in that The method further comprises: An additional reference coordinate system is introduced, and the position and attitude parameters of all cameras are transformed into relative position and attitude parameters relative to the reference coordinate system through coordinate transformation.

Images