System, calibration apparatus and method for determining a coordinate transformation for two cameras

Abstract

A system (100), a method and a calibration apparatus (110) for determining a coordinate transformation between a first coordinate system of a first camera (A) and a second coordinate system of a second camera (B) are disclosed. A distant object (P) is present in both camera's fields of view. A first marker (R) and a second marker (L) are visible in the second camera's (B) field of view. The calibration apparatus (110) solves a mathematical function dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera (B) while minimizing a cost function relating to reprojection error for at least one reprojection point in the first camera's (A) field of view and the second camera's (B) field of view, to obtain estimates of the orientation and translation components and the focal length of the second camera (B).

Description

[0001] SYSTEM, CALIBRATION APPARATUS AND METHOD FOR DETERMINING A COORDINATE

[0002] TRANSFORMATION FOR TWO CAMERAS

[0003] TECHNICAL FIELD

[0004] The embodiments herein relate to calibration of two cameras, having at least partially overlapping fields of view. In particular, a method and a calibration apparatus for for determining a coordinate transformation between a first coordinate system of a first camera and a second coordinate system of a second camera are disclosed. A corresponding computer program and a carrier therefor as well as a related system are also disclosed.

[0005] BACKGROUND

[0006] When tracking sports projectiles, such as golf balls, a tracking sensor is used to capture the sports projectile's trajectory as it is moving.

[0007] In some scenarios, a broadcast camera for broadcasting a video showing the sport projectile is also used. The tracked trace from the tracking sensor is then rendered in the video to enhance the experience for viewers of the video. In order to do this, the tracked trace must be converted, or transformed, from a coordinate system of the tracking sensor to a coordinate system of the broadcast camera. Thus, there is a need for calibration of the tracking sensor and the broadcast camera.

[0008] Related literature discloses a variety of methods for stereo camera calibration, such as the 8- point algorithm: https: / / en.wikipedia.org / wiki / Eight-point_algorithm. In common for many methods described in the literature is a need for having a relatively large distance between the tracking camera and the broadcast camera and accurately measured positions of numerus reference markers, such as a checkboards, markers or other district points that are visible in both cameras.

[0009] In the case of broadcast golf, there are no or limited possibilities to add items such as markers, flags, stickers, chessboards, signs or any kind of external reference points, visible to the broadcast camera. Also, it is not possible to establish a wide base between the cameras, given the geometrical limitation of a tee box and the distance to far end of a golf ball trajectory, typically over 200 meters. When arbitrary placement of the broadcast camera and the tracking sensor occurs at a golf tee box, the coordinate systems of the broadcast camera and the tracking sensor, respectively, have no visible fixed points in the real world. The camera's coordinate-axes have no representation and are of course invisible in the golf course environment. Thus, orientation and translation transformations cannot be directly observed or measured.

[0010] SUMMARY

[0011] An object may be to overcome, or at least reduce, one or more of the abovementioned problems and / or disadvantages.

[0012] This, and other object, may be achieved by the solutions set forth in the appended independent claims.

[0013] According to an aspect, the object is achieved by a method, performed by a calibration apparatus, for determining a coordinate transformation between a first coordinate system of a first camera and a second coordinate system of a second camera. The first and second cameras are spaced away from each other and located above a ground plane. A first field of view of the first camera at least partially overlaps with a second field of view of the second camera. A distant object is present in both the first and second fields of view. A first marker and a second marker are located at the ground plane and the first and second markers are visible in the second field of view. The calibration apparatus obtains a set of measures comprising at least five of:

[0014] • a first measure of a first distance, projected on the ground plane, between the first camera and the first marker,

[0015] • a second measure of a second distance, projected on the ground plane, between the first camera and the second marker,

[0016] • a third measure of a third distance, projected on the ground plane, between the first camera and the second camera,

[0017] • a fourth measure of a fourth distance, projected on the ground plane, between the second camera and the first marker, • a fifth measure of a fifth distance, projected on the ground plane, between the second camera and the second marker, or

[0018] • a sixth measure of a sixth distance, projected on the ground plane, between the first marker and the second marker.

[0019] When the set of measures comprises all but one measure of the first, second, third, fourth, fifth and sixth measures, the calibration apparatus calculates said one measure based on the set of measures. The calibration apparatus estimates a focal length of the second camera based on

[0020] • the fourth measure,

[0021] • the fifth measure,

[0022] • a first position of the first marker in the second field of view,

[0023] • a second position of the second marker in the second field of view,

[0024] • a two-dimensional sensor size of the second camera,

[0025] • the sixth measure, and

[0026] • a two-dimensional image resolution of the second camera.

[0027] The calibration apparatus obtains an object distance between the distant object and a location at the first camera and / or the second camera. The calibration apparatus determines a first angle, in a first plane that is parallel with the ground plane, indicative of the distant object's deviation from the first camera's straight viewing direction based on:

[0028] • the distant object's position in the first camera,

[0029] • sensor size of the first camera,

[0030] • a focal length of the first camera, and

[0031] • the object distance.

[0032] The calibration apparatus determines a second angle, in the first plane, indicative of the distant object's deviation from the second camera's straight viewing direction based on:

[0033] • the distant object's position in the second camera,

[0034] • sensor size of the second camera,

[0035] • the focal length of the second camera, and

[0036] • the object distance. The calibration apparatus determines a third angle, in the first plane, having a vertex at the distant object's position and two rays to the first and second cameras, respectively, based on:

[0037] • the third measure, and

[0038] • the object distance.

[0039] The calibration apparatus estimates a first orientation component in the first plane based on the first angle, the second angle and the third angle. The calibration apparatus defines a first circle, being centered at a point defined by a projection of an optical center of the second camera at the ground plane and having a radius defined by the third measure, and a second circle, being centered at the second marker and having a radius defined by the second measure.

[0040] The calibration apparatus determines a first intersection and a second intersection between the first and second circles. Next, the calibration apparatus selects one of the first and second intersections as a position of the first camera on the ground plane based on a comparison between the first measure and a first distance between the first intersection and the second marker and between the first measure and a second distance between the second intersection and the second marker. The calibration apparatus estimates a first and a second translation component based on at least the selected intersection and the second marker.

[0041] Furthermore, the calibration apparatus estimates a first and a second translation component based on the first orientation component and a coordinate of the selected intersection.

[0042] The calibration apparatus obtains at least a first indication and at least a second indication. The first indication relates to a first distance between the first camera and a first feature point. The second indication relates to a second distance between the first camera and a second feature point. The first and second feature points are included in the first and second fields of view. The calibration apparatus estimates a second orientation component based on a first feature angle and a second feature angle. The first feature angle is given in the coordinate system of the first camera based on the first and second feature points and the second feature angle is given in the coordinate system of the second camera based on the first and second feature points.

[0043] The calibration apparatus also obtains an indication of a third translation component. The calibration apparatus determines a fourth angle, in a second plane that is perpendicular to the ground plane and that is parallel with the first camera's straight viewing direction. The fourth angle is indicative of the distant object's deviation from the first camera's straight viewing direction. The determining is based on:

[0044] • the distant object's position in the first camera,

[0045] • sensor size of the first camera,

[0046] • the focal length of the first camera, and

[0047] • the object distance.

[0048] The calibration apparatus determines a fifth angle, in the second plane, indicative of the distant object's deviation from the second camera's straight viewing direction based on:

[0049] • the distant object's position in the second camera,

[0050] • sensor size of the second camera,

[0051] • the focal length of the second camera, and

[0052] • the object distance.

[0053] The calibration apparatus determines a sixth angle, in the second plane having a vertex at the distant object's position and two rays to the first and second cameras, respectively, based on:

[0054] • the third translation component, and

[0055] • the object distance.

[0056] The calibration apparatus estimates a third orientation component based on the fourth angle, the fifth angle and the sixth angle.

[0057] The calibration apparatus solves a mathematical function dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera while minimizing a cost function relating to reprojection error for at least one reprojection point in the first camera's field of view and the second camera's field of view, to obtain estimates of the orientation and translation components and the focal length of the second camera.

[0058] Thanks to at least some examples of the method herein, a coordinate transformation between the coordinate system of the first camera and the coordinate system of the second camera is obtained. A problem solved with the method may also be related to the fact that the second camera does typically not have a fixed focal length due to varying zoom and focus levels. That is, not only the transformation components, but also the focal length is unknown. This means that in total seven unknown parameters need to be estimated, herein referred to as translation components, tO in 3D, and orientation, aka rotation, components, rO in 3D, and the aforementioned focal length for the second camera, typically a broadcast camera.

[0059] An advantage, with at least some embodiments herein, is that an accurate estimation of the coordinate transformation may be achieved by using a number of measures of an environment, e.g., a portion of a golf course, a tee, a tee box or the like. In some examples, the environment includes a tracked object in flight. The aforementioned number of measures are relatively easily obtained by means of common measurement devices, such as a laser range finder, a measuring tape, and / or the like.

[0060] The calibration apparatus can receive, from the first camera, a tracking stream relating to a tracked object. The tracked object can be a sports projectile, such as a golf ball, table tennis ball, a football, a soccer ball, a tennis ball, badminton ball, or the like.

[0061] As an example, the tracking stream includes information for generating a tracking trace relating to the tracked object. The tracked object can move in a three-dimensional environment in the fields of view of the first camera and the second camera. Information, such as image frame(s), radar data or the like. A path of the tracked object is captured by the first camera. The tracking stream can include one or more of image frames captured by the first camera, tracking data, blob information, paths, tracks or the like.

[0062] In some embodiments, the calibration apparatus can receive a video stream from the second camera. The calibration apparatus can render the tracking trace relating to the tracked object in the video stream based on the tracking stream, while using the estimates of the orientation and translation components and the focal length of the second camera. In this manner, the determined coordinate transformation, e.g. including the estimates of the orientation and translation components and the focal length of the second camera, is used when providing the tracking track in the video stream. Advantageously, an enhanced experience for viewers of the video stream can be achieved. According to another aspect, the object is achieved by a calibration apparatus configured for determining a coordinate transformation between a first coordinate system of a first camera and a second coordinate system of a second camera. The first and second cameras are spaced away from each other and located above a ground plane. A first field of view of the first camera at least partially overlaps with a second field of view of the second camera. A distant object is present in both the first and second fields of view. A first marker and a second marker are located at the ground plane and the first and second markers are visible in the second field of view. The calibration apparatus is configured for:

[0063] The calibration apparatus is configured for obtaining a set of measures comprising at least five of:

[0064] • a first measure of a first distance, projected on the ground plane, between the first camera and the first marker,

[0065] • a second measure of a second distance, projected on the ground plane, between the first camera and the second marker,

[0066] • a third measure of a third distance, projected on the ground plane, between the first camera and the second camera,

[0067] • a fourth measure of a fourth distance, projected on the ground plane, between the second camera and the first marker,

[0068] • a fifth measure of a fifth distance, projected on the ground plane, between the second camera and the second marker, or

[0069] • a sixth measure of a sixth distance, projected on the ground plane, between the first marker and the second marker.

[0070] When the set of measures comprises all but one measure of the first, second, third, fourth, fifth and sixth measures, the calibration apparatus is configured for calculating said one measure based on the set of measures.

[0071] The calibration apparatus is configured for estimating a focal length of the second camera based on

[0072] • the fourth measure,

[0073] • the fifth measure,

[0074] • a first position of the first marker in the second field of view,

[0075] • a second position of the second marker in the second field of view, • a two-dimensional sensor size of the second camera,

[0076] • the sixth measure, and

[0077] • a two-dimensional image resolution of the second camera.

[0078] The calibration apparatus is configured for obtaining an object distance between the distant object and a location at the first camera and / or the second camera.

[0079] The calibration apparatus is configured for determining a first angle, in a first plane that is parallel with the ground plane, indicative of the distant object's deviation from the first camera's straight viewing direction based on:

[0080] • the distant object's position in the first camera,

[0081] • sensor size of the first camera,

[0082] • a focal length of the first camera, and

[0083] • the object distance.

[0084] The calibration apparatus is configured for determining a second angle, in the first plane, indicative of the distant object's deviation from the second camera's straight viewing direction based on:

[0085] • the distant object's position in the second camera,

[0086] • sensor size of the second camera,

[0087] • the focal length of the second camera, and

[0088] • the object distance.

[0089] The calibration apparatus is configured for determining a third angle, in the first plane, having a vertex at the distant object's position and two rays to the first and second cameras, respectively, based on:

[0090] • the third measure, and

[0091] • the object distance.

[0092] The calibration apparatus is configured for estimating a first orientation component in the first plane based on the first angle, the second angle and the third angle.

[0093] The calibration apparatus is configured for defining a first circle, being centered at a point defined by a projection of an optical center of the second camera at the ground plane and having a radius defined by the third measure, and a second circle, being centered at the second marker and having a radius defined by the second measure. The calibration apparatus is configured for determining a first intersection and a second intersection between the first and second circles.

[0094] The calibration apparatus is configured for selecting one of the first and second intersections as a position of the first camera on the ground plane based on a comparison between the first measure and a first distance between the first intersection and the second marker and between the first measure and a second distance between the second intersection and the second marker.

[0095] The calibration apparatus is configured for estimating a first and a second translation component based on at least the selected intersection and the second marker(L).

[0096] The calibration apparatus is configured for estimating a first and a second translation component based on the first orientation component and a coordinate of the selected intersection.

[0097] The calibration apparatus is configured for obtaining at least a first indication and at least a second indication. The first indication relates to a first distance between the first camera and a first feature point, and wherein the second indication relates to a second distance between the first camera and a second feature point. The first and second feature points are included in the first and second fields of view.

[0098] The calibration apparatus is configured for estimating a second orientation component, based on a first feature angle and a second feature angle. The first feature angle is given in the coordinate system of the first camera based on the first and second feature points and the second feature angle is given in the coordinate system of the second camera based on the first and second feature points. The calibration apparatus is configured for obtaining an indication of a third translation component.

[0099] The calibration apparatus is configured for determining a fourth angle, in a second plane that is perpendicular to the ground plane and that is parallel with the first camera's straight viewing direction. The fourth angle is indicative of the distant object's deviation from the first camera's straight viewing direction. The calibration apparatus is configured for determining the fourth angle based on:

[0100] • the distant object's position in the first camera,

[0101] • sensor size of the first camera,

[0102] • the focal length of the first camera, and

[0103] • the object distance. The calibration apparatus is configured for determining a fifth angle, in the second plane, indicative of the distant object's deviation from the second camera's straight viewing direction based on:

[0104] • the distant object's position in the second camera,

[0105] • sensor size of the second camera,

[0106] • the focal length of the second camera, and

[0107] • the object distance.

[0108] The calibration apparatus is configured for determining a sixth angle, in the second plane having a vertex at the distant object's position and two rays to the first and second cameras, respectively, based on:

[0109] • the third translation component, and

[0110] • the object distance.

[0111] The calibration apparatus is configured for estimating a third orientation component based on the fourth angle, the fifth angle and the sixth angle.

[0112] The calibration apparatus is further configured for solving a mathematical function dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera while minimizing a cost function relating to reprojection error for at least one reprojection point in the first camera's field of view and the second camera's field of view, to obtain estimates of the orientation and translation components and the focal length of the second camera.

[0113] According to a further aspect, the object is achieved by a system comprising the first and second camera and the calibration apparatus.

[0114] BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 is an overview of an example of a system according to some embodiments herein. Figure 2a and Figure 2b are side views of the first and second cameras, respectively, of the system according to some embodiments herein.

[0116] Figure 3 is a flowchart illustrating the method according to some embodiments herein. Figure 4 is a top view of the first and second cameras in Figure 1.

[0117] Figure 5 in another top view illustrating angles and distance in relation to the first and second cameras. Figure 6 is an illustration of how to obtain estimations of two translation components.

[0118] Figure 7 includes two side views illustrating how to obtain an estimate of a rotation component.

[0119] Figure 8 is a side view of the first and second cameras in Figure 1.

[0120] Figure 9 is a block diagram illustrating an example of the calibration apparatus according to some embodiments herein.

[0121] DETAILED DESCRIPTION

[0122] In the following description, the same, or similar, reference numerals have been used to denote the same, or similar, features, examples, items, elements or the like, when applicable.

[0123] As used herein, the expression "coordinate system of a camera" may refer to that the camera's field of view is associated with, such as aligned with, corresponding to, or the like, a coordinate system, such as in three dimensions. For example, the coordinate system may describe a position of the camera and / or a rotation of the camera along three spatial dimensions, x (width), y (height) and z (length / depth).

[0124] As used herein, the term "marker" may refer to a tee marker, a corner of tee box, an identification post for marking the tee, or the like.

[0125] Figure 1, Figure 2a and Figure 2b illustrate an exemplifying set up of a system 100 configured to perform the methods described herein.

[0126] The system 100 includes a first camera A, such as a tracking sensor, or the like, and a second camera B, such as a broadcast camera, a video camera or the like. The tracking sensor can include a video camera and optionally a radar, which then can be fixedly mounted relatively each other.

[0127] The system 100 further includes a calibration apparatus 110, such as a calibration device, a computer, a computer function, a cloud function, a serverless backend function, an image processing device, a processing module or the like. The calibration apparatus 110 may be separated, but connected to, the first and second cameras A, B. In some examples, the calibration apparatus 110 can be realized as a computer function residing and executable in a cloud, such as a virtual and / or physical computer in a data center or the like. In some examples, the calibration apparatus 110 can be built into at least one of the first and second cameras A, B. Further, the calibration apparatus 110 may be realized as a separate physical entity in the form of a calibration device.

[0128] In the examples herein, the calibration apparatus 110 is configured to be connectable to the first and second camera A, B for transferring of information, such as images, distance information, angle information, instructions, blob information, trace information, tracking stream(s), messages, or the like, to / from the first and second cameras A, B.

[0129] The first and second cameras A, B may be located spaced away from each other, e.g., at a distance from each other. As shown in Figure 1, the first and second cameras A, B are standing on a ground plane G, e.g., on the same side of the ground plane G. A respective optical center of each of the first and second cameras A, B is located at a distance from the ground plane G. The ground plane G may be defined by an x-axis and a z-axis as shown in Figure 1. A y-axis is shown perpendicularly to the x-axis and the z-axis. The y-axis represents height, the x-axis represents width, and the z-axis represents depth, or distance, e.g., in the camera's viewing direction or in the main direction of a movement of the sports projectile, such as a golf ball, or the like.

[0130] A first field of view of the first camera A at least partially overlaps with a second field of view of the second camera B. A distant object P, shown in Figure 8, is present in both the first and second fields of view. Furthermore, a first marker R and a second marker L are visible in the second field of view, e.g., at least the second field of view. The first marker R and the second marker L may thus be visible in the first field of view, but not necessarily. Generally, the first and second marker R, L can be so called tee markers that are commonly used on a golfing area, such as a golf course, a driving range or the like, to indicate a tee, from which golf balls can be shot by a player. In some examples, the first and second marker R, L can be a front, right corner and a front, left corner of a bounding box that indicates the tee. For golf tracing purposes, a method to calibrate a tracking sensor and a broadcast camera without adding any reference items, such as reference markings at known positions, to the golf course environment is provided. Reference items can in this context refer to any aid that is placed in the golf course environment for the purpose, e.g. only purpose, of calibration, or at least not for the purpose of playing golf. With the method herein, seven unknown variables, referred to as tO.x, tO.y, tOz, rO.x, rO.y, rO.z and f' which will be explained further below, are estimated. Thereby a golf ball tracking sensor can be placed at an arbitrary position on the tee box, well separated from the broadcast camera. The solution may benefit from using a measuring tape, a plumb bob, a laser range finder, and / or the like. It may also be preferred that both tee markers, referred to as a first and second marker herein, are visible in the image of the second camera B, and optionally also in the image of the first camera A. Notably, the first and second markers are part of the golf course environment for playing purposes. The embodiments herein thus advantageously use the already existing first and second markers.

[0131] It may be noted that the following notations are used herein: tO: three-dimensional (x,y,z) position / translation of one camera in the coordinate system of another camera. In more detail, a first translation component is denoted tO.x, a second translation component is denoted tO.z, and a third translation component is denoted tO.y. rO: three-dimensional (x,y,z) orientation (rotation along the x,y and z axis) of one camera in the coordinate system of another camera. In more detail, a first orientation component is denoted rO.y, a second orientation component is denoted rO.z, and a third orientation component is denoted rO.x. f: focal length of a camera. f': the distance, in millimetres, between the focal point of the lens and the image sensor at a given zoom level and focus level (when a lens is focused at infinity f' is equal to the focal length of the lens / camera f) p': two-dimensional (x,y) position in a camera's image plane.

[0132] Cs: two-dimensional (x,y) sensor size in millimetres of a camera (cs for a Camera A or Camera B is denoted csa and csb respectively). Res: two-dimensional (x,y) image resolution in pixels of a camera (res for a Camera

[0133] A or a Camera B is denoted resa and resb respectively).

[0134] Camera A: a first camera, such as a tracking camera.

[0135] Camera B: a second camera, such as the broadcast camera. fn(...): A combination of various well-known geometrical and mathematical functions, such as tangent, sinus, cosine, squared root, Pythagoras theorem, triangle conformity etc. n denotes a counter used to indicate that this function may be different from one or more of the other functions disclosed herein.

[0136] P A distinct object far away visible in both cameras A, B.

[0137] For the purposes of the method herein, for the first camera A, f is considered to be constant and known and f' is assumed to be approximately equal to f.

[0138] Figure 3 shows a schematic flowchart, illustrating an exemplifying method according to the embodiments herein. Thus, the calibration apparatus 110 performs a computer- implemented method for determining a coordinate transformation between a first coordinate system of the first camera A and a second coordinate system of the second camera B, e.g., a coordinate transformation from the first coordinate system of the first camera A to the second coordinate system of the second camera B, or vice versa by inverting the coordinate transformation. As mentioned, the first and second cameras A, B are spaced away from each other and located above a ground plane G, xz-plane. As an example, the first and second cameras A, B are standing on the ground plane G. A first field of view of the first camera A at least partially overlaps with a second field of view of the second camera B. A distant object is present in both the first and second fields of view. A first marker R, such as a right tee marker, and a second marker L, such as a left tee marker, are located at the ground plane G and the first and second markers R, L are visible in the second field of view.

[0139] One or more of the following actions can be performed, e.g. in any suitable order. The actions can be performed by one or more physical and / or virtual units.

[0140] Action A110

[0141] With reference to Figure 2a and Figure 2b, it can be seen that: A' denotes the projected optical center of camera A in the ground plane G, B' denotes the projected optical center of camera B in the ground plane G, R denotes the position of the first marker in the ground plane G, and L denotes the position of the second marker in the ground plane G.

[0142] The projected position of the camera lens' optical center on the ground plane G is estimated, e.g., using a plumb bob with a string. The position on the ground may be temporarily marked. This is done for the first camera A and the second camera B. These temporary or virtual points A', B' on the ground plane G may be referred to as a first point A' and a second point B'.

[0143] With reference to Figure 4, it can be seen that:

[0144] A'B' denotes distance from A' to B',

[0145] A'L denotes distance from A' to L, A'R denotes distance from A' to R, B'R denotes distance from B' to R, B'L denotes distance from B' to L, and LR denotes distance from L to R.

[0146] Accordingly, the calibration apparatus 110 obtains, such as receives, fetches or the like, a set of measures, e.g. as user input, from a memory or the like. The set of measures includes at least five of: a first measure A'R of a first distance, projected on the ground plane G, between the first camera A and the first marker R, a second measure A'L of a second distance, projected on the ground plane G, between the first camera A and the second marker L, a third measure A'B' of a third distance, projected on the ground plane G, between the first camera A and the second camera B, a fourth measure B'R of a fourth distance, projected on the ground plane G, between the second camera B and the first marker R, a fifth measure B'L of a fifth distance, projected on the ground plane G, between the second camera B and the second marker L, or a sixth measure RL of a sixth distance, projected on the ground plane G, between the first marker R and the second marker L. It may in particular be noted that for example the first distance is a projection of the distance between the first camera A and the first marker R on the ground plane G. Likewise: the second distance is a projection of the distance between the first camera A and the second marker L on the ground plane G, the fourth distance is a projection of the distance between the second camera B and the first marker R on the ground plane G, the fifth distance is a projection of the distance between the second camera B and the second marker L on the ground plane G.

[0147] These are noteworthy since the markers' and the cameras' centers differ in height above the ground plane G. In this manner, it is thus the projections of the distances between the markers and cameras on the ground plane G that are reflected by the measures mentioned above.

[0148] Action A115

[0149] When the set of measures includes all but one measure of the first, second, third, fourth, fifth and sixth measures, the calibration apparatus 110 can calculate said one measure based on the set of measures. As an example, at least five of these six measures can be measured, e.g., using a measuring tape, a laser measuring device, or the like. Preferably, the measures are taken along the ground plane G.

[0150] Action A120

[0151] The calibration apparatus 110 estimates a focal length of the second camera B based on the fourth measure B'R, the fifth measure B'L, a first position of the first marker R in the second field of view, r', a second position of the second marker L in the second field of view, I', a two-dimensional sensor size csb of the second camera B, e.g., in the xy-plane, the sixth measure RL, and a two-dimensional image resolution of the second camera B, resb, e.g. in the x- and y-dimensions. As an example, positions of the first and second markers in the second field of view of the second camera B may be annotated, i.e., I' and r' respectively. As an example, the first and second positions can be obtained from an image captured by the second camera B, such as in the xy-plane of the image. The captured image can be obtained, such as received or the like, by the calibration apparatus 110 in order for the calibration apparatus 110 to gain access to the image and obtain, such as identify, or the like, the first and second positions in the image of the second camera B.

[0152] Triangle conformity yields an initial estimate of f' for camera B, denoted f'b. Notably, the focal length of the first camera A is considered to be constant and known as mentioned above. f'b ~ fl( B'R, B'L, r', I' , csb, RL, resb). As an example, fl= (B'R + B'L) * csb.x * abs(r'.x - I'.x) / (2 * RL * resb.x).

[0153] It shall be noted that the formula fl, and the other formulas presented below, are examples. In other examples, different formulas can be used for calculating the desired entities. Furthermore, "arcus sine" is the same as "arcsin", "arcus tangent" is the same as "arctan" and "arcus cosine" is the same as "arccos" as applicable.

[0154] Action A130

[0155] The calibration apparatus 110 obtains an object distance D between the distant object P and a location AB at the first camera A and / or the second camera B. The distant object may be visible in both the first and second cameras A, B. In this manner, the calibration apparatus 110 may obtain an indication of the object distance D, e.g., a measure thereof. This may mean that the indication may be obtained by means of a measurement. As an example, a laser range finder may be used to measure the distance D from camera A to the distinct object P, e.g., being located far away. A position of the distant object P on the image sensors in camera A and B are denoted pa and pb, respectively. TO.x is approximated to the third measure A'B' in the first iteration of the calculations below, see e.g., action A160. It may here be noted that the object distance D may be obtained between a point on distant object P and the location AB. The location AB can be at the first camera A, the second camera B, between the first and second cameras A, B, or near the first or second camera A, B, or the like. As an example, the calibration apparatus 110 can receive, e.g., by input from a user, the indication. Additionally, and / or alternatively, the calibration apparatus 110 can obtain the indication from a memory, accessible by the calibration apparatus 110. In some examples, the indication is received from a measurement device. Such measurement device may be integrated with one of the first and second cameras A, B or be separated therefrom.

[0156] In view of the above, it shall be understood that the object distance D can be used as an approximation and can refer to one or more of the following distances: a distance between the distant object P and the first camera A, e.g., an optical center of optics of the first camera A, a distance between the distant object P and the second camera B, e.g., an optical center of optics of the second camera B, a distance between the distant object P and a location AB in the vicinity of or between the first camera A and / or the second camera B, or a combination thereof.

[0157] Action A140

[0158] The calibration apparatus 110 determines a first angle a, in a first plane G', xz that is parallel with the ground plane G, indicative of the distant object's P deviation from the first camera's A straight viewing direction, aka camera axis, based on: the distant object's P position in the first camera A, pa, sensor size along x-axis of the first camera A, csa, a two-dimensional image resolution of the first camera A, resa, and the object distance D, such as a distance between the distant object P and the first camera A or the like.

[0159] Referring to action A140, A150 and A160, as observed along the y-axis, e.g., down the negative y-axis, the three first, second and third angles a, |3, Y in the xz-plane may be calculated using ordinary trigonometry, see also Figure 5.

[0160] A ~ f2(pa, csa, f'a, D)

[0161] As an example, f2= arcus sinus (((pa.x - resa.x * 0.5) *csa.x / resa.x) / D). Action A150

[0162] The calibration apparatus 110 determines a second angle |3, in the first plane G', indicative of the distant object's deviation from the second camera's B straight viewing direction based on: the distant object's position in the second camera B, pb, sensor size , e.g., along x-axis, of the second camera B, csb, a two-dimensional image resolution of the second camera B, resb, the distance between the first or second (or point in vicinity of camera a and b) camera A and the distant object P, D.

[0163] P ~ f2(pb, csb, f'b, D), f2 can be the same function as above. As an example, f2 = arcus sinus (( ( pb.x - resb.x * 0.5) *csb.x / resb.x) / D).

[0164] Action A160

[0165] The calibration apparatus 110 determines a third angle Y, in the first plane G', having a vertex at the distant object's position and two rays to the first and second cameras A, B, respectively, based on: the third measure, and the object distance D.

[0166] The third measure is used as an initial estimation of tO.x.

[0167] Y ~ f3(tO.x, D). As an example, f3= tO.x / D.

[0168] Action A170

[0169] The calibration apparatus 110 estimates a first orientation component rO.y of the coordinate transformation in the first plane G' based on the first angle, the second angle and the third angle.

[0170] In some embodiments, the calibration apparatus 110 estimates the first orientation component rO.y by calculating the first orientation component rO.y as the first angle reduced by the second angle and adding the third angle.

[0171] Expressed using the notation herein: rO.y ~ a - |3 + Y.

[0172] Action A180 The calibration apparatus 110 defines a first circle cl, being centered at a point B' defined by a projection of an optical center of the second camera B at the ground plane G and having a radius defined by the third measure A'B', and a second circle c2, being centered at the second marker L and having a radius defined by the second measure A'L. Reference is made to Figure 6. In more detail, the first and second circles cl, c2 are parallel with the ground plane G.

[0173] Action A190

[0174] The calibration apparatus 110 determines a first intersection il and a second intersection i2 between the first and second circles cl, c2. Reference is again made to Figure 6.

[0175] Action A200

[0176] The calibration apparatus 110 selects one of the first and second intersections il, i2 as a position of the first camera A on the ground plane based on a comparison between 1) the first measure A'R and a first distance DI between the first intersection il and the second marker L and between 2) the first measure A'R and a second distance D2 between the second intersection i2 and the second marker L.

[0177] Action A210

[0178] The calibration apparatus 110 estimates a first and a second translation component of the coordinate transformation based on at least the selected intersection and the second marker L.

[0179] In view of action A180, A190, A200 and A210, it may be concluded that the position A' of the first camera A on the plane ground G must have the same position as il or i2. There are then two possible solutions. The correct solution (i) is chosen by calculating the distance to R from both the first and second intersections il, i2 and comparing those distances with the measured distance A'L. This yields an estimation of tO.x and tO.z.

[0180] Action A220

[0181] The calibration apparatus 110 estimates a first translation component of the transformation tO.x and a second translation component of the transformation tO.z based on the first orientation component rO.y and a coordinate of the selected intersection. In this manner, initial estimates of the first and second translation components are obtained.

[0182] As an example, by means of the first orientation component rO.y, a transformation from the coordinate of the selected intersection il, i2 to the coordinate system of the first camera A is created. A condition is that it is assumed that the two-coordinate systems of the two cameras A, B are only rotated around the y-axis by the amount of the rO.y.

[0183] At this stage, the calibration apparatus 110 may generally iterate over actions A140 through A220 to improve the estimations of rO.y, tO.x and tO.z. The iteration may stop when the estimations of e.g., rO.y, tO.x and tO.z have converged sufficiently, e.g., below a threshold that may be set, e.g., depending on a desired accuracy.

[0184] Action A230

[0185] The calibration apparatus 110 obtains at least a first indication and at least a second indication. The first indication relates to a first distance between the first camera A and a first feature point, and wherein the second indication relates to a second distance between the first camera A and a second feature point. The first and second feature points are included in the first and second fields of view. As an example, a laser range finder can be used to measure the first distance from the first camera A to any number of feature points visible in both cameras, such as the first and second feature points.

[0186] Action A240

[0187] The calibration apparatus 110 estimates a second orientation component rO.z of the coordinate transformation based on a first feature angle vA and a second feature angle vB. The first feature angle vA is given in the coordinate system of the first camera A based on the first and second feature points and the second feature angle vB is given in the coordinate system of the second camera B based on the first and second feature points.

[0188] The first and second feature points may preferably be centered, but spaced away from each other. The first and second feature points are typically located distant to the first and second cameras A, B. The first and / or second feature points may be one or more points of, such as at, on or the like, the distant object P. However, the first and / or second feature points may alternatively or also, be taken from one or more other points identified in the first and second fields of view.

[0189] As an example, rO.z is estimated as the difference between the two angles, i.e., rO.z ~ vA - vB. Reference is made to Figure 7.

[0190] The first and second feature points pl, p2 can generally be visible in the fields of view of both cameras. Their two-dimensional image positions are denoted as pla, plb, p2a and p2b. This yields that: vA = arcus tangent ((pla.y - p2a.y) / ( pla.x - p2a.x)), and vB = arcus tangent ((plb.y - p2b.y) / ( plb.x - p2b.x)).

[0191] Action A250

[0192] The calibration apparatus 110 obtains an indication of a third translation component tO.y, e.g., by means of measurement.

[0193] As an example, an estimation of tO.y may be obtained by measuring the height above ground, e.g., the ground plane G, with a measuring tape for both the first and second camera A, B. The difference in height is the estimation of the third translation component, tO.y.

[0194] Reference is made to Figure 8 in relation to actions A260 through A290.

[0195] Action A260

[0196] The calibration apparatus 110 determines a fourth angle a2, in a second plane yz that is perpendicular to the ground plane G and that is parallel with the first camera's A straight viewing direction. The fourth angle a2 is indicative of the distant object's P deviation from the first camera's A straight viewing direction. The determining / determination A260 is based on: the distant object's position in the first camera A, pa, sensor size along y-axis of the first camera A, csa, the focal length of the first camera A, f'a, and the object distance D, such as a distance between the distant object P and the first camera A or the like.

[0197] A2 ~ f2( pa, csa, f'a, D)

[0198] See example of f2 above. Hence, as an example a2 ~ arcus sinus (((pa .y - resa.y * 0.5) *csa.y / resa.y) / D).

[0199] Action A270

[0200] The calibration apparatus 110 determines a fifth angle |32, in the second plane yz, indicative of the distant object's P deviation from the second camera's B straight viewing direction based on: the distant object's position in the second camera B, sensor size along y-axis of the second camera B, the focal length of the second camera B, and the object distance D, such as a distance between the distant object P and the second camera B or the like.

[0201] |32 ~ f2(pb, csb, f'b, D). f2 may be the same function as above, e.g., [52 ~ arcus sinus (((pb.y — resb.y * 0.5) *csb.y / resb.y) / D).

[0202] Action A280

[0203] The calibration apparatus 110 determines a sixth angle Y2, in the second plane yz having a vertex at the distant object's position and two rays to the first and second cameras A, B, respectively, based on: the third translation component, and the object distance D.

[0204] Y2 ~ f3(tO.y, D), where f3 can be the same function as above. As an example, Y2= tO.y / D.

[0205] In view of the actions relating to the fourth, fifth and sixth angle, looking down the negative x-axis, the three angles a2, [52 and Y2 in the yz-plane have been calculated using ordinary trigonometry. Action A290

[0206] The calibration apparatus 110 estimates a third orientation component rO.x of the coordinate transformation based on the fourth angle, the fifth angle and the sixth angle.

[0207] In some embodiments, the calibration apparatus 110 estimates the third orientation component rO.x by calculating the third orientation component rO.x as the fourth angle reduced by the fifth angle and adding the sixth angle.

[0208] Expressed using the notation herein: rO.x = a2 - |32 + Y2.

[0209] Action A300

[0210] The calibration apparatus 110 solves a mathematical function, which is dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera B, while minimizing a cost function relating to reprojection error for at least one reprojection point in the first cameras A field of view and the second camera's B field of view, to obtain estimates of the orientation and translation components and the focal length of the second camera B. Said at least one reprojection point, or reprojection point(s), may be point(s) on the distant object P and / or the first feature point and / or the second feature point and / or any other point in the fields of view of the first and second cameras A, B.

[0211] Reprojection errors refer to an error, e.g., in position, for a point that is visible in both the first and second camera's A, B fields of view between the coordinate transformation's transformed position of the point e.g. in the coordinate system of camera B and an detected, or actual, position of the point in the coordinate system of camera B. Explicitly, the coordinate system of camera B can refer to an image plane coordinate system of camera B and / or the coordinate system of camera A can refer to the image plane coordinate system of camera A.

[0212] Expressed differently, with the initial estimation of tO, rO and f'b, a stereo camera pair with camera A and camera B is created. By calculation, the quality of any estimation of tO, rO and f' can be evaluated by comparing estimated positions of, e.g., a reprojection point or the like, and actual, or detected, positions of the reprojection point(s). Also, the sum of all reprojection errors is an indication of accuracy of the estimated coordinate transformation's quality. An optimization problem for rO and f'b, is formulated, where the cost function is a weighted sum of all reprojection error(s). As an example, a known gradient descend procedure may be used to find a solution to the 7-degrees of freedom problem, i.e., the degrees are rO in 3D, tO in 3D and f'b.

[0213] Gradient descent procedures are well-known in related mathematical literature and refers to a first-order iterative optimization algorithm for finding a local minimum of a differentiable function.

[0214] Hence, there are final estimations of tO, rO and f'b that that be used for transforming any coordinate in the first coordinate system to a transformed coordinate in the second coordinate system, such as a tracking trace in the first coordinate system to a transformed tracking trace in the second coordinate system. The transformed tracking trace can then be rendered in a video stream from the second camera B.

[0215] It can here be noted that actions A310, A320 and A330 are described as being performed by the calibration apparatus 110. However, these actions can be performed by one or more further computer devices (not shown).

[0216] Action A310

[0217] The calibration apparatus 110 can receive, from the first camera A, a tracking stream relating to a tracked object 120. The tracked object 120 can be a sports projectile, such as a golf ball, table tennis ball, a football, a soccer ball, a tennis ball, badminton ball, or the like.

[0218] As an example, the tracking stream includes information for generating a tracking trace relating to the tracked object 120. The tracked object 120 can move in a three-dimensional environment in the fields of view of the first camera A and the second camera B. Information, such as image frame(s), radar data or the like. A path of the tracked object 120 is captured by the first camera A. The tracking stream can include one or more of image frames captured by the first camera A, tracking data, blob information, paths, tracks or the like. Action A320

[0219] The calibration apparatus 110 can receive a video stream from the second camera B. Accordingly, the video stream is captured by the second camera B.

[0220] Action A330

[0221] The calibration apparatus 110 can render the tracking trace relating to the tracked object 120 in the video stream based on the tracking stream, while using the estimates of the orientation and translation components and the focal length of the second camera B. Any procedure for generating the tracking trace based on the tracking stream can used, e.g. before or in combination with action A330.

[0222] In this manner, the final estimations can be used to transform a tracking trace in the coordinate system of the first camera A to a transformed tracking trace in the coordinate system of the second camera B, i.e. the same coordinate system as the video stream. Next, the tracking trace can be rendered in the video stream. As a result, the experience of the video stream is enhanced for viewers of the video stream.

[0223] With reference to Figure 9, a schematic block diagram of embodiments of the calibration apparatus 110 of Figure 1 is shown.

[0224] The calibration apparatus 110 can include a processing unit 901 for performing the methods described herein. The processing unit can be embodied in the form of one or more hardware units and / or one or more software units. The term "unit" may thus refer to a circuit, a software block or the like according to various embodiments as described below.

[0225] The calibration apparatus 110 may further include a memory 902. The memory can include, such as contain or store, instructions, e.g., in the form of a computer program 903, which can include computer readable code units.

[0226] According to some embodiments herein, the calibration apparatus 110 and / or the processing unit 901 includes a processing circuit 904 as an exemplifying hardware unit, which can include one or more processors. Accordingly, the processing unit 901 may be embodied in the form of, or 'realized by', the processing circuit 904. The instructions may be executable by the processing circuit 904, whereby the calibration apparatus 110 is operative to perform the methods of Figure 3. As another example, the instructions, when executed by the calibration apparatus 110 and / or the processing circuit 904, may cause the calibration apparatus 110 to perform the method according to Figure 3.

[0227] In view of the above, in one example, there is provided a calibration apparatus 110 for determining a coordinate transformation between a first coordinate system of a first camera A and a second coordinate system of a second camera B. As mentioned, the first and second cameras A, B are spaced away from each other and located above a ground plane G, wherein a first field of view of the first camera A at least partially overlaps with a second field of view of the second camera B, wherein a distant object is present in both the first and second fields of view, wherein a first marker R and a second marker L are located at the ground plane G and the first and second markers R, L are visible in the second field of view. Again, the memory 902 contains the instructions executable by said processing circuit 904 whereby the calibration apparatus 110 is operative for: obtaining a set of measures comprising at least five of: a first measure A'R of a first distance, projected on the ground plane G, between the first camera A and the first marker R, a second measure A'L of a second distance, projected on the ground plane G, between the first camera A and the second marker L, a third measure A'B' of a third distance, projected on the ground plane G, between the first camera A and the second camera B, a fourth measure B'R of a fourth distance, projected on the ground plane G, between the second camera B and the first marker R, a fifth measure B'L of a fifth distance, projected on the ground plane G, between the second camera B and the second marker L, or a sixth measure RL of a sixth distance, projected on the ground plane G, between the first marker R and the second marker L, wherein when the set of measures includes all but one measure of the first, second, third, fourth, fifth and sixth measures, calculating A115 said one measure based on the set of measures, 1 estimating a focal length of the second camera B based on the fourth measure B'R, the fifth measure B'L, a first position of the first marker R in the second field of view, a second position of the second marker L in the second field of view, a two-dimensional sensor size of the second camera B xy-plane, the sixth measure RL, and a two-dimensional image resolution of the second camera B, obtaining an object distance D between the distant object P and a location AB at the first camera A and / or the second camera B, determining a first angle a, in a first plane G', xz that is parallel with the ground plane G, indicative of the distant object's P deviation from the first camera's A straight viewing direction based on: the distant object's P position in the first camera A, sensor size of the first camera A, a focal length of the first camera A, and the object distance D, determining a second angle |3, in the first plane G', indicative of the distant object's deviation from the second camera's B straight viewing direction based on: the distant object's position in the second camera B, sensor size of the second camera B, the focal length of the second camera B, and the object distance D, determining a third angle Y, in the first plane G', having a vertex at the distant object's position and two rays to the first and second cameras A, B, respectively, based on: the third measure, and the object distance D, estimating a first orientation component rO.y in the first plane G' based on the first angle, the second angle and the third angle, defining a first circle cl, being centered at a point B' defined by a projection of an optical center of the second camera B at the ground plane G and having a radius defined by the third measure A'B', and a second circle c2, being centered at the second marker L and having a radius defined by the second measure A'L, determining a first intersection il and a second intersection i2 between the first and second circles cl, c2, selecting one of the first and second intersections il, i2 as a position of the first camera A on the ground plane based on a comparison between the first measure A'R and a first distance DI between the first intersection and the second marker L and between the first measure A'R and a second distance D2 between the second intersection and the second marker L, estimating a first and a second translation component of the coordinate transformation based on at least the selected intersection and the second marker L, estimating a first and a second translation component based on the first orientation component and a coordinate of the selected intersection, obtaining at least a first indication and at least a second indication, wherein the first indication relates to a first distance between the first camera A and a first feature point, and wherein the second indication relates to a second distance between the first camera A and a second feature point, wherein the first and second feature points are included in the first and second fields of view, estimating a second orientation component rO.z, based on a first feature angle vA and a second feature angle vB, wherein the first feature angle vA is given in the coordinate system of the first camera A based on the first and second feature points and the second feature angle vB is given in the coordinate system of the second camera B based on the first and second feature points, obtaining an indication of a third translation component tO.y, determining a fourth angle a2, in a second plane yz that is perpendicular to the ground plane G and that is parallel with the first camera's A straight viewing direction, wherein the fourth angle a2 is indicative of the distant object's P deviation from the first camera's A straight viewing direction, wherein the calibration apparatus 110 is configured for determining the fourth angle a2 based on: the distant object's position in the first camera A, sensor size of the first camera A, the focal length of the first camera A, and the object distance D, determining a fifth angle |32, in the second plane yz, indicative of the distant object's P deviation from the second camera's B straight viewing direction based on: the distant object's position in the second camera B, sensor size of the second camera B, the focal length of the second camera B, and the object distance D, determining a sixth angle Y2, in the second plane yz having a vertex at the distant object's position and two rays to the first and second cameras A, B, respectively, based on: the third translation component, and the object distance D, estimating a third orientation component rO.x based on the fourth angle, the fifth angle and the sixth angle, solving a mathematical function dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera B, while minimizing a cost function relating to reprojection error for at least one reprojection point in the first camera's A field of view and the second camera's B field of view, to obtain estimates of the orientation and translation components and the focal length of the second camera B.

[0228] Figure 9 further illustrates a carrier 905, or program carrier, which provides, such as includes, mediates, supplies and the like, the computer program 903 as described directly above. The carrier 905 may be one of an electronic signal, an optical signal, a radio signal, and a computer readable medium.

[0229] In some embodiments, the calibration apparatus 110 and / or the processing unit 901 can include one or more of an obtaining unit 910, an estimating unit 920, a determining unit 930, a defining unit 940, a selecting unit 950, a solving unit 960, a calculating unit 970, a receiving unit 980 and a rendering unit 990 as exemplifying hardware units. The term "unit" may refer to a circuit when the term "unit" refers to a hardware unit. In other examples, one or more of the aforementioned exemplifying units may be implemented as one or more software units.

[0230] Moreover, the calibration apparatus 110 and / or the processing unit 901 can include an Input / Output unit 906, which may be exemplified by the receiving unit and / or the sending unit when applicable.

[0231] Accordingly, the calibration apparatus 110 is configured for determining a coordinate transformation between a first coordinate system of a first camera A and a second coordinate system of a second camera B. The first and second cameras A, B are spaced away from each other and located above a ground plane G. A first field of view of the first camera A at least partially overlaps with a second field of view of the second camera B. A distant object is present in both the first and second fields of view. A first marker R and a second marker L are located at the ground plane G and the first and second markers R, L are visible in the second field of view.

[0232] Therefore, according to the various embodiments described above, the calibration apparatus 110 and / or the processing unit 901 and / or the obtaining unit 910 is configured for obtaining a set of measures comprising at least five of: a first measure A'R of a first distance, projected on the ground plane G, between the first camera A and the first marker R, a second measure A'L of a second distance, projected on the ground plane G, between the first camera A and the second marker L, a third measure A'B' of a third distance, projected on the ground plane G, between the first camera A and the second camera B, a fourth measure B'R of a fourth distance, projected on the ground plane G, between the second camera B and the first marker R, a fifth measure B'L of a fifth distance, projected on the ground plane G, between the second camera B and the second marker L, or a sixth measure RL of a sixth distance, projected on the ground plane G, between the first marker R and the second marker L.

[0233] When the set of measures includes all but one measure of the first, second, third, fourth, fifth and sixth measures, the calibration apparatus 110 and / or the processing unit 901 and / or the calculating unit 970 is configured for calculating said one measure based on the set of measures.

[0234] The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 is configured for estimating a focal length of the second camera B based on the fourth measure B'R, the fifth measure B'L, a first position of the first marker R in the second field of view, a second position of the second marker L in the second field of view, a two-dimensional sensor size of the second camera B xy-plane, the sixth measure RL, and a two-dimensional image resolution of the second camera B.

[0235] The calibration apparatus 110 and / or the processing unit 901 and / or the obtaining unit 910 is configured for obtaining an object distance D between the distant object P and a location AB at the first camera A and / or the second camera B.

[0236] The calibration apparatus 110 and / or the processing unit 901 and / or the determining unit 930 is configured for determining a first angle a, in a first plane G', xz that is parallel with the ground plane G, indicative of the distant object's P deviation from the first camera's A straight viewing direction based on: the distant object's P position in the first camera A, sensor size of the first camera A, a focal length of the first camera A, and the object distance D.

[0237] The calibration apparatus 110 and / or the processing unit 901 and / or the determining unit 930 is configured for determining a second angle |3, in the first plane G', indicative of the distant object's deviation from the second camera's B straight viewing direction based on: the distant object's position in the second camera B, sensor size of the second camera B, the focal length of the second camera B, and the object distance D.

[0238] The calibration apparatus 110 and / or the processing unit 901 and / or the determining unit 930 is configured for determining a third angle Y, in the first plane G', having a vertex at the distant object's position and two rays to the first and second cameras A, B, respectively, based on: the third measure, and the object distance D.

[0239] The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 is configured for estimating a first orientation component rO.y in the first plane G' based on the first angle, the second angle and the third angle.

[0240] The calibration apparatus 110 and / or the processing unit 901 and / or the defining unit 940 is configured for defining a first circle cl, being centered at a point B' defined by a projection of an optical center of the second camera B at the ground plane G and having a radius defined by the third measure A'B', and a second circle c2, being centered at the second marker L and having a radius defined by the second measure A'L.

[0241] The calibration apparatus 110 and / or the processing unit 901 and / or the determining unit 930 is configured for determining a first intersection il and a second intersection i2 between the first and second circles cl, c2.

[0242] The calibration apparatus 110 and / or the processing unit 901 and / or the selecting unit 950 is configured for selecting one of the first and second intersections il, i2 as a position of the first camera A on the ground plane based on a comparison between the first measure A'R and a first distance DI between the first intersection and the second marker L and between the first measure A'R and a second distance D2 between the second intersection and the second marker L. The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 is configured for estimating a first and a second translation component of the coordinate transformation based on at least the selected intersection and the second marker L.

[0243] The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 is configured for estimating a first and a second translation component based on the first orientation component and a coordinate of the selected intersection.

[0244] The calibration apparatus 110 and / or the processing unit 901 and / or the obtaining unit 910 is configured for obtaining at least a first indication and at least a second indication. The first indication relates to a first distance between the first camera A and a first feature point, and wherein the second indication relates to a second distance between the first camera A and a second feature point. The first and second feature points are included in the first and second fields of view.

[0245] The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 is configured for estimating a second orientation component rO.z, based on a first feature angle vA and a second feature angle vB. The first feature angle vA is given in the coordinate system of the first camera A based on the first and second feature points and the second feature angle vB is given in the coordinate system of the second camera B based on the first and second feature points.

[0246] The calibration apparatus 110 and / or the processing unit 901 and / or the obtaining unit 910 is configured for obtaining an indication of a third translation component tO.y.

[0247] The calibration apparatus 110 and / or the processing unit 901 and / or the determining unit 930 is configured for determining a fourth angle a2, in a second plane yz that is perpendicular to the ground plane G and that is parallel with the first camera's A straight viewing direction. The fourth angle a2 is indicative of the distant object's P deviation from the first camera's A straight viewing direction. The calibration apparatus 110 is configured for determining the fourth angle a2 based on: the distant object's position in the first camera A, sensor size of the first camera A, the focal length of the first camera A, and the object distance D.

[0248] The calibration apparatus 110 and / or the processing unit 901 and / or the determining unit 930 is configured for determining a fifth angle [32, in the second plane yz, indicative of the distant object's P deviation from the second camera's B straight viewing direction based on: the distant object's position in the second camera B, sensor size of the second camera B, the focal length of the second camera B, and the object distance D.

[0249] The calibration apparatus 110 and / or the processing unit 901 and / or the determining unit 930 is configured for determining a sixth angle Y2, in the second plane yz having a vertex at the distant object's position and two rays to the first and second cameras A, B, respectively, based on: the third translation component, and the object distance D.

[0250] The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 is configured for estimating a third orientation component rO.x based on the fourth angle, the fifth angle and the sixth angle.

[0251] The calibration apparatus 110 and / or the processing unit 901 and / or the solving unit 960 is configured for solving a mathematical function dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera B, while minimizing a cost function based on distance and reprojection errors to obtain estimates of the orientation and translation components and the focal length of the second camera B.

[0252] The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 can be configured for estimating the first orientation component rO.y by calculating the first orientation component rO.y as the first angle reduced by the second angle and adding the third angle.

[0253] The calibration apparatus 110 and / or the processing unit 901 and / or the estimating unit 920 can be configured for estimating the third orientation component rO.x by calculating the third orientation component rO.x as the fourth angle reduced by the fifth angle and adding the sixth angle.

[0254] The calibration apparatus 110 and / or the processing unit 901 and / or the receiving unit 980 can be configured for receiving, from the first camera A, a tracking stream relating to a tracked object 120.

[0255] The calibration apparatus 110 and / or the processing unit 901 and / or the receiving unit 980, or another receiving unit (not shown), can be configured for receiving a video stream from the second camera B.

[0256] The calibration apparatus 110 and / or the processing unit 901 and / or the rendering unit 990 can be configured for rendering the tracking trace relating to the tracked object 120 in the video stream based on the tracking stream, while using the estimates of the orientation and translation components and the focal length of the second camera B.

[0257] As used herein, the term "unit" may refer to one or more functional units, each of which may be implemented as one or more hardware units and / or one or more software units and / or a combined software / hardware unit in a node. In some examples, the unit may represent a functional unit realized as software and / or hardware of the node.

[0258] As used herein, the term "computer program carrier", "program carrier", or "carrier", may refer to one of an electronic signal, an optical signal, a radio signal, and a computer readable medium. In some examples, the computer program carrier may exclude transitory, propagating signals, such as the electronic, optical and / or radio signal. Thus, in these examples, the computer program carrier may be a non-transitory carrier, such as a non- transitory computer readable medium. As used herein, the term "processing unit" can include one or more hardware units, one or more software units or a combination thereof.

[0259] As used herein, the term "software unit" may refer to a software application, a Dynamic Link Library (DLL), a software component, a software module, a software object, a React component, an object according to Component Object Model (COM), a software function, a software engine, an executable binary software file or the like.

[0260] The terms "processing unit" or "processing circuit" may herein comprise one or more processors, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or the like. The processing circuit or the like may comprise one or more processor kernels.

[0261] As used herein, the expression "configured to / for" may mean that a processing circuit is configured to, such as adapted to or operative to, by means of software configuration and / or hardware configuration, perform one or more of the actions described herein.

[0262] As used herein, the term "action" may refer to an action, a step, an operation, a response, a reaction, an activity or the like. It shall be noted that an action herein may be split into two or more sub-actions as applicable. Moreover, also as applicable, it shall be noted that two or more of the actions described herein may be merged into a single action.

[0263] As used herein, the term "memory" may refer to a hard disk, a magnetic storage medium, a portable computer diskette or disc, flash memory, random access memory (RAM) or the like. Furthermore, the term "memory" may refer to an internal register memory of a processor or the like.

[0264] As used herein, the term "computer readable medium" may be a Universal Serial Bus (USB) memory, a Digital Versatile Disc (DVD), a Blu-ray disc, a software unit that is received as a stream of data, a Flash memory, a hard drive, a memory card, such as a MemoryStick, a Multimedia Card (MMC), Secure Digital (SD) card, etc. One or more of the aforementioned examples of computer readable medium may be provided as one or more computer program products.

[0265] As used herein, the term "computer readable code units" may be text of a computer program, parts of or an entire binary file representing a computer program in a compiled format or anything there between.

[0266] Any feature disclosed for one example and / or embodiment may be combined with one or more features disclosed for one or more other examples and / or embodiments without departing from the scope herein.

Claims

CLAIMS1. A method for determining a coordinate transformation between a first coordinate system of a first camera (A) and a second coordinate system of a second camera (B), wherein the first and second cameras (A, B) are spaced away from each other and located above a ground plane (G, xz-plane), wherein a first field of view of the first camera (A) at least partially overlaps with a second field of view of the second camera (B), wherein a distant object (P) is present in both the first and second fields of view, wherein a first marker (R) and a second marker (L) are located at the ground plane (G) and the first and second markers (R, L) are visible in the second field of view, wherein the method comprises: obtaining (A110) a set of measures comprising at least five of: a first measure (A'R) of a first distance, projected on the ground plane (G), between the first camera (A) and the first marker (R), a second measure (A'L) of a second distance, projected on the ground plane (G), between the first camera (A) and the second marker (L), a third measure (A'B') of a third distance, projected on the ground plane (G), between the first camera (A) and the second camera (B), a fourth measure (B'R) of a fourth distance, projected on the ground plane (G), between the second camera (B) and the first marker (R), a fifth measure (B'L) of a fifth distance, projected on the ground plane (G), between the second camera (B) and the second marker (L), or a sixth measure (RL) of a sixth distance, projected on the ground plane (G), between the first marker (R) and the second marker (L), wherein when the set of measures comprises all but one measure of the first, second, third, fourth, fifth and sixth measures, calculating (A115) said one measure based on the set of measures; estimating (A120) a focal length of the second camera (B) based on the fourth measure (B'R), the fifth measure (B'L), a first position of the first marker (R) in the second field of view, a second position of the second marker (L) in the second field of view, a two-dimensional sensor size of the second camera (B),the sixth measure (RL), and a two-dimensional image resolution of the second camera (B); obtaining (A130) an object distance (D) between the distant object (P) and a location (AB) at the first camera (A) and / or the second camera (B); determining (A140) a first angle (a), in a first plane (G', xz) that is parallel with the ground plane (G), indicative of a distant object's (P) deviation from a first camera's (A) straight viewing direction based on: a distant object's (P) position in the first camera (A), a sensor size of the first camera (A), a focal length of the first camera (A), and the object distance (D); determining (A150) a second angle (P), in the first plane (G'), indicative of a distant object's deviation from a second camera's (B) straight viewing direction based on: a distant object's position in the second camera (B), a sensor size of the second camera (B), the focal length of the second camera (B), and the object distance (D); determining (A160) a third angle (Y), in the first plane (G'), having a vertex at the distant object's position and two rays to the first and second cameras (A, B), respectively, based on: the third measure, and the object distance (D); estimating (A170) a first orientation component (rO.y) in the first plane (G') based on the first angle, the second angle and the third angle; defining (A180) a first circle (cl), being centered at a point (B') defined by a projection of an optical center of the second camera (B) at the ground plane (G) and having a radius defined by the third measure (A'B'), and a second circle (c2), being centered at the second marker (L) and having a radius defined by the second measure (A'L); determining (A190) a first intersection (il) and a second intersection (i2) between the first and second circles (cl, c2);selecting (A200) one of the first and second intersections (il, i2) as a position of the first camera (A) on the ground plane based on a comparison between the first measure (A'R) and a first distance (DI) between the first intersection and the second marker (L) and between the first measure (A'R) and a second distance (D2) between the second intersection and the second marker (L); estimating (A210) a first and a second translation component based on at least the selected intersection and the second marker (L); estimating (A220) a first and a second translation component based on the first orientation component and a coordinate of the selected intersection; obtaining (A230) at least a first indication and at least a second indication, wherein the first indication relates to a first distance between the first camera (A) and a first feature point, and wherein the second indication relates to a second distance between the first camera (A) and a second feature point, wherein the first and second feature points are included in the first and second fields of view; estimating (A240) a second orientation component, rO.z, based on a first feature angle (vA) and a second feature angle (vB), wherein the first feature angle (vA) is given in the coordinate system of the first camera (A) based on the first and second feature points and the second feature angle (vB) is given in the coordinate system of the second camera (B) based on the first and second feature points; obtaining (A250) an indication of a third translation component (tO.y); determining (A260) a fourth angle (a2), in a second plane (yz) that is perpendicular to the ground plane (G) and that is parallel with the first camera's (A) straight viewing direction, wherein the fourth angle (a2) is indicative of the distant object's (P) deviation from the first camera's (A) straight viewing direction, wherein the determining (A260) is based on: the distant object's position in the first camera (A), sensor size of the first camera (A), the focal length of the first camera (A), and the object distance; determining (A270) a fifth angle (P2), in the second plane (yz), indicative of the distant object's (P) deviation from the second camera's (B) straight viewing direction based on:the distant object's position in the second camera (B), sensor size of the second camera (B), the focal length of the second camera (B), and the object distance; determining (A280) a sixth angle (Y2), in the second plane (yz) having a vertex at the distant object's position and two rays to the first and second cameras (A, B), respectively, based on: the third translation component, and the object distance; estimating (A290) a third orientation component (rO.x) based on the fourth angle, the fifth angle and the sixth angle; and solving (A300) a mathematical function dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera (B) while minimizing a cost function relating to reprojection error for at least one reprojection point in the first camera's (A) field of view and the second camera's (B) field of view, to obtain estimates of the orientation and translation components and the focal length of the second camera (B).

2. The method according to claim 1, wherein the estimation (A170) of the first orientation component (rO.y) comprises calculating the first orientation component (rO.y) as the first angle reduced by the second angle and adding the third angle.

3. The method according to claim 1 or 2, wherein the estimation (A290) of the third orientation component (rO.x) comprises calculating the third orientation component (rO.x) as the fourth angle reduced by the fifth angle and adding the sixth angle.

4. The method according to any one of the preceding claims, wherein the method comprises: receiving (A310), from the first camera (A), a tracking stream relating to a tracked object (120), wherein the tracking stream includes information for generating a tracking trace, wherein the tracking stream includes one or more of image frames captured by the first camera (A), tracking data, blob information, paths, or tracks;receiving (A320) a video stream from the second camera (B); and rendering (A330) the tracking trace relating to the tracked object (120) in the video stream based on the tracking stream, while using the estimates of the orientation and translation components and the focal length of the second camera (B).

5. A calibration apparatus (110) configured for determining a coordinate transformation between a first coordinate system of a first camera (A) and a second coordinate system of a second camera (B), wherein the first and second cameras (A, B) are spaced away from each other and located above a ground plane (G), wherein a first field of view of the first camera (A) at least partially overlaps with a second field of view of the second camera (B), wherein a distant object is present in both the first and second fields of view, wherein a first marker (R) and a second marker (L) are located at the ground plane (G) and the first and second markers (R, L) are visible in the second field of view, wherein the calibration apparatus (110) is configured for: obtaining a set of measures comprising at least five of: a first measure (A'R) of a first distance, projected on the ground plane (G), between the first camera (A) and the first marker (R), a second measure (A'L) of a second distance, projected on the ground plane (G), between the first camera (A) and the second marker (L), a third measure (A'B') of a third distance, projected on the ground plane (G), between the first camera (A) and the second camera (B), a fourth measure (B'R) of a fourth distance, projected on the ground plane (G), between the second camera (B) and the first marker (R), a fifth measure (B'L) of a fifth distance, projected on the ground plane (G), between the second camera (B) and the second marker (L), or a sixth measure (RL) of a sixth distance, projected on the ground plane (G), between the first marker (R) and the second marker (L), wherein when the set of measures comprises all but one measure of the first, second, third, fourth, fifth and sixth measures, calculating (A115) said one measure based on the set of measures; estimating a focal length of the second camera (B) based on the fourth measure (B'R),the fifth measure (B'L), a first position of the first marker (R) in the second field of view, a second position of the second marker (L) in the second field of view, a two-dimensional sensor size of the second camera (B) , the sixth measure (RL), and a two-dimensional image resolution of the second camera (B); obtaining an object distance (D) between the distant object (P) and a location (AB) at the first camera (A) and / or the second camera (B); determining a first angle (a), in a first plane (G', xz) that is parallel with the ground plane (G), indicative of a distant object's (P) deviation from a first camera's (A) straight viewing direction based on: a distant object's (P) position in the first camera (A), a sensor size of the first camera (A), a focal length of the first camera (A), and the object distance (D); determining a second angle ( |3), in the first plane (G'), indicative of a distant object's deviation from a second camera's (B) straight viewing direction based on: a distant object's position in the second camera (B), a sensor size of the second camera (B), the focal length of the second camera (B), and the object distance (D); determining a third angle (Y), in the first plane (G'), having a vertex at the distant object's position and two rays to the first and second cameras (A, B), respectively, based on: the third measure, and the object distance (D); estimating a first orientation component (rO.y) in the first plane (G') based on the first angle, the second angle and the third angle; defining a first circle (cl), being centered at a point (B') defined by a projection of an optical center of the second camera (B) at the ground plane (G) and having a radius defined by the third measure (A'B'), and a second circle (c2), being centered at the second marker (L) and having a radius defined by the second measure (A'L);determining a first intersection (il) and a second intersection (i2) between the first and second circles (cl, c2); selecting one of the first and second intersections (il, i2) as a position of the first camera (A) on the ground plane based on a comparison between the first measure (A'R) and a first distance (DI) between the first intersection and the second marker (L) and between the first measure (A'R) and a second distance (D2) between the second intersection and the second marker (L); estimating a first and a second translation component based on at least the selected intersection and the second marker (L); estimating a first and a second translation component based on the first orientation component and a coordinate of the selected intersection; obtaining at least a first indication and at least a second indication, wherein the first indication relates to a first distance between the first camera (A) and a first feature point, and wherein the second indication relates to a second distance between the first camera (A) and a second feature point, wherein the first and second feature points are included in the first and second fields of view; estimating a second orientation component (rO.z), based on a first feature angle (vA) and a second feature angle (vB), wherein the first feature angle (vA) is given in the coordinate system of the first camera (A) based on the first and second feature points and the second feature angle (vB) is given in the coordinate system of the second camera (B) based on the first and second feature points; obtaining an indication of a third translation component (tO.y); determining a fourth angle (a2), in a second plane (yz) that is perpendicular to the ground plane (G) and that is parallel with the first camera's (A) straight viewing direction, wherein the fourth angle (a2) is indicative of the distant object's (P) deviation from the first camera's (A) straight viewing direction, wherein the calibration apparatus (110) is configured for determining the fourth angle (a2) based on: the distant object's position in the first camera (A), sensor size of the first camera (A), the focal length of the first camera (A), and the object distance (D);determining a fifth angle (P2), in the second plane (yz), indicative of the distant object's (P) deviation from the second camera's (B) straight viewing direction based on: the distant object's position in the second camera (B), sensor size of the second camera (B), the focal length of the second camera (B), and the object distance (D); determining a sixth angle (Y2), in the second plane (yz) having a vertex at the distant object's position and two rays to the first and second cameras (A, B), respectively, based on: the third translation component, and the object distance (D); estimating a third orientation component (rO.x) based on the fourth angle, the fifth angle and the sixth angle; and solving a mathematical function dependent on the orientation and translation components of the coordinate transformation and the focal length of the second camera (B), while minimizing a cost function relating to reprojection error for at least one reprojection point in the first camera's (A) field of view and the second camera's (B) field of view, to obtain estimates of the orientation and translation components and the focal length of the second camera (B).

6. The calibration apparatus (110) according to claim 5, wherein the calibration apparatus is configured for estimating the first orientation component (rO.y) by calculating the first orientation component (rO.y) as the first angle reduced by the second angle and adding the third angle.

7. The calibration apparatus (110) according to claim 5 or 6, wherein the calibration apparatus (110) is configured for estimating the third orientation component (rO.x) by calculating the third orientation component (rO.x) as the fourth angle reduced by the fifth angle and adding the sixth angle.

8. The calibration apparatus (110) according to any one of claims 5-7, wherein the calibration apparatus (110) is configured for: receiving, from the first camera (A), a tracking stream relating to a tracked object (120), wherein the tracking stream includes information for generating a tracking trace, receiving a video stream from the second camera (B), and rendering the tracking trace relating to the tracked object (120) in the video stream based on the tracking stream, while using the estimates of the orientation and translation components and the focal length of the second camera (B).

9. A computer program (903), comprising computer readable code units which when executed on a calibration apparatus (110) causes the calibration apparatus (110) to perform the method according to any one of claims 1-4.

10. A non-transitory computer-readable medium comprising computer readable code units which when executed on a calibration apparatus cause the calibration apparatus to perform the method of any one of claims 1-4.

11. A system (100) comprising a first camera (A), a second camera (B) and a calibration apparatus (110) according to any one of claims 5 -8.

12. The system (100) according to claim 11, wherein the first camera (A) is a tracking sensor and / or the second camera (B) is a broadcast camera.

Images